Imbue - training a 70B model from scratch! (w/ Bowei - head of infra)
Become a Patreon: https://www.patreon.com/theaiepiphany π¨βπ©βπ§βπ¦ Join our Discord community: https://discord.gg/peBrCpheKE Bowei joined us from Imbue to talk about Imbue's latest endeavor: building the infra to support a 70B model training! They wrote up an amazing blog post with a lot of details describing the grind that was needed to set everything up from scratch. :) β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬ Blog post: https://imbue.com/research/70b-infrastructure/ (From bare metal to a 70B model: infrastructure set-up and scripts) β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬ βοΈ Timetable: 00:00 - 00:45 Intro 00:45 - 02:25 Hyperstack GPUs (sponsored) 02:25 - 11:30 Bowei's background 11:30 - 18:30 More on Imbue, their research, their focus 18:30 - 26:20 Training a 70B model 26:20 - 45:40 Building a cluster from scratch 45:40 - 59:25 Anecdotes, Q&A β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬ π° SPONSOR The AI Epiphany - https://www.patreon.com/theaiepiphany One-time donation - https://www.paypal.com/paypalme/theaiepiphany Huge thank you to these AI Epiphany patreons: Eli Mahler Petar VeliΔkoviΔ β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬ πΌ LinkedIn - https://www.linkedin.com/in/aleksagordic/ π¦ Twitter - https://twitter.com/gordic_aleksa π¨βπ©βπ§βπ¦ Discord - https://discord.gg/peBrCpheKE πΊ YouTube - https://www.youtube.com/c/TheAIEpiphany/ π Medium - https://gordicaleksa.medium.com/ π» GitHub - https://github.com/gordicaleksa π’ AI Newsletter - https://aiepiphany.substack.com/ β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬ #imbue #infra #70b #llm
