BigScience BLOOM | 3D Parallelism Explained | Large Language Models | ML Coding Series

❤️ Become The AI Epiphany Patreon ❤️ https://www.patreon.com/theaiepiphany 👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦 https://discord.gg/peBrCpheKE In this 4th video of the Large Language Model series I walk you through the BigScience's BLOOM model codebase! The main focus is on understanding the 3D parallelism: * Pipeline parallelism * Model parallelism * Data parallelism A set of beautiful engineering ideas that are behind all of the recent scaling efforts and ML success stories! ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ ✅ BLOOM code: https://github.com/bigscience-workshop/Megatron-DeepSpeed ✅ Ultimate Guide to Scaling Video: https://www.youtube.com/watch?v=hc0u4avAkuM ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ ⌚️ Timetable: 00:00:00 Intro - focusing on the 3D parallelism! 00:02:00 Quick setup 00:05:00 Stepping through the eval script 00:11:13 3D paralellism - model construction 00:15:00 Sharding the embedding table (model parallelism) 00:20:09 Sharding the transformer layer 00:22:30 LayerNorm fused kernels 00:23:50 Sharding the attention layer 00:25:15 ColumnParallel and RowParallel sharding 00:31:30 Synchronizing input and output embedding tables 00:34:45 Building the dataset (data parallelism) 00:39:15 3D parallelism - forward pass 00:39:25 Pipeline parallelism communication 00:43:35 Pass through the sharded embedding table 00:52:15 Pass through the sharded transformer layer 01:01:36 Sharded logit and cross-entropy computation 01:05:30 Recap 01:11:15 Outro ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ If these videos, GitHub projects, and blogs help you, consider helping me out by supporting me on Patreon! The AI Epiphany - https://www.patreon.com/theaiepiphany One-time donation - https://www.paypal.com/paypalme/theaiepiphany Huge thank you to these AI Epiphany patreons: Eli Mahler Petar Veličković ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 🐦 Twitter - https://twitter.com/gordic_aleksa 👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE 📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/ 📚 Medium - https://gordicaleksa.medium.com/ 💻 GitHub - https://github.com/gordicaleksa 📢 AI Newsletter - https://aiepiphany.substack.com/ ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ #bloom #bigscience #3dparallelism #scaling