Ultimate Guide To Scaling ML Models - Megatron-LM | ZeRO | DeepSpeed | Mixed Precision

🚀 Sign up for AssemblyAI's speech API using my link 🚀 https://www.assemblyai.com/?utm_source=youtube&utm_medium=social&utm_campaign=theaiepiphany 👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦 https://discord.gg/peBrCpheKE In this video I show you what it takes to scale ML models up to trillions of parameters! I cover the fundamental ideas behind all of the recent big ML models you must have heard of like Meta's OPT-175B, BigScience BLOOM 176B, EleutherAI's GPT-NeoX-20B, GPT-J, OpenAI's GPT-3, Google's PaLM, DeepMind's Chinchilla/Gopher models, etc. I cover the ideas of data parallelism, model/pipeline parallelism (e.g. GPipe, PipeDream, etc.), model/tensor parallelism (Megatron-LM), activation checkpointing, mixed precision training, ZeRO (zero redundancy optimizer) from Microsoft's DeepSpeed library and many more. ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ Papers: ✅ Megatron-LM paper: https://arxiv.org/abs/1909.08053 ✅ ZeRO (DeepSpeed) paper: https://arxiv.org/abs/1910.02054v3 ✅ Mixed precision training paper: https://arxiv.org/abs/1710.03740 ✅ Gpipe (pipeline parallelism) paper: https://arxiv.org/abs/1811.06965 Articles: ✅ Collective ops: https://en.wikipedia.org/wiki/Collective_operation ✅ IEEE float16 format: https://en.wikipedia.org/wiki/Half-precision_floating-point_format ✅ Google Brain's bfloat16 format: https://cloud.google.com/blog/products/ai-machine-learning/bfloat16-the-secret-to-high-performance-on-cloud-tpus ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ ⌚️ Timetable: 00:00:00 Intro to training Large ML models (trillions of params!) 00:02:04 (sponsored) AssemblyAI's speech transcription API 00:03:31 Data parallelism 00:01:52 Pipeline/model parallelism 00:14:52 Megatron-LM paper (tensor/model parallelism) 00:18:22 Splitting the MLP block vertically 00:30:07 Splitting the attention block vertically 00:39:24 Activation checkpointing 00:42:12Combining data + model parallelism 00:45:42 Scaling is all you need and 3D parallelism 00:47:57 Mixed precision training paper 00:49:57 Single vs half vs bfloat number formats 00:51:32 Storing master weights in single precision 00:55:41 Loss scaling 00:58:13 Arithmetic precision matters 01:00:32 ZeRO optimizer paper (DeepSpeed library) 01:06:37 Partitioning is all you need? 01:11:02 Where did all the memory go? 01:21:42 Outro ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ If these videos, GitHub projects, and blogs help you, consider helping me out by supporting me on Patreon! The AI Epiphany - https://www.patreon.com/theaiepiphany One-time donation - https://www.paypal.com/paypalme/theaiepiphany Huge thank you to these AI Epiphany patreons: Eli Mahler Petar Veličković ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 🐦 Twitter - https://twitter.com/gordic_aleksa 👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE 📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/ 📚 Medium - https://gordicaleksa.medium.com/ 💻 GitHub - https://github.com/gordicaleksa 📢 AI Newsletter - https://aiepiphany.substack.com/ ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ #scaling #deepspeed #megatron