OPT-175B: Open Pretrained Transformer | ML Coding Series
❤️ Become The AI Epiphany Patreon ❤️ https://www.patreon.com/theaiepiphany 👨👩👧👦 Join our Discord community 👨👩👧👦 https://discord.gg/peBrCpheKE In this video I do a deep dive into the metaseq (Open Pretrained Transformer, OPT-175B) codebase. I first show you how to set up the code on your machine, and then I walk you through the codebase behind Meta's large language model (LLM). Along the way I cover key concepts behind mixed precision training such as loss scaling and unscaling, and much more. ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ ✅ My fork of metaseq: https://github.com/gordicaleksa/metaseq ✅ Instructions for Apex: https://github.com/bigscience-workshop/Megatron-DeepSpeed/blob/main/start_fast.md ✅ Ultimate guide to scaling vid: https://www.youtube.com/watch?v=hc0u4avAkuM ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ ⌚️ Timetable: 00:00:00 Intro - open pretrained transformer 00:01:25 Setup (creating the cond env) 00:06:00 Setup (patch the code) 00:11:40 Collecting train script arguments 00:17:49 Training script walk-through 00:25:25 Constructing a dummy task 00:28:30 Building the transformer model 00:35:22 CUDA kernels (C++ code) 00:46:30 Preparing a dummy dataset 00:50:00 Training loop 00:54:05 Zero grad (loss scaling) 00:56:15 Forward pass through a transformer 01:02:20 (IMPORTANT) loss, scaling, mixed precision, error handling 01:13:25 Outro ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ If these videos, GitHub projects, and blogs help you, consider helping me out by supporting me on Patreon! The AI Epiphany - https://www.patreon.com/theaiepiphany One-time donation - https://www.paypal.com/paypalme/theaiepiphany Huge thank you to these AI Epiphany patreons: Eli Mahler Petar Veličković ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 🐦 Twitter - https://twitter.com/gordic_aleksa 👨👩👧👦 Discord - https://discord.gg/peBrCpheKE 📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/ 📚 Medium - https://gordicaleksa.medium.com/ 💻 GitHub - https://github.com/gordicaleksa 📢 AI Newsletter - https://aiepiphany.substack.com/ ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ #opt #meta #largelanguagemodels #transformers
