GPT-NeoX-20B | BigScience BLOOM | OPT-175B | Training Large Language Models | Papers Explained
🚀 Find out how to get started using Weights & Biases 🚀 http://wandb.me/ai-epiphany ❤️ Become The AI Epiphany Patreon ❤️ https://www.patreon.com/theaiepiphany In this video I cover 3 publicly shared LLM 🚀 projects/papers and the pain they experience training them (🍿🍿🍿): 1. "What Language Model to Train if You Have One Million GPU Hours?" introducing BLOOM 176 billion parameter model by BigScience! 2. "OPT: Open Pre-trained Transformer Language Models" introducing 175 billion parameter model OPT-175B! 3. "GPT-NeoX-20B: An Open-Source Autoregressive Language Model" introducing, well, GPT-NeoX-20B, a 20 billion parameter LLM! All 3 projects shared their weights, code, and papers, so it's a great way to dig into large language models and understand them better. I walk you through the papers and the chronicles/logs they've shared sharing the pain they experienced training at these scales! :)) Cluster deletion, 2 million consecutive backslash symbols in the dataset, and much more - it's fun! ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ ✅ BLOOM paper: https://openreview.net/forum?id=rI7BL3fHIZq ✅ OPT paper: https://arxiv.org/abs/2205.01068 ✅ GPT-NeoX-20B paper: https://arxiv.org/abs/2204.06745 ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ ⌚️ Timetable: 00:00 Intro 00:33 (sponsored) Weights & Biases 02:43 BLOOM paper 19:37 BLOOM chronicles 25:08 OPT paper 31:20 OPT chronicles 33:46 GPT-NeoX-20B paper 45:30 Outro ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ If these videos, GitHub projects, and blogs help you, consider helping me out by supporting me on Patreon! The AI Epiphany - https://www.patreon.com/theaiepiphany One-time donation - https://www.paypal.com/paypalme/theaiepiphany Huge thank you to these AI Epiphany patreons: Eli Mahler Petar Veličković ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 🐦 Twitter - https://twitter.com/gordic_aleksa 👨👩👧👦 Discord - https://discord.gg/peBrCpheKE 📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/ 📚 Medium - https://gordicaleksa.medium.com/ 💻 GitHub - https://github.com/gordicaleksa 📢 AI Newsletter - https://aiepiphany.substack.com/ ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ #gptneox #bigscience #opt
