Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (μTransfer)

👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦 https://discord.gg/peBrCpheKE In this video I cover "Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (μTransfer)" paper that makes optimal hyperparameters stable w.r.t. width scaling! ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ ✅ Paper: https://arxiv.org/abs/2203.03466 ✅ Their previous paper: https://arxiv.org/abs/2011.14522 ✅ μTransfer tool: https://github.com/microsoft/mup ✅ DeepNet paper: https://arxiv.org/abs/2203.00555 ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ ⌚️ Timetable: 00:00 uTransfer introduced 04:30 Previous work (tensor programs IV) 10:00 NTK - neural tangent kernel recap 16:30 abc parametrization 19:25 How does learning happen in NTK? 33:50 Connections to Central Limit Theorem 39:30 Maximal Update Parametrization in Practice 45:50 DeepNet paper connection 48:20 Results (width is all you need?) ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ THE AI EPIPHANY PATREONS WALL ❤️ Eli Mahler Kevin Stone Petar Veličković ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 🐦 Twitter - https://twitter.com/gordic_aleksa 👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE 📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/ 📚 Medium - https://gordicaleksa.medium.com/ 💻 GitHub - https://github.com/gordicaleksa 📢 AI Newsletter - https://aiepiphany.substack.com/ ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ #μTransfer #MaximalUpdateParametrization #scalingl