Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (ΞΌTransfer)
π¨βπ©βπ§βπ¦ Join our Discord community π¨βπ©βπ§βπ¦ https://discord.gg/peBrCpheKE In this video I cover "Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (ΞΌTransfer)" paper that makes optimal hyperparameters stable w.r.t. width scaling! β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬ β Paper: https://arxiv.org/abs/2203.03466 β Their previous paper: https://arxiv.org/abs/2011.14522 β ΞΌTransfer tool: https://github.com/microsoft/mup β DeepNet paper: https://arxiv.org/abs/2203.00555 β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬ βοΈ Timetable: 00:00 uTransfer introduced 04:30 Previous work (tensor programs IV) 10:00 NTK - neural tangent kernel recap 16:30 abc parametrization 19:25 How does learning happen in NTK? 33:50 Connections to Central Limit Theorem 39:30 Maximal Update Parametrization in Practice 45:50 DeepNet paper connection 48:20 Results (width is all you need?) β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬ THE AI EPIPHANY PATREONS WALL β€οΈ Eli Mahler Kevin Stone Petar VeliΔkoviΔ β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬ πΌ LinkedIn - https://www.linkedin.com/in/aleksagordic/ π¦ Twitter - https://twitter.com/gordic_aleksa π¨βπ©βπ§βπ¦ Discord - https://discord.gg/peBrCpheKE πΊ YouTube - https://www.youtube.com/c/TheAIEpiphany/ π Medium - https://gordicaleksa.medium.com/ π» GitHub - https://github.com/gordicaleksa π’ AI Newsletter - https://aiepiphany.substack.com/ β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬ #ΞΌTransfer #MaximalUpdateParametrization #scalingl
