ALiBi | Train Short, Test Long: Attention With Linear Biases Enables Input Length Extrapolation
π¨βπ©βπ§βπ¦ JOIN OUR DISCORD COMMUNITY: Discord βΊ https://discord.gg/peBrCpheKE π’ SUBSCRIBE TO MY MONTHLY AI NEWSLETTER: Substack βΊ https://aiepiphany.substack.com/ β€οΈ Become The AI Epiphany Patreon β€οΈ βΊ https://www.patreon.com/theaiepiphany In this video I cover ALiBi model from the "Train Short, Test Long: Attention With Linear Biases Enables Input Length Extrapolation" paper. Instead of using positional embeddings (like e.g. the original transformer) they added non-learnable biases directly into the query-key matrix and achieved extraordinary extrapolation results (i.e. great perf on longer sequences). β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬ β Paper: https://ofir.io/train_short_test_long.pdf β Code: https://github.com/ofirpress/attention_with_linear_biases β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬ βοΈ Timetable: 00:00 Intro 02:00 Main results 03:30 Time and memory tradeoffs 05:00 ALiBi method explained 10:00 Results, low data regime 12:35 Generalization to different datasets 13:15 Results, big data regime 16:00 Why does it work and future work 21:10 Outro β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬ π° BECOME A PATREON OF THE AI EPIPHANY β€οΈ If these videos, GitHub projects, and blogs help you, consider helping me out by supporting me on Patreon! The AI Epiphany βΊ https://www.patreon.com/theaiepiphany One-time donation: https://www.paypal.com/paypalme/theaiepiphany Much love! β€οΈ Huge thank you to these AI Epiphany patreons: Eli Mahler Petar VeliΔkoviΔ Zvonimir Sabljic β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬ π‘ The AI Epiphany is a channel dedicated to simplifying the field of AI using creative visualizations and in general, a stronger focus on geometrical and visual intuition, rather than the algebraic and numerical "intuition". β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬ π CONNECT WITH ME ON SOCIAL LinkedIn βΊ https://www.linkedin.com/in/aleksagordic/ Twitter βΊ https://twitter.com/gordic_aleksa Instagram βΊ https://www.instagram.com/aiepiphany/ Facebook βΊ https://www.facebook.com/aiepiphany/ π¨βπ©βπ§βπ¦ JOIN OUR DISCORD COMMUNITY: Discord βΊ https://discord.gg/peBrCpheKE π’ SUBSCRIBE TO MY MONTHLY AI NEWSLETTER: Substack βΊ https://aiepiphany.substack.com/ π» FOLLOW ME ON GITHUB FOR ML PROJECTS: GitHub βΊ https://github.com/gordicaleksa π FOLLOW ME ON MEDIUM: Medium βΊ https://gordicaleksa.medium.com/ β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬β¬ #alibi #transformers #extrapolation
