Flash Attention 2.0 with Tri Dao (author)! | Discord server talks

❤️ Become The AI Epiphany Patreon ❤️ https://www.patreon.com/theaiepiphany 👨‍👩‍👧‍👦 Join our Discord community 👨‍👩‍👧‍👦 https://discord.gg/peBrCpheKE Hey guys! I've been uploading less frequently lately since I started my startup - as I'm trying to focus and learn as much as possible. Until I get back into the rhythm I'll be experimenting a bit with the channel: expect talks from my Discord server, live streaming, etc. Here is a talk we had in my Discord server with Tri Dao from Stanford, talking about his recent work on Flash Attention 2.0! ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ ✅ My Flash Attention blog: https://gordicaleksa.medium.com/eli5-flash-attention-5c44017022ad ✅ Horace's blog: https://horace.io/brrr_intro.html ✅ Flash Attention paper: https://arxiv.org/abs/2205.14135 ✅ Flash Attention 2.0 paper: https://tridao.me/publications/flash2/flash2.pdf ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ ⌚️ Timetable: 00:00 Main talk starts - intro & motivation 10:20 Behind the scenes: how Tri got started with Flash Attention 16:40 Motivation: modelling long sequences 19:50 Brief recap of attention 23:15 Memory bottleneck, IO awareness 35:10 Flash Attention 2.0 improvements 37:20 Behind the scenes of Flash Attention 2.0 (refactor of CUTLASS 3) 43:45 Future directions 46:45 Q&A ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 💰 BECOME A PATREON OF THE AI EPIPHANY ❤️ If these videos, GitHub projects, and blogs help you, consider helping me out by supporting me on Patreon! The AI Epiphany - https://www.patreon.com/theaiepiphany One-time donation - https://www.paypal.com/paypalme/theaiepiphany Huge thank you to these AI Epiphany patreons: Eli Mahler Petar Veličković ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 💼 LinkedIn - https://www.linkedin.com/in/aleksagordic/ 🐦 Twitter - https://twitter.com/gordic_aleksa 👨‍👩‍👧‍👦 Discord - https://discord.gg/peBrCpheKE 📺 YouTube - https://www.youtube.com/c/TheAIEpiphany/ 📚 Medium - https://gordicaleksa.medium.com/ 💻 GitHub - https://github.com/gordicaleksa 📢 AI Newsletter - https://aiepiphany.substack.com/ ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ #flashattention #tridao #mlsys