MoE Token Routing Explained: How Mixture of Experts Works (with Code)

This video dives deep into Token Routing, the core algorithm of Mixture of Experts (MoE) models. Slides: https://huggingface.co/ariG23498/moe-routing-algorithm/blob/main/MoE_routing_algorithm.pdf Colab Notebook: https://huggingface.co/ariG23498/moe-routing-algorithm/blob/main/routing_mechanism.ipynb Chapter Timestamp: Introduction: 00:00 Laying the Foundation for Mixture of Experts (MoE): 00:09 Focus on Token Routing: 00:50 What is a Mixture of Experts Layer?: 02:36 Problem Statement and Configurations: 04:48 Compute Router Logits: 08:31 Sparsity and Selecting Top K Experts: 10:54 Normalizing Logits to Router Probabilities: 12:43 Slot Selection: 14:39 Dropping Oversubscribed Tokens: 16:51 Updated Normalized Token Weights: 20:36 Updated Slot Selection and Token Slots: 21:34 Final Weight Matrix Construction: 24:35 Conclusion: 32:41 Correction: As @denisflavius5365 rightly suggested the router matrix in the slides should be of the shape 3x4 and not 4x4.