I Trained Qwen 4B with Distillation to Catch Qwen 27B
Can a 27 billion parameter model teach a 4 billion model to be as good as itself? Last time, RL got Qwen 4B from 52% to 58.4% and then hit a wall. This time I used Qwen 27B as a teacher and tried all three kinds of distillation on two DGX Sparks: training on the teacher's answers, training on its probabilities, and on-policy distillation where the 27B grades every token the 4B writes. Sponsored by Dex, the talent agent for senior engineers. Free for engineers: https://meetdex.link/promptengineering-1 Links - Previous video (RL, the sandbox, the 58% wall): https://youtu.be/Pl_2TelxjdY - Code: https://github.com/PromtEngineer/smoldataenvs-grpo-spark - SmolDataEnvs (the tasks): https://huggingface.co/datasets/FineEnvs/SmolDataEnvs - Thinking Machines, On-Policy Distillation: https://thinkingmachines.ai/blog/on-policy-distillation/ - Qwen3 Technical Report (strong-to-weak distillation): https://arxiv.org/abs/2505.09388 - Hinton, Vinyals, Dean (2015), Distilling the Knowledge in a Neural Network: https://arxiv.org/abs/1503.02531 - On-policy distillation papers: MiniLLM https://arxiv.org/abs/2306.08543 · GKD https://arxiv.org/abs/2306.13649 Chapters 0:00 Can Qwen 27B teach Qwen 4B? 0:25 What RL did last time, and the wall 1:47 The short answer 2:23 The setup: SmolDataEnvs and a strict checker 4:03 Why RL stopped helping 4:48 How distillation works 5:24 Why the teacher's probabilities matter 6:17 Three ways to distill 8:10 Two DGX Sparks: teacher and student 8:53 Method 1: train on the teacher's answers 09:55 Sponsor: Dex 11:00 Method 2: train on the teacher's probabilities 12:14 Why the probabilities added nothing 13:32 The limit of learning from transcripts 13:55 Method 3: on-policy distillation 14:38 A real run, graded token by token 15:43 Did it break the 58% wall? 17:13 All six methods on one chart 18:12 What the 4B model already knows 19:41 The 4-tries test 20:47 Caveats 21:04 Two gotchas: tokenizers and vLLM memory 22:06 What should you use? #distillation #qwen #llm #reinforcementlearning #localai #dgxspark
