I Trained Qwen 4B With RL to Catch Qwen 27B
Can RL Make a 4B Model Compete with 27B? My Open RL Environment Experiments (GRPO, LoRA, DGX) In this video I test whether open-source RL environments can make a small model perform closer to a much larger one by training Qwen 3.5 4B (with a LoRA) against Kaggle-style “small data” RL tasks on two DGX Sparks and comparing to Qwen 27B on a held-out exam. I explain what RL environments are (sandbox, tools, checker), why you need a strict checker to avoid reward hacking, and how GRPO works and only learns from tasks the model sometimes gets right. After weak gains from random tasks, I filter for “edge” tasks and get clear improvement, closing over half the gap, mostly by teaching the model to finish and answer correctly. Further RL hits a ceiling, so I try distillation from 27B, then RL on top, which gives the best results on unseen datasets. I share takeaways and put code, checker, and recipes on GitHub. Thanks to FireCrawl for sponsoring this video: Firecrawl, the context API for AI agents (free tier to get started) https://firecrawl.link/promptengineer 💻 Code: harness, strict checker, training scripts and results https://github.com/PromtEngineer/smoldataenvs-grpo-spark 📚 Resources SmolDataEnvs (the RL environments): https://huggingface.co/datasets/FineEnvs/SmolDataEnvs FineEnvs: https://github.com/adithya-s-k/FineEnvs Qwen3.5-4B: https://huggingface.co/Qwen/Qwen3.5-4B Qwen3.5-27B: https://huggingface.co/Qwen/Qwen3.5-27B TRL GRPO Trainer: https://huggingface.co/docs/trl/grpo_trainer NVIDIA DGX Spark: https://www.nvidia.com/en-us/products/workstations/dgx-spark/ Xiaomi MiMo-V2.6 (open RL environments): https://mimo.mi.com/docs/en-US/news/latest/v2-6 TechCrunch on RL environments: https://techcrunch.com/2025/09/21/silicon-valley-bets-big-on-environments-to-train-ai-agents/ Website: https://engineerprompt.ai/ RAG Beyond Basics Course: https://prompt-s-site.thinkific.com/courses/rag Let's Connect: 🦾 Discord: https://discord.com/invite/t4eYQRUcXB ☕ Buy me a Coffee: https://ko-fi.com/promptengineering |🔴 Patreon: https://www.patreon.com/PromptEngineering 💼Consulting: https://calendly.com/engineerprompt/consulting-call 📧 Business Contact: engineerprompt@gmail.com Become Member: http://tinyurl.com/y5h28s6h 💻 Pre-configured localGPT VM: https://bit.ly/localGPT (use Code: PromptEngineering for 50% off). Signup for Newsletter, localgpt: https://tally.so/r/3y9bb0 00:00 Can RL Shrink the Gap 01:24 What RL Environments Are 02:45 Reward Hacking and Checkers 03:37 Where RL Fits in Training 04:32 GRPO and Learning Signal 05:17 Hardware Setup and Baselines 06:18 Sponsor Firecrawl 07:51 Training on Edge Tasks 09:39 Before and After Runs 11:37 Why Progress Stalls 12:38 Distillation from 27B 13:51 Best Combo Model Example 14:52 Takeaways and Wrap Up
