NVIDIA's Secret 2 Bit Model
In this video, I look at a new model and a set of techniques that NVIDIA has used to make this 2-bit model perform at a level much better than expected. Research Report: https://research.nvidia.com/labs/nemotron/Nemotron-3.5-Bolt-24GB/ HF: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB ParetoQ: https://arxiv.org/pdf/2502.02631 2nd Channel: https://www.youtube.com/channel/UCUF7WtR4JpVbQqcQGrLeADw Twitter: https://x.com/Sam_Witteveen 🕵️ Interested in building LLM Agents? Fill out the form below Building LLM Agents Form: https://drp.li/dIMes 👨💻Github: https://github.com/samwit/llm-tutorials ⏱️Time Stamps: 0:00 Intro 0:14 Microsoft’s secret 2-bit model 1:02 Nemotron 3.5 Bolt 74B-A9B 1:54 Why the recipe matters 3:00 How normal quantization works 3:43 Why going below 4-bit breaks 4:38 ParetoQ 5:31 Why 2-bit matters 7:04 Mixed precision 8:02 Stage 1: Pruning 9:23 Stage 2: Post-training 9:40 Stage 3: Quantization-aware training 11:48 NVIDIA’s twist: QAD 14:29 Demo on the ZGX Fury 16:00 Pelican SVG test 17:55 Agentic tool-calling tests 19:42 Concurrency tests 20:47 Which hardware Bolt is for 22:04 Conclusion
