How DeepSeek Serves Millions Without Going Bankrupt on GPUs
DeepSeek just open-sourced DeepInfer, the inference engine behind its own API, along with a detailed write-up of how it serves DeepSeek V4.1 Flash at scale: 155 trillion tokens a day from a single cluster. In this video we follow one request through the system: deepseek-recipe tokenization and prompt-injection protection, a KV cache that DeepSeek keeps for days (96% of input tokens are cache hits), prefill and decode on different machines, expert parallelism across 64+ GPUs, DSpark speculative decoding, and batch-invariant inference. Then I test deepseek-recipe and the temperature-0 batching effect on my own Mac, and explain why DeepInfer won't run on a DGX Spark. DeepInfer (GitHub): https://github.com/deepseek-ai/DeepInfer deepseek-recipe (GitHub): https://github.com/deepseek-ai/deepseek-recipe DeepSeek's deployment write-up (Chinese): https://zhuanlan.zhihu.com/p/2095560295930508953 English translation (Latent Space): https://x.com/latentspacepod/status/2112588380416647205 DeepSeek's announcement: https://x.com/deepseek_ai/status/2112476496283775135 DeepSeek V4.1 Flash 1017 weights: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017 DeepSeek API pricing (cache hit vs miss): https://api-docs.deepseek.com/quick_start/pricing Related videos: Speculative Decoding Explained: DeepSeek DSpark: https://www.youtube.com/watch?v=eFgknPFK-g0 Run a 284B Model in 128 GB (DwarfStar): https://www.youtube.com/watch?v=9gHcmhUDJfw Chapters 00:00 DeepInfra 01:21 Prefill vs Decode Basics 03:07 Request Flow and Token Safety 04:31 KV Cache and Prefill Tricks 07:31 Decode at Scale MoE and Speculation 09:17 Batch Invariance and Real Limits 12:31 Hands On Tests and Key Takeaways
