This 27B Model Shouldn't Run On Your Phone. It Does.
Running a 27B Model on iPhone: Prism ML’s 1‑Bit Bonsai Quantization Explained I break down how Prism ML ran a 27B parameter Qwen-based model on an iPhone 17 Pro at about 11 tokens/sec by compressing it from ~54GB to 3.9GB using their 1-bit “Bonsai” approach. I explain quantization basics (16-bit to 8/4-bit, ternary, and 1-bit), why naive post-training quantization fails due to compounding error, and how prior open-source methods (llama.cpp mixed precision, GPTQ calibration-aware, AWQ activation-aware) work well down to ~4-bit but collapse at 2/1-bit. Then I cover quantization-aware training inspired by BitNet, end-to-end 1-bit/ternary variants (including the LM head), multimodal support with a 4-bit vision tower, a 262K context window, and speculative decoding (DeepSeek DGX Spark drafter) for latency. I also discuss benchmarks, intelligence density per GB, tool-calling weaknesses, and my own 1-bit tests on an M2 Max showing ~29–30 tok/s plus looping on harder tasks. LINKS: Blog: https://prismml.com/news/bonsai-27b Whitepaper: https://github.com/PrismML-Eng/Bonsai-demo/blob/main/bonsai-27b-whitepaper.pdf Huggingface: https://huggingface.co/collections/prism-ml/bonsai-27b Github: https://github.com/PrismML-Eng/Bonsai-demo/ Demo: https://huggingface.co/spaces/webml-community/bonsai-webgpu-kernels https://youtu.be/eFgknPFK-g0 My voice to text App: whryte.com Website: https://engineerprompt.ai/ RAG Beyond Basics Course: https://prompt-s-site.thinkific.com/courses/rag Signup for Newsletter, localgpt: https://tally.so/r/3y9bb0 Let's Connect: 🦾 Discord: https://discord.com/invite/t4eYQRUcXB ☕ Buy me a Coffee: https://ko-fi.com/promptengineering |🔴 Patreon: https://www.patreon.com/PromptEngineering 💼Consulting: https://calendly.com/engineerprompt/consulting-call 📧 Business Contact: engineerprompt@gmail.com Become Member: http://tinyurl.com/y5h28s6h 💻 Pre-configured localGPT VM: https://bit.ly/localGPT (use Code: PromptEngineering for 50% off). Signup for Newsletter, localgpt: https://tally.so/r/3y9bb0 00:00 27B Model on iPhone 00:27 Bonsai Compression Idea 01:32 Quantization Limits Explained 03:52 From PTQ to QAT 06:11 Bonsai End to End Setup 08:36 Speed and Benchmarks 10:55 Bonsai local test
