The Local AI Engine You Should Be Using (FreeToken)
FreeToken runs 284B models on gaming PCs via mixture-of-experts routing, an open-source local AI engine faster than llama.cpp and Ollama, free, Linux NVIDIA-only. FreeToken is a free Apache-2.0 local AI engine that serves massive mixture-of-experts models on consumer NVIDIA GPUs by streaming only the active experts into VRAM while parking the full model in system RAM. The trick: models like DeepSeek-V4-Flash (284B parameters) and GLM-5.2 (753B) only activate a tiny fraction of their weights per token, so FreeToken keeps dormant experts in cheap DDR5 and swaps working experts across the PCIe bus word by word. On an RTX 4060 laptop with 8GB VRAM and 32GB system RAM, it hits 39.3 tokens/sec on a 35B model; an RTX 5090 desktop serves DeepSeek-V4-Flash at 22, 25 tokens/sec; one independent test on a ThinkPad A5500 saw 60 tokens/sec. Against llama.cpp, Ollama, KTransformers and MoE-Infinity, FreeToken claims 1.3x, 2.3x higher throughput and sub-44-second time-to-first-token. Built by UC Berkeley and UT Austin researchers (Ion Stoica, Matei Zaharia, Song Han and others behind Spark and vLLM), it shipped v0.1.2 in August 2026 and auto-tunes via bandwidth measurement. The ceiling is real: 32GB RAM holds 35B, 192GB holds 284B, 512GB holds 753B, you're swapping a VRAM wall for a system-RAM ladder. NVIDIA x86-Linux only; no Mac, no AMD. Dense models gain nothing. For builders running one-user inference on an RTX card with 64, 512GB RAM, this cuts setup from weekends of tuning to three commands. Chapters: 0:00 290B on your gaming PC 0:30 Why your GPU hits a wall 1:23 CPU fallback: the speed cliff 2:28 Routers wake only what's needed 4:00 39 tokens on 8GB laptop 5:00 284B moves at gaming speed 5:56 The Spark and vLLM team built this 6:55 Brand new, already 8.7k stars 7:48 2x faster than llama.cpp and Ollama 8:57 One author's benchmarks so far 10:10 The forum test that beat the paper 11:24 Three commands, zero tuning required 12:35 RAM is the real cost now 13:52 NVIDIA Linux only, no Mac support 15:17 753B needs 512GB, and the actual hardware 16:29 Should you run it tonight? Tools & resources mentioned: - FreeToken: https://github.com/FlashML-org/FreeToken - FreeToken arXiv paper: https://arxiv.org/html/2608.16157v1 - llama.cpp - Ollama - vLLM - KTransformers - MoE-Infinity About The Stack The Stack helps you build with AI. Each video takes one tool, model, or workflow and shows how it works in a few focused minutes, with the real benchmarks and real costs. We go deep on Claude Code and Cursor for AI coding, AI agents and MCP servers, the open-source AI tools and GitHub repos most people miss, RAG and vector search, fine-tuning, and running local LLMs on your own machine with Ollama and LM Studio. We compare models like ChatGPT and Claude, test AI automation with Zapier, Make, and n8n, and flag the tools that actually ship. Subscribe for new breakdowns: https://www.youtube.com/@the-stack-ai?sub_confirmation=1 #lm studio vs ollama #local AI #NVIDIA inference
