How This Tiny $8 Chip Runs an LLM With Almost No RAM
Machine learning meets an $8 ESP32 LLM: how per-layer embeddings run a 28.9M-parameter model in 512KB of RAM Machine learning on a microcontroller sounds impossible until you see the trick: an $8 ESP32-S3 board running a 28.9-million-parameter large language model with no cloud, no server, and half a megabyte of working memory. This video breaks down slvDev's open-source esp32-ai project, which borrows Per-Layer Embeddings, the memory technique Google built for its Gemma models, to keep a 25-million-parameter word-lookup table out of active RAM entirely, reading just a few hundred bytes from flash storage per generated word instead. The compute core runs on Andrej Karpathy's llama2.c, a full Llama 2 inference engine written in one file of pure C, and generates TinyStories-style children's fiction at roughly 9-9.5 tokens per second, over 100x larger than the previous ESP32 record of about 260,000 parameters. We walk through the actual arithmetic: 4-bit quantization shrinking the model to 14.9MB, the three separate memory tiers on the board (512KB SRAM, 8MB PSRAM, 16MB flash), and why TinyStories' stripped-down vocabulary is just as responsible for making this work as the memory engineering is. We also get honest about the limits, this thing can't answer a question, and fitting the same trick into a real shipping product means competing for that same 512KB with a wifi stack, sensor code, and OTA updates. This is for builders, embedded tinkerers, and anyone curious about edge AI, deep learning on constrained hardware, and where large language models actually can and can't go once you take away the server. Chapters: 0:00 The $8 chip that shouldn't write fiction 0:33 One chip, three hidden memories 1:36 100x bigger than the old record 2:35 The math that shouldn't add up 3:29 Most of the model is a phone book 4:47 The slowdown that never happened 5:45 Stolen straight from your phone 6:35 An entire AI in one C file 7:34 Why the model's job got shrunk too 8:52 What your $8 actually buys you 9:48 The memory was never really free 11:08 The real wall nobody expected Tools & resources mentioned: - esp32-ai (slvDev): https://www.reddit.com/r/esp32/comments/1v4mq5t/how_i_fit_a_289m_llm_on_an_esp32s3_9_toks_fully - llama2.c (Andrej Karpathy): https://github.com/karpathy/llama2.c - TinyStories dataset overview: https://www.emergentmind.com/topics/tinystories-dataset - CircuitDigest coverage: https://circuitdigest.com/news/esp32-s3-runs-a-289-million-parameter-llm-borrowing-a-memory-trick-from-googles-gemma - Edge AI state-of-the-art analysis: https://needcode.io/edge-ai-wireless-integration-companies-2026 - TinyStories research talk (Harvard CMSA): https://www.math.harvard.edu/event/the-tinystories-dataset-how-small-can-language-models-be-and-still-speak-coherent-english About The Stack The Stack helps you build with AI. Each video takes one tool, model, or workflow and shows how it works in a few focused minutes, with the real benchmarks and real costs. We go deep on Claude Code and Cursor for AI coding, AI agents and MCP servers, the open-source AI tools and GitHub repos most people miss, RAG and vector search, fine-tuning, and running local LLMs on your own machine with Ollama and LM Studio. We compare models like ChatGPT and Claude, test AI automation with Zapier, Make, and n8n, and flag the tools that actually ship. Subscribe for new breakdowns: https://www.youtube.com/@the-stack-ai?sub_confirmation=1 #esp32 #edgeai #machinelearning #llm #embeddedai
