Run AI Faster With Lossless Compressed Memory

Lossless KV cache compression cuts memory without losing data, ZipFusion shows how to speed AI decoding by fusing decompression into attention itself. ZipFusion is a new approach to lossless KV cache compression that solves the central problem with older compressors: they unpacked data inside the generation loop, slowing decoding down. The paper, posted to arXiv on October twenty-fourth by Yuetao Chen, Peng Wang and Hong Xu, shows that by keeping the key-value cache compressed in GPU memory and rebuilding values in registers during attention using a single integer addition, you can avoid the unpacking penalty entirely. Running Llama three point one eight-B at batch size thirty-two, ZipFusion delivered about fifteen percent faster attention and roughly twenty-six percent more overall throughput, though most of that gain comes from fitting nearly a quarter more concurrent conversations onto the same H one hundred, not each conversation finishing faster. The compression works by exploiting redundancy in BF16 exponents: instead of storing eight bits per exponent per value, ZipFusion records one shared base exponent per block and stores only a four-bit offset for each value, shrinking each eligible block to twelve bits, seventy-five percent of the original size. Blocks with exponents too spread out stay uncompressed. This matters because memory bandwidth, not compute, limits generation speed during decoding, so fewer bytes read from GPU memory at every step translates directly to faster generation. The preprint leaves critical gaps: no public code, testing only on H one hundred across batches of sixteen or larger, and excluded prefill and initial compression from timing. FP8 quantization already cuts cache memory in half and delivers similar throughput gains today in vLLM, but it is lossy, ZipFusion keeps every bit identical. For builders running high-precision tasks where cached values cannot drift, or serving many concurrent requests on data-center GPUs, this is the architecture to watch; for local single-batch inference or early-stage testing, FP8 is available now and worth testing on your own workload. Chapters: 0:00 Intro 1:26 Why Memory Sets the Speed 2:31 Twelve Bits and One Addition 4:57 More Conversations per GPU 5:54 What the 26 Percent Measures 7:42 Lossless or FP8 9:38 Verdict Tools & resources mentioned: - ZipFusion: https://arxiv.org/html/2610.33719v1 - vLLM: https://vllm.ai/ - SGLang: https://github.com/sgl-project/sglang - FlashInfer: https://flashinfer.ai/ About The Stack The Stack helps you build with AI. Each video takes one tool, model, or workflow and shows how it works in a few focused minutes, with the real benchmarks and real costs. We go deep on Claude Code and Cursor for AI coding, AI agents and MCP servers, the open-source AI tools and GitHub repos most people miss, RAG and vector search, fine-tuning, and running local LLMs on your own machine with Ollama and LM Studio. We compare models like ChatGPT and Claude, test AI automation with Zapier, Make, and n8n, and flag the tools that actually ship. Subscribe for new breakdowns: https://www.youtube.com/@the-stack-ai?sub_confirmation=1 #kv-cache #llm-optimization #ai-inference