Google TurboQuant Compresses LLM Cache to 3 Bits with 8x Speedup and Zero Accuracy Loss

Google published TurboQuant, a training-free algorithm that compresses LLM key-value cache memory to 3 bits using a polar coordinate technique called PolarQuant, delivering 6x memory reduction and up to 8x inference speedup on NVIDIA H100 GPUs with zero accuracy loss. The algorithm requires no fine-tuning and has negligible runtime overhead. Memory and storage stocks dropped on the announcement. The paper will be presented at ICLR 2026. The practical impact is immediate for organizations running inference at scale, as it enables larger batch sizes on existing hardware without quality degradation.