TurboQuant: Google Redefines AI Efficiency with Extreme Compression
Google Research introduces a set of advanced, theoretically grounded quantization algorithms that enable massive compression for large language models and vector search engines — with zero accuracy loss.
Vectors are the fundamental way AI models understand and process information. Small vectors describe simple attributes, while "high-dimensional" vectors capture complex information such as the features of an image, the meaning of a word, or the properties of a dataset. High-dimensional vectors are incredibly powerful, but they also consume vast amounts of memory, leading to bottlenecks in the key-value (KV) cache — a high-speed digital reference store that allows a model to retrieve frequently used computations instantly without re-processing from scratch.
Vector quantization is a powerful, classical data compression technique that reduces the size of high-dimensional vectors. However, traditional methods usually introduce their own "memory overhead" as most require calculating and storing quantization constants for every small block of data in full precision — adding 1–2 extra bits per number and partially defeating the purpose of quantization.
Today, we introduce TurboQuant (to be presented at ICLR 2026), a compression algorithm that optimally addresses the challenge of memory overhead in vector quantization. We also present QJL (Quantized Johnson-Lindenstrauss) and PolarQuant (to be presented at AISTATS 2026), which TurboQuant uses to achieve its results.
TurboQuant overview — Google Research
How TurboQuant Works
TurboQuant achieves high compression with zero accuracy loss, making it ideal for supporting both key-value (KV) cache compression and vector search. It accomplishes this via two key steps:
The Two-Step Compression Pipeline
- High-quality compression (PolarQuant): TurboQuant starts by randomly rotating the data vectors. This simplifies the data's geometry, making it easy to apply a high-quality quantizer to each part of the vector individually. This first stage uses most of the compression power (the majority of bits) to capture the main concept and strength of the original vector.
- Eliminating hidden errors (QJL): TurboQuant uses a small, residual amount of compression power — just 1 bit — to apply the QJL algorithm to the tiny error left over from the first stage. The QJL stage acts as a mathematical error-checker that eliminates bias in the attention score, leading to greater accuracy.
QJL: The Zero-Overhead, 1-Bit Trick
QJL uses the Johnson-Lindenstrauss Transform to shrink complex, high-dimensional data while preserving the essential distances and relationships between data points. It reduces each resulting vector number to a single sign bit (+1 or -1). This algorithm creates a high-speed shorthand that requires zero memory overhead.
To maintain accuracy, QJL uses a special estimator that strategically balances a high-precision query with the low-precision, simplified data. This allows the model to accurately calculate the attention score — the process used to decide which parts of its input are important and which parts can be safely ignored.
PolarQuant: A New "Angle" on Compression
PolarQuant addresses the memory overhead problem with a completely different approach. Instead of representing a memory vector using standard Cartesian coordinates (X, Y, Z axis distances), PolarQuant converts the vector into polar coordinates. This is comparable to replacing "Go 3 blocks East, 4 blocks North" with "Go 5 blocks total at a 37-degree angle."
This results in two pieces of information: the radius (how strong the core data is) and the angle (the data's direction or meaning). Because the pattern of the angles is known and highly concentrated, the model no longer needs the expensive data normalization step — it maps data onto a fixed, predictable "circular" grid where boundaries are already known, rather than a "square" grid where boundaries change constantly.
PolarQuant Mechanism
- Groups pairs of coordinates from a d-dimensional vector
- Maps them onto a polar coordinate system
- Gathers radii in pairs for recursive polar transformations
- Repeats until distilled into a single final radius and a collection of descriptive angles
- Result: zero memory overhead, eliminating the 1–2 extra bits required by traditional methods
Experiments and Results
All three algorithms were rigorously evaluated across standard long-context benchmarks including LongBench, Needle In A Haystack, ZeroSCROLLS, RULER, and L-Eval using open-source LLMs (Gemma and Mistral). The data demonstrate that TurboQuant achieves optimal scoring performance in terms of both dot product distortion and recall while simultaneously minimising the KV memory footprint.
TurboQuant proved it can quantize the key-value cache to just 3 bits without requiring training or fine-tuning, and without any compromise in model accuracy — all while achieving a faster runtime than the original LLMs. It is exceptionally efficient to implement and incurs negligible runtime overhead.
For long-context "needle-in-a-haystack" tasks — tests designed to see if a model can find a single specific piece of information buried in massive amounts of text — TurboQuant achieves perfect downstream results across all benchmarks while reducing KV memory size by a factor of at least 6x. PolarQuant is also nearly lossless for this task.
Key Performance Metrics at a Glance
- 6x+ KV cache memory reduction at 3-bit quantization, zero accuracy loss
- 8x attention logit speedup at 4-bit vs. 32-bit unquantized on H100 GPUs
- No training or fine-tuning required — works with existing models
- Near-zero preprocessing time — data-oblivious operation
- State-of-the-art recall in high-dimensional vector search (GloVe d=200)
- Validated on Gemma, Mistral, and Llama-3.1-8B-Instruct
Vector Search: Superior Recall at Minimal Memory
TurboQuant was also evaluated for high-dimensional vector search against state-of-the-art methods (PQ and RaBitQ) using the 1@k recall ratio, which measures how frequently the algorithm captures the true top inner product result within its top-k approximations.
TurboQuant consistently achieves superior recall ratios compared to baseline methods — despite those baselines utilising inefficient large codebooks and dataset-specific tuning. This confirms TurboQuant's robustness and efficiency for high-dimensional search tasks.
Looking Ahead: Implications for Search and LLMs
TurboQuant, QJL, and PolarQuant are more than practical engineering solutions — they are fundamental algorithmic contributions backed by strong theoretical proofs. These methods don't just work well in real-world applications; they are provably efficient and operate near theoretical lower bounds. This rigorous foundation is what makes them robust and trustworthy for critical, large-scale systems.
While a major application is solving the key-value cache bottleneck in models like Gemini, the impact extends further. Modern search is evolving beyond keywords to understand intent and meaning — requiring vector search at the scale of billions of data points. Techniques like TurboQuant are critical for this mission: they enable building and querying large vector indices with minimal memory, near-zero preprocessing time, and state-of-the-art accuracy.
As AI becomes more integrated into all products — from LLMs to semantic search — this work in fundamental vector quantization will be more critical than ever.
Key Takeaways
- TurboQuant combines PolarQuant and QJL to compress KV caches by 6x+ at just 3 bits with no accuracy loss
- QJL uses the Johnson-Lindenstrauss Transform to reduce each vector element to a single sign bit (+1/−1) with zero memory overhead
- PolarQuant eliminates quantization constants by converting to polar coordinates, removing 1–2 bits of traditional overhead
- 4-bit TurboQuant delivers 8x speedup over unquantized 32-bit inference on H100 GPUs
- No fine-tuning needed — works with Gemma, Mistral, Llama and any standard LLM
- Methods are theoretically grounded and operate near proven lower bounds — not just empirically effective
- Directly addresses Gemini's KV cache bottleneck and enables faster semantic search at Google scale
Frequently Asked Questions
What is TurboQuant?
TurboQuant is a compression algorithm developed by Google Research that reduces the memory footprint of large language model KV caches by at least 6x using vector quantization, with zero accuracy loss. It combines PolarQuant for high-quality compression and QJL for bias elimination, and will be presented at ICLR 2026.
How much faster is TurboQuant compared to standard inference?
4-bit TurboQuant achieves up to 8x performance increase over 32-bit unquantized keys on H100 GPU accelerators when computing attention logits in the KV cache, measured relative to a highly optimised JAX baseline.
What is PolarQuant and how does it eliminate memory overhead?
PolarQuant converts vector data from Cartesian to polar coordinates. Because the angular distribution of transformed data is highly concentrated and predictable, it eliminates the need to store per-block quantization constants — which traditionally add 1–2 extra bits per number. This removes the hidden overhead that undermines traditional quantization methods.
Does TurboQuant require model fine-tuning or retraining?
No. TurboQuant can quantize the KV cache to just 3 bits without requiring any training or fine-tuning, while achieving zero compromise in model accuracy. It works as a post-hoc compression layer on existing LLMs including Gemma, Mistral, and Llama.
How does TurboQuant compare to existing vector search methods?
In direct comparisons against Product Quantization (PQ) and RaBitQ on the GloVe dataset, TurboQuant consistently achieves superior 1@k recall ratios. It does this despite those baselines using large, dataset-specific codebooks — demonstrating that TurboQuant's data-oblivious approach is both more efficient and more accurate.
Source: Google Research Blog | Authors: Amir Zandieh (Research Scientist) & Vahab Mirrokni (VP and Google Fellow), Google Research. Research conducted in collaboration with Praneeth Kacham, Majid Hadian (Google DeepMind), Insu Han (KAIST), Majid Daliri (NYU), Lars Gottesbüren, and Rajesh Jayaram (Google Research).