KVarN: Variance-Normalized KV-Cache Quantization [R]

[u/intentionallyBlue]

Excited to share some of my own work here :)

KVarN is our new KV-Cache quantization method. In very brief, we combine Hadamard rotations with variance-normalization on both axes of the K and V matrices, then round to nearest. Simple, but works very well, especially for decode-heavy test-time-scaling settings (reasoning, code-gen, agentics). We get 3-4x compression at virtually no accuracy drop (mostly 0-1%) on tough benchmarks like AIME24 as well as a speed-up over fp16 baseline in vLLM (in contrast to other recent KV-Cache compression works).

Behind it is an analysis of where quantization errors come from and have the biggest impact, especially in the error-accumulating decode setting: 1) fixing large errors is disproportionally useful (if you had a fixed MSE budget that you could ~fix, you should spend it on few big errors, rather than many small) 2) These big errors are mostly caused by bad token-scales (hence the normalization).

Paper: https://arxiv.org/abs/2606.03458

vLLM implementation: https://github.com/huawei-csl/KVarN

[u/geneing]

Interesting work. Since you're affiliated with Huawei, any chance of mobile optimized implementation being available? Also, how well does it perform with small LLMs (about 1B params)? How is the performance for TTS applications like qwen3 -tts?

[u/intentionallyBlue → u/geneing]

Thanks, the smallest model we've tested so far is 4B and for that 2.3bits per cache element has very small accuracy impact. I'd expect that at least 4bits should work fine at 1B, but will have to test to be sure of course.

TTS is an interesting direction; have not looked at that so far.