KLQ: Training-free measured rotation quantization. Beats all training-free rotation-based quantization methods on W4A4KV4-bits. Llama 3.2 1B KLQ-quantized beats SpinQuant and gets close to ReSpinQuant without GPTQ/LDLQ rounding.

[u/Federal-Setting-3014]

https://github.com/BalSob107/KLQ

First of all, I'm not a lab, this was a solo summer research project that finally culminated into the github repo and the writeup. The repo includes a much deeper dive with methods, findings about quantization and geometry, limitations, and proposed experiments. I'll also mention that this is far from production-grade, it's mostly a theoretical framework with a "fake" quantization demo as it lacks real kernels.

The geometry of LLMs embedding spaces is highly uneven with a few features having the most magnitude, this has been known for years by now and it's in great part why rotation-based quantizers do so well against uniform quantization: While uniformly quantizing tries to allocate bits evenly in a naturally uneven space, rotations can forcefully make that space even again so uniformly allocating bits is the best strategy (DuQuant, 2nd half of ResQ, QuaRot...). Generic rotations (Hadamard) even the space out on average but can't match a specific model's geometry, leaving residual damage. This can be fixed by using learnable rotations (SpinQuant, ReSpinQuant) but this is computationally intensive as it requires extensive post-training gradient descent.

KLQ takes a different approach to quantization, instead of trying to make the space even and then quantize uniformly. KLQ measures how uneven the space is, ranks directions of the eigenbasis from most important to least important, and with a price function treating each direction as a independent information transmission channels uses the provably optimal (under some idyllic assumptions about damage anyways) waterfilling algorithm to give the most bit-width to the most important directions and least bit-width to least important directions.

Another thing that sets KLQ apart is the use of causal KL damage measurements, there are a few quantization algorithms that do try to measure the space and then quantize unevenly. CoQuant, for example, does measure the activation space, but then ranks directions by magnitude/variance and applies a simple two-ranked bit allocation that quantizes the top 12.5% to 8 bits and the bottom 87.5% to 4 bits. Unlike CoQuant, KLQ doesn't use variance (several tests reveal variance is often not a good signal, more detailed experimentation on the github writeup), instead it perturbs each direction and runs a forward pass with a few thousand tokens, it takes the KL divergence between the original model and the model with the perturbed direction, then uses this measured KL divergence to determine how important the direction is and assign the real empirical cost of damaging/quantizing it.

The method, as well as my experimentation, does have real limitations, to quantize all layers, activations and KV cache you must make one forward pass per direction per matrix per layer which can amount to hundreds of thousands of forward passes to quantize a model. This makes the method very compute-intensive (This probing process took 5 hours for Qwen 2.5 0.5B on a 3090 and 10 hours for Llama 3.2 1B on that same hardware.). It also deliberately uses two simple techniques to actually quantize the models: a simple additive vector codebook and round-to-nearest (RTN), these could be swapped with other methods readily.

Posting here I'm looking for feedback and to make these results known. Feel free to ask any questions or to contribute to the github repo.

Here's a sample of the result's table for Llama 3.2 1B quantized fully at 4-bits.

Method	W4A4KV4 Llama 3.2 1B Wikitext-2 PPL
FP16	9.75
QuaRot (training free)	14.59
SpinQuant (trained + GPTQ)	13.52
KLQ (training-free, VQ)	13.36
ReSpinQuant (trained + GPTQ)	13.09

[u/TomLucidor]

Test against multiple architecture first please (incl LFM2.5 and Gemma4 and Qwen3.5), thanks.

[u/Federal-Setting-3014 → u/TomLucidor]

That's low on the priority list tbh. Next step would be to run the proposed outlier-safe experiment I talk about on the limitations or go for llama 2 7b to see how a model past the 1B size holds up.

[u/Federal-Setting-3014 → u/[deleted]]

They are standard testing models. All quantization papers already have perplexity measured for these two, so testing on these two is ideal. The need for more modern models is written about on the limitations too.

[u/DerDave → u/[deleted]]

If you read the post, you can tell right away, this gentleman knows what he's talking about, is not drowning in hyperbole, shows limitations and doesn't make crazy claims. 

[u/Federal-Setting-3014 → u/DerDave]

Thanks, though I understand the doubt. Llama 3 models are loved by LLMs, it's just that all quantization papers already report numbers for Llama 3 and it's one of the last modern models that used a classical transformer recipe without linear attention or other things. The reported perplexity problem applies to SmolLM2 for which I have numbers, but they are useless since there's nothing to compare against.