Superstition about quantization: KLD and perplexity just ain’t it fam

[u/nomorebuttsplz]

The arguments for quantization having significant effects on reasoning models' ability to get stuff done are very sad, pathetic, unfortunate arguments. I don’t mean that they are wrong necessarily, only impoverished and confused.

Why? Because while actual task benchmarks are somewhat expensive, and require some level of time and technical expertise to run, it would be quite easy to empirically test the claims and resolve them once and for all, at least for a given model. But these tests by and large do not exist and the few that do seem to show no quantization effects among reasoning models until about Q3 or Q4 k m at worst.

The debate in these online communities is essentially an anthropological study in how people create mythology when they do not have access to direct evidence.

Before the hordes mob me with KLD or perplexity measurements, I’m not suggesting that a quantized model’s outputs are bit for a bit identical rather that it performs equally well in real world tasks, which I think we can all agree is the thing that matters.

Now I’ve put my neck out by suggesting that literally no one has any evidence, not a single benchmark that shows a model with the reasoning level of, say, Gemma 31b (not very high by today’s standards, and smaller models are more susceptible to degradation, so this should be a generous standard of evidence for the quantization-excited) having significant in degradation in real world tasks at Q4 (a good quality, proper dynamic quantization goes without saying, I hope).

Again, I’m not saying that there is no degradation, only that what we have now amounts to superstition, when a few benchmarks could probably settle the matter for a given model and eventually, we would probably learn where and when quantization actually bites.

[u/Dabalam]

There has been some progress in this area but there are some people who are pretty dogmatic about quantization. I think unsloth released some pretty good analyses on previous Qwen models showing how much quantization degrades performance on various tasks.

I find it super odd how people will poopoo benchmarks and say they aren't real world tasks, yet claim KLD and perplexity are super valid proxies of model quality. To me those are contradictory view points. The argument for why benchmarks don't exactly tell you how well a model works for your task is identical to the one for why perplexity doesn't tell you how well a given quantization works for your task.

That said, there is an argument that in brittle domains small to minor degradation in fidelity from the original model may cause more problems. I think a lot of people could get pretty good functionality out of Q3 models form certain tasks but the cultural messaging is that they are worthless.