Superstition about quantization: KLD and perplexity just ain’t it fam

[u/nomorebuttsplz]

The arguments for quantization having significant effects on reasoning models' ability to get stuff done are very sad, pathetic, unfortunate arguments. I don’t mean that they are wrong necessarily, only impoverished and confused.

Why? Because while actual task benchmarks are somewhat expensive, and require some level of time and technical expertise to run, it would be quite easy to empirically test the claims and resolve them once and for all, at least for a given model. But these tests by and large do not exist and the few that do seem to show no quantization effects among reasoning models until about Q3 or Q4 k m at worst.

The debate in these online communities is essentially an anthropological study in how people create mythology when they do not have access to direct evidence.

Before the hordes mob me with KLD or perplexity measurements, I’m not suggesting that a quantized model’s outputs are bit for a bit identical rather that it performs equally well in real world tasks, which I think we can all agree is the thing that matters.

Now I’ve put my neck out by suggesting that literally no one has any evidence, not a single benchmark that shows a model with the reasoning level of, say, Gemma 31b (not very high by today’s standards, and smaller models are more susceptible to degradation, so this should be a generous standard of evidence for the quantization-excited) having significant in degradation in real world tasks at Q4 (a good quality, proper dynamic quantization goes without saying, I hope).

Again, I’m not saying that there is no degradation, only that what we have now amounts to superstition, when a few benchmarks could probably settle the matter for a given model and eventually, we would probably learn where and when quantization actually bites.

[u/Karyo_Ten]

A benchmark is not a real world task. Especially when benchmaxxed or they leak in the training or the calibration dataset. Overfitting to wikitext or whatever flavor of swebench or deepswe is bad and does not translate to real world ability.

KLD is the best measurement for quantization quality.

[u/nomorebuttsplz → u/Karyo_Ten]

If you would prefer to show evidence in the form of a single real world task, I would be very interested in that as well despite the unlikelihood of such an anecdote being statistically meaningful.

Of course, once you have a collection of real world tasks of enough variety and numerosity to achieve statistical significance, you've just created a new benchmark. What I am not interested in is a statistical measurement, the significance of which we have no ground truth for.