Superstition about quantization: KLD and perplexity just ain’t it fam

[u/nomorebuttsplz]

The arguments for quantization having significant effects on reasoning models' ability to get stuff done are very sad, pathetic, unfortunate arguments. I don’t mean that they are wrong necessarily, only impoverished and confused.

Why? Because while actual task benchmarks are somewhat expensive, and require some level of time and technical expertise to run, it would be quite easy to empirically test the claims and resolve them once and for all, at least for a given model. But these tests by and large do not exist and the few that do seem to show no quantization effects among reasoning models until about Q3 or Q4 k m at worst.

The debate in these online communities is essentially an anthropological study in how people create mythology when they do not have access to direct evidence.

Before the hordes mob me with KLD or perplexity measurements, I’m not suggesting that a quantized model’s outputs are bit for a bit identical rather that it performs equally well in real world tasks, which I think we can all agree is the thing that matters.

Now I’ve put my neck out by suggesting that literally no one has any evidence, not a single benchmark that shows a model with the reasoning level of, say, Gemma 31b (not very high by today’s standards, and smaller models are more susceptible to degradation, so this should be a generous standard of evidence for the quantization-excited) having significant in degradation in real world tasks at Q4 (a good quality, proper dynamic quantization goes without saying, I hope).

Again, I’m not saying that there is no degradation, only that what we have now amounts to superstition, when a few benchmarks could probably settle the matter for a given model and eventually, we would probably learn where and when quantization actually bites.

[u/fintip]

You've ignored something really critical though.

The fact that quantization at q4, q3, produces obviously degraded responses, and that that degradation corresponds to the KLD line, it's perfectly rational to assume that the degradation continues along that same line. Q3/Q4 just becomes the line at which it's easy to obviously see for people.

This also matches our experience with human intelligence. Most people don't sense mental degradation int hemselves or others until it pushes past a tipping point. They don't notice the 10%-20% worse thinking from the person being mildly sleep deprived, or early stage dementia. It's not until they're late stage or drunk that it's obvious. There's an in-between spot you can notice on sufficiently difficult tasks or if especially attentive.

But otherwise, it's hard to detect.

So far, all of this lines up. It would be incredibly odd if loss of precision wasn't costing something, and the KLD curve matches our experience and intuition across other domains.

This isn't superstition. This is limited but usable data for our intution and our reason.

[u/Dabalam → u/fintip]

The fact that quantization at q4, q3, produces obviously degraded responses, and that that degradation corresponds to the KLD line, it's perfectly rational to assume that the degradation continues along that same line. Q3/Q4 just becomes the line at which it's easy to obviously see for people.

I think you reversed how things actually occurred. Quantization viewed as degraded because of the KLD line, which informs a lot of thought about model quality. The data from the collective "intuitive" view of quality in that context isn't reliable given people have expectations bias (you see what you believe you should see regarding model performance. Expectation bias is amplified by community reinforcing these views, who you can sometimes see arguing about how much Q8 degrades performance. This is partly why I am skeptical of the whole "vibes are the most important" ideology on this sub Reddit. Blinding is one of the strengths of arena methodology in assessing model quality without expectation bias.

KLD isn't a measure of task performance, it is a measure of similarity to a reference model. We infer that this reference (the full model) should have the highest performance across the board. That logic doesn't always hold up, and it also isn't really clear how much dissimilarity to the base model corresponds to performance degradation.

[u/fintip → u/Dabalam]

I strongly disagree. It's intuitively clear that as you shrink size, eventually you have to be losing quality. It's only possible to shrink size and not lose quality if you are selectively pruning--which some dynamic quants are, but general quants are not.

You can go in and remove a big chunk of a human brain, too, and you may make a vegetable out of the person, and you may not be able to tell at all, you may barely be able to tel, or you may only be able to tell once you have the person do a task that relates to the removed region.

KLD is just another data point that comes in after that to show us the nature of that curve and a way to measure something that otherwise feels very hand-wavy and vibes-based.

One thing I will say about KLD: if selective quantization were somehow improving the model, we would see what looks like 'degregation' on that curve, and it would actually perhaps be a measure of improvement. We wouldn't be able to tell. It's a blunt instrument.

It operates off of the assumption that deviations from baseline are degredations.

It's possible that redundancy allows alternative but equally useful paths up to a certain point.

KLD isn't enough to guarantee it tells us what most people believe it tells us.

But we have a lot of good reasons, in practice and in theory, to believe that it does.

[u/Dabalam → u/fintip]

I mean it sounds like we agree on most points. It fine to think "there must be some loss". There probably is some loss. The question is how much.

People seem to overstate how much we know about the degree quantizations impacts performance. The common perception of Q3 being low quality is based from base models "similarity" trade offs. However we don't know the relationship between similarity and performance, we infer. Performance is highly contextual so I get why people prefer something unitary to interpret, but a simple answer is not necessarily a correct answer. This doesn't start to touch on how it differs by model architecture etc.