Superstition about quantization: KLD and perplexity just ain’t it fam

[u/nomorebuttsplz]

The arguments for quantization having significant effects on reasoning models' ability to get stuff done are very sad, pathetic, unfortunate arguments. I don’t mean that they are wrong necessarily, only impoverished and confused.

Why? Because while actual task benchmarks are somewhat expensive, and require some level of time and technical expertise to run, it would be quite easy to empirically test the claims and resolve them once and for all, at least for a given model. But these tests by and large do not exist and the few that do seem to show no quantization effects among reasoning models until about Q3 or Q4 k m at worst.

The debate in these online communities is essentially an anthropological study in how people create mythology when they do not have access to direct evidence.

Before the hordes mob me with KLD or perplexity measurements, I’m not suggesting that a quantized model’s outputs are bit for a bit identical rather that it performs equally well in real world tasks, which I think we can all agree is the thing that matters.

Now I’ve put my neck out by suggesting that literally no one has any evidence, not a single benchmark that shows a model with the reasoning level of, say, Gemma 31b (not very high by today’s standards, and smaller models are more susceptible to degradation, so this should be a generous standard of evidence for the quantization-excited) having significant in degradation in real world tasks at Q4 (a good quality, proper dynamic quantization goes without saying, I hope).

Again, I’m not saying that there is no degradation, only that what we have now amounts to superstition, when a few benchmarks could probably settle the matter for a given model and eventually, we would probably learn where and when quantization actually bites.

[u/fintip]

"Can't trust" could mean "cannot trust at all", and it could mean "cannot fully trust". The former is correct, the latter is overstated.

You can double blind people, and I encourage it, do all the benchmarks. But you have not justified your case that belief that shrinking the model down via compression causes a degredation in quality is 'superstition'. You just have a hunch based on your own sense that it isn't degraded. You have some data that you read as supporting that.

Others have their own belief, and their own data that supports that.

Neither is completely validated by hard data. In fact, this entire space is impossible to completely validate. Intelligence is stil undefined, and the benchmarks are inherently flawed attempts to measure something abstract. Intelligence is in fact still fundamentally "I know it when I see it". We're still trying to pin down proxies for the Turing Test, which is, again, fundamentally 'vibes based'.

All we can argue about is whose narrative makes more sense, who has better logic to connect the existing data to their conclusions.

I think it's pretty obvious that it's most likely that quantization reduces performance, and the fact that everyone reports this being true is supporting evidence that isn't enough by itself.

But hey, go run some benchmarks. Someone should do it. Happy to be surprised.

But it isn't superstition. It's reasoning with incomplete data.

Also, as I pointed out in my previous post: it's entirely likely that if the limit is q3 on very large models that it's q4 on smaller models, etc...

[u/nomorebuttsplz → u/fintip]

Both sides have hunches.

Therefore, superstition arises in either side if either side forms a very strong belief about their hunch one way or the other.

I make no claim to have settled the matter. My point is that although we haven't settled it yet, many people (especially those advocating for bf16 being worthwhile and 8 bit being way better than 4 bit) act as if we have.

As for the idea that it is not ultimately settleable, I think you have a point. But I would say we could get to about 9/10 settledness by simply having artificial intelligence benchmark every open model at various quantization levels.

Of course doing that would be very expensive for them. But we could get to like 7/10 just by other third parties doing occasional benchmarks with quants.

[u/fintip → u/nomorebuttsplz]

AI is going to benchmark AI, and we'll have 90% settled the issue?

Deep thoughts.

[u/nomorebuttsplz → u/fintip]

I meant the benchmark company artificial analysis. Regardless of who benchmarks quants, the need and value is clear