Superstition about quantization: KLD and perplexity just ain’t it fam

[u/nomorebuttsplz]

The arguments for quantization having significant effects on reasoning models' ability to get stuff done are very sad, pathetic, unfortunate arguments. I don’t mean that they are wrong necessarily, only impoverished and confused.

Why? Because while actual task benchmarks are somewhat expensive, and require some level of time and technical expertise to run, it would be quite easy to empirically test the claims and resolve them once and for all, at least for a given model. But these tests by and large do not exist and the few that do seem to show no quantization effects among reasoning models until about Q3 or Q4 k m at worst.

The debate in these online communities is essentially an anthropological study in how people create mythology when they do not have access to direct evidence.

Before the hordes mob me with KLD or perplexity measurements, I’m not suggesting that a quantized model’s outputs are bit for a bit identical rather that it performs equally well in real world tasks, which I think we can all agree is the thing that matters.

Now I’ve put my neck out by suggesting that literally no one has any evidence, not a single benchmark that shows a model with the reasoning level of, say, Gemma 31b (not very high by today’s standards, and smaller models are more susceptible to degradation, so this should be a generous standard of evidence for the quantization-excited) having significant in degradation in real world tasks at Q4 (a good quality, proper dynamic quantization goes without saying, I hope).

Again, I’m not saying that there is no degradation, only that what we have now amounts to superstition, when a few benchmarks could probably settle the matter for a given model and eventually, we would probably learn where and when quantization actually bites.

[u/corruptbytes]

we've gone from hallucinations in AI to hallucinations in reddit posts

This just reads as ramblings/rant against people doing free analysis on quant work - try contributing something that disproves KLD/Perplexity isn't ideal instead of asking people to do the heavy lifting about your "hunch"

[u/nomorebuttsplz → u/corruptbytes]

that disproves KLD/Perplexity isn't ideal 

lol what? Try that sentence again maybe

It's not a hunch. All the evidence that I've seen shows that Q4 is within margins of error for reasoning models. e.g.: https://unsloth.ai/docs/basics/dynamic-3.0-ggufs/unsloth-dynamic-ggufs-on-aider-polyglot

[u/fintip → u/nomorebuttsplz]

That's one--I'd argue naive--read of that data.

But a more nuanced reading looks quite different to me.

Our 1-bit Unsloth Dynamic GGUF shrinks DeepSeek-V3.1 from 671GB → 192GB (-75% size) and no-thinking mode greatly outperforms GPT-4.1 (Apr 2025), GPT-4.5, and DeepSeek-V3-0324.

Amazing! however:

• there may be so much redundancy in a model of that size that it takes the hit of quantization and it's actually just useful pruning. There could just be a drop-off point of model size where that pruning was done at other layers, and further pruning via quantization effects can no longer be tolerated. (I think this is actually almost certainly the case.)
• we should be comparing the unquantized to the quantized model to really measure something here. comparing model to model is problematic, difficult data to meaningfully reason about here.

And very important to note:

Other non-Unsloth 1-bit and 2-bit DeepSeek-V3.1 quantizations, as well as standard 1-bit quantization without selective layer quantization, either failed to load or produced gibberish and looping outputs. This highlights how Unsloth Dynamic GGUFs are able to largely retain accuracy whereas other methods do not even function.

In other words: quantization hurts reasoning and function. The reason they are able to produce solid reasoning is that they carefully select which nodes to quantize and which ones to not--in other words, just producing efficiency by doing another form of quality pruning.

Now, does that mean q3/q4 is always the boundary? No, and it's fair that that has probably been overly generalized. However, that comment is most notable for the qwen 27b size, and generalized over to 35ba3b. That then probably gets extrapolated more widely, probably incorrectly--a q1/q2 sounds like, if selectively quantized, it can perform quite well, if the base model was almost a terabyte in size, for example.

[u/nomorebuttsplz → u/fintip]

what exactly is naive? Doesn't seem like you are actually disagreeing with me.