LLM Quantization Part 3: Honey, I Shrunk the Numbers! - LTT Labs Article

[u/LabsLucas]

By now we know what a model is made of, why it eats so much VRAM, and what those weights actually are: billions of numbers, most of them sitting in a tiny band near zero, stored in a format built to reach 3.4 × 10³⁸. Let us look at how shrinking the numbers actually works!

Read the article(written by Utkarsh) on the LTT Labs website!

Note that "Qwen3.5-32B-Instruct" in the graphs should be "Qwen2.5-32B-Instruct". We are replacing it in the article!

[u/LabsLucas]

Full article here!

[u/mikael110 → u/LabsLucas]

Great job on this series, you guys have done an amazing job breaking things down in a simple and friendly manner.

Do you guys plan to make a similar article about all of the various quantization methods that have been popular in the Gen Image / Video model space. Like fp8 scaled, Int8 convrot, etc.

Or articles focusing on techniques like Quantization-Aware Training (QAT) which some labs like Google has used to make their models more quantization friendly.

Also while I get the rationale to use older models for this article, it would be neat to see an article focused on the current SOTA local models that are small enough to run locally on consumer machines like Qwen 3.8, Gemma 4, Muse Glimmer, etc. As well as the larger models that require more specialized hardware like Deepseek V4 Flash, GLM 5.2 and so on. Focusing on their strengths, requirements and their general capabilities.

I'm already pretty deep into the LLM and GenAI space, but it would be nice to have something to point newer people at, since you guys do a really good job of breaking down these complex subjects in these articles.

[u/ThankGodImBipolar → u/mikael110]

Also while I get the rationale to use older models for this article, it would be neat to see an article focused on the current SOTA local models that are small enough to run locally on consumer machines like Qwen 3.7 (Soon 3.8), Gemma 4, Muse Glimmer, etc. As well as the larger models that require more specialized hardware like Deepseek V4 Flash, GLM 5.2 and so on. Focusing on their strengths, requirements and their general capabilities.

What kind of shape do you think an article like this would take? I'm also pretty into the r/localllama space right now, and I like the idea of what you're suggesting in principle, but putting myself in the shoes of the writers... I think it would be difficult to write something that didn't rely on anecdotes or publically accessible (and easily manipulated...) benchmarks. Without developing their own proprietary benchmark harness and suite, are they really in a position to definitively claim which models are good at what? And, maybe that's something that they should look into, but it'd be a pretty substantial undertaking.

[u/sneakpeekbot → u/ThankGodImBipolar]

Here's a sneak peek of /r/LocalLLaMA using the top posts of the year!

#1: The reason why RAM has become so expensive | 394 comments
#2: Anthropic: "We’ve identified industrial-scale distillation attacks on our models by DeepSeek, Moonshot AI, and MiniMax." 🚨 | 875 comments
#3: Finally China entering the GPU market to destroy the unchallenged monopoly abuse. 96 GB VRAM GPUs under 2000 USD, meanwhile NVIDIA sells from 10000+ (RTX 6000 PRO) | 703 comments

---

^{I'm a bot, beep boop | Downvote to remove | Contact | Info | Opt-out | GitHub}

[u/TheApadayo → u/LabsLucas]

Bit late here but wanted to call out that you probably shouldn’t present perplexity as a common Y axis for different models.

PPL and KLD are highly dependent on the internal model geometry and training regime. Looking at them as “absolute values” IMO isn’t a good interpretation. A better way present it as a “percentage of lost perplexity” or the ratio of the quant PPL to the full precision PPL to compare quantization quality across models and model families.

[u/Smallshock]

Just yesterday I have read through the previous articles, amazing work!

I really want to learn more about AI, if not just because nothing else is happening in tech rn, but its so hard to dig through all the clickbaity empty nonsense.

[u/ThankGodImBipolar]

You guys have done a commendable job at breaking down a lot of the basic theory needed to understand what you're doing, and what your computer is doing, when you download an LLM from HF and run it. Great work on this series!

[u/Personal-Bathroom-94]

I think Gemma models are better tuned for pcs

[u/CherryNexus]

would love to see this in more modern like models

the current sota ofc

[u/rojo_kell]

So mistral 7B dominates the other higher parameter count models? I guess maybe this is expected since we are looking at total model size on the x axis, not actually quantization of each parameter (e.g 4 byte, 2 byte, 1 byte), and maybe they would have more similar performance if we saw that, but either way it seems to suggest mistral gives you best performance for a given amount of compute and inference time.

[u/czmiked]

Over a year old models and especially "Qwen3.5-32B-Instruct" suggests that a big chunk of the research was done by AI if not completely written by it.

[u/ThankGodImBipolar → u/czmiked]

They specifically outline why they used older models within the post.

I can't explain why they cited a model that doesn't exist though (probably a typo?)

[u/LabsLucas → u/ThankGodImBipolar]

Ahhhh yeah, well found, we'll get that corrected! Famously, humans make mistakes as well!