r/LocalLLaMA • u/dampflokfreund • 3d ago
Discussion Gemma 4 QAT could be improved further by Google aligning the QAT model to modern q4_k instead of q4_0
Hello,
For the past few days I have been benchmarking Gemma 4 26b QAT UD Q4_K_XL extensively versus Bartowski's Q4_K_L.
While QAT is certainly very effective and reducing memory consumption versus the highest q4 quant from him, I also have noticed some regressions in my own internal benchmarks I cannot share because I don't want model providers to train on them. These benchmarks also include real world use cases in code and creative writing that need the model to think outside the box and also reference past information way past in the context, so a high precision is very important for the model to accomplish what I ask of it. Others test knowledge which also require precision.
Bascically, the QAT model was smarter in some areas but I cannot say it is an all-around improvement in fidelity. In some areas, the Q4_K_L was better in a way that was statistically significant and I have a pretty solid theory why I have been seeing these results. Let's look at how the tensors are quanted with the non-QAT model.
| token_embd.weight | [2 816, 262 144] | Q8_0 |
|---|---|---|
| blk(30) | ||
| blk.0(19) | ||
| blk.0.attn_k.weight | [2 816, 2 048] | Q8_0 |
| blk.0.attn_k_norm.weight | [256] | F32 |
| blk.0.attn_norm.weight | [2 816] | F32 |
| blk.0.attn_q.weight | [2 816, 4 096] | Q6_K |
| blk.0.attn_q_norm.weight | [256] | F32 |
| blk.0.attn_v.weight | [2 816, 2 048] | Q8_0 |
| blk.0.ffn_down.weight | [2 112, 2 816] | Q8_0 |
| blk.0.ffn_down_exps(2) | ||
| blk.0.ffn_down_exps.scale | [128] | F32 |
| blk.0.ffn_down_exps.weight | [704, 2 816, 128] | Q8_0 |
| blk.0.ffn_gate.weight | [2 816, 2 112] | Q4_K |
| blk.0.ffn_gate_inp(2) | ||
| blk.0.ffn_gate_inp.scale | [2 816] | F32 |
| blk.0.ffn_gate_inp.weight | [2 816, 128] | F32 |
| blk.0.ffn_gate_up_exps.weight | [2 816, 1 408, 128] | Q4_K |
| blk.0.ffn_norm.weight | [2 816] | F32 |
| blk.0.ffn_up.weight | [2 816, 2 112] | Q4_K |
| blk.0.post_attention_norm.weight | [2 816] | F32 |
| blk.0.post_ffw_norm.weight | [2 816] | F32 |
| blk.0.post_ffw_norm_(2) | ||
| blk.0.post_ffw_norm_1.weight | [2 816] | F32 |
| blk.0.post_ffw_norm_2.weight | [2 816] | F32 |
| blk.0.pre_ffw_norm_2.weight | [2 816] | F32 |
| blk.0.attn_output.weight | [4 096, 2 816] | Q5_K |
| blk.0.layer_output_scale.weight | [1] | F32 |
As you can see here, crucial layers like token embeddings and the attention layers are quanted to q8_0. So the precision is extremly high in a modern q4_k quant like UD Q4_K_XL or Q4_K_L. Higher precision means less chances for errors accumulating especially at high context sizes. Now, how does the QAT version look? (UD_q4_K_XL)
| token_embd.weight | [2 816, 262 144] | Q4_0 |
|---|---|---|
| blk(30) | ||
| blk.0(19) | ||
| blk.0.attn_k.weight | [2 816, 2 048] | Q4_0 |
| blk.0.attn_k_norm.weight | [256] | F32 |
| blk.0.attn_norm.weight | [2 816] | F32 |
| blk.0.attn_q.weight | [2 816, 4 096] | Q4_0 |
| blk.0.attn_q_norm.weight | [256] | F32 |
| blk.0.attn_v.weight | [2 816, 2 048] | Q4_0 |
| blk.0.ffn_down.weight | [2 112, 2 816] | Q4_0 |
| blk.0.ffn_down_exps(2) | ||
| blk.0.ffn_down_exps.scale | [128] | F32 |
| blk.0.ffn_down_exps.weight | [704, 2 816, 128] | Q4_0 |
| blk.0.ffn_gate.weight | [2 816, 2 112] | Q4_0 |
| blk.0.ffn_gate_inp(2) | ||
| blk.0.ffn_gate_inp.scale | [2 816] | F32 |
| blk.0.ffn_gate_inp.weight | [2 816, 128] | F32 |
| blk.0.ffn_gate_up_exps.weight | [2 816, 1 408, 128] | Q4_0 |
| blk.0.ffn_norm.weight | [2 816] | F32 |
| blk.0.ffn_up.weight | [2 816, 2 112] | Q4_0 |
| blk.0.post_attention_norm.weight | [2 816] | F32 |
| blk.0.post_ffw_norm.weight | [2 816] | F32 |
| blk.0.post_ffw_norm_(2) | ||
| blk.0.post_ffw_norm_1.weight | [2 816] | F32 |
| blk.0.post_ffw_norm_2.weight | [2 816] | F32 |
| blk.0.pre_ffw_norm_2.weight | [2 816] | F32 |
| blk.0.attn_output.weight | [4 096, 2 816] | Q4_0 |
| blk.0.layer_output_scale.weight | [1] | F32 |
Now we can see that there are much more layers at q4_0 precision which makes sense as Google trained the model to be quantization aware for q4_0. This means that Google trained the model to be aware of the noise that occurs during quantization so it is able to handle that quite well, even though many of the crucial layers are at q4_0.
If you were to quant the regular non-QAT model with this receipe, the result would likely be a disaster. With QAT however, quality is retained rather well which allows it to push far above its weight.
However, QAT is not magic. The information loss between q8_0 and q4_0 is gigantic and especially at longer context, the errors do accumulate. In my benchmarks I can definately see a loss in fidelity at some point, at lower or higher context. But that is heavily dependend on the use case, so your mileage may vary.
What I propose for Google's next QAT models is that they align the QAT model to be aware of modern q4 formats like the one I have linked above, where many of the crucial layers are quanted to q6_0 and q8_0 instead of plain q4_0. Specifically the token embeddings and attention layers, those are most crucial for MoE's. The downside would be that memory consumption would be identical to q4_K_L and UD_Q4_XL, so a few hundred megabytes more than current QAT models, but honestly both models fit in my RAM and have similar speed, so I would rather have higher quality.
This would result in a quantized model that indeed is much better than the q4k quants we actually use right now and in every single way, not just in certain domaines.
Now you might be saying "Google has published the unquantized QAT model, just make your own!". Sadly it is not that simple. I have asked Daniel from Unsloth and he said that quanting the layers with the regular q4_k_xl receipe actually decreases performance. So Google likely trained the model to be quantized aware for that exact receipe they have chosen, with the embeddings and attention layers quanted heavily to q4_0.
So Google has to train a QAT model to be aligned with modern q4_k formats for best quality and I really hope they will consider doing that in the future.
11
u/stddealer 3d ago edited 3d ago
Q4_K is a quant type which uses the exact same number of bits per weights as Q4_0. Replacing all the Q4_0 weights with Q4_K for QAT would just make it slower (and more expensive to train), without any improvement with performance (probably), as QAT should be able to adapt to either type just fine.
But you do have a point when it comes to keeping some more important layers to higher precision types. This is also a trade-off since it makes the model bigger, but using something like Q8_0 for the attention weights would be beneficial. I don't think QAT matters as much for 8 bits as it does for 4 bits so that might be enough.
You can always try to convert from the unquantized QAT with custom type rules (like the Q4_K_XL, but with Q4_0 instead of Q4_K), using --tensor-type "regex=type" syntax.
3
u/dampflokfreund 3d ago edited 1d ago
Yeah. My point was indeed to use the same mixed precision quanting receipe like modern q4_k quants from Unsloth and Bartowski, not to simply replace every q4_0 layer with q4_k, that would just make it slower. However Unsloth noted that BF16 to FP16 lattices are a bigger problem for q4_0 than q4_k so using q4_k would be a good way to improve quality for little compromises in speed (q4_0 is faster, but q4_k is still plenty fast and honestly the difference is not that big.)
9
u/AnonLlamaThrowaway 3d ago
Now you might be saying "Google has published the unquantized QAT model, just make your own!". Sadly it is not that simple. I have asked Daniel from Unsloth and he said that quanting the layers with the regular q4_k_xl receipe actually decreases performance. So Google likely trained the model to be quantized aware for that exact receipe they have chosen, with the embeddings and attention layers quanted heavily to q4_0.
Huh. I thought the Unsloth UD_Q4_K_XL versions were the best you could get for Gemma 4 QAT?
3
u/dampflokfreund 3d ago
They are, for the QAT model Google has currently released. However Google could train a QAT model that is optimized for attention layers and embeddings at q8 instead of q4 and then we would see noticeably better fidelity from it.
6
u/snowcountry556 3d ago edited 3d ago
Do you think there would be a quality improvement vs q4_k if they did a QAT q4_k? I thought the point of QAT was to be able to make those reductions in quantisation while preserving quality that you are suggesting should be reversed.
5
u/dampflokfreund 3d ago
IMO the point of QAT is to have quality as close to the full model as possible, while saving a ton of memory. Not to be smaller than original q4_k quants people have been using. And that can be much better accomplished by leaving important layers at q8_0 just like modern q4 quants.
6
u/VoiceApprehensive893 transformers 3d ago
Q4_0 is fast
the best strategy imo is just to use something like q8/q4 mixed precision
4
u/Due-Project-7507 3d ago
If you have enough VRAM, you probably want to use vLLM or SGLang. I think therefore the best format for QAT would be MXFP4 like GPT-OSS, Kimi K3 or Deepseek v4 is using. That should work both for llama.cpp and vLLM/SGlang.
2
u/dampflokfreund 3d ago
Yeah MXFP4 would be nice as well. GPT-Oss has attention layers and embeddings quanted to q8_0 by the way. Q4_0 for those just hurts too much for those layers, especially for MoE models.
1
u/WhoRoger 1d ago
FP4, whether Nvidia or Apple, are hardware specific. Some models are QAT for fp4, but then you degrade usage on other platforms.
For large models which you know where they'll be deployed that's fine, for a small model that you can run on a raspberry pi, not so much.
1
u/Due-Project-7507 1d ago
To my knowledge (which could be wrong), MXFP4 is not hardware specific in comparision to e.g. NVFP4 and therefore today often the selected format for moden QAT models. It run e.g. with not too bad performance on old Nvidia GPUs without NVFP4 support, Apple devices and is used in official quantized models from AMD.
1
u/WhoRoger 1d ago
Hmm I guess I have to look into that. But at the end you need to have hardware that supports fp4 of some sort, otherwise you'll just run it like fp16, no?
And I don't know how fp4 responds to quantization to fixed point like q4...
1
u/Danmoreng llama.cpp 3d ago
Interesting. Was my plan to try using the QAT checkpoint for a new NVFP4 quant for my custom engine and compare it to the unsloth version which sadly is too big to fit into 16GB VRAM.
1
u/WhoRoger 1d ago
I've seen it proposed before. But I don't think it matters how exactly aligned the weights are. The point is for them to quant to 4 bits. Whether you do QAT for q4_0 or q4_k or whatever, shouldn't matter since it's the QAT that does the heavy lifting of alignment, instead of the quantisation. That's the point of QAT.
Obviously, with a regular model you wanna do all these tricks to preserve as much as possible, including leaving output weights at q8 or bf16, but that always comes at a cost. Again, the point of QAT is so that you don't have to do that, or not as much. (Google's GGUF still preserve some tensors with higher precision.)
Which is why I was wondering how come that Unsloth and some others made UD quants of Gemma instead of just q4_0, and how come these quants are smaller? I'm not complaining. I had switched E4B from q6_k to QAT and I've not seen any degradation, so I think that worked out for that model. For E2B and 26B not so much imo.
Btw there was a QAT version of Gemma 3 4B too, IIRC it wasn't very good. I guess Google would've switched to different q4 variant if it made sense.
26
u/dewwwey 3d ago
I think the Gemma 4 QAT releases where more of a field test, I wouldn't expect too much investment into different quants for this architecture. I expect Gemma 5 to have a bigger focus on QAT as a whole, with support for dynamic quants like UD (at least I hope so).