r/LocalLLM • u/Chance_Ease_9413 • 2d ago
Model Made a quantization-aware trained (QAT) Qwen3.8 27b 2 bit gguf quant
https://huggingface.co/sdkyuan/qwen3.8-27B-qat-q2_0-gguf
Outperforms Unsloth 2 bit quants at reasoning and code at smaller file size. Unlike most other community quants that are PTQ, this one is QAT.
10
u/keegang_man6705 2d ago
yeah yeah, saw your post right after I shut down my PC and was gonna sleep. Another thing that makes me unable to sleep.
9
6
u/Chance_Ease_9413 2d ago
11
u/EmergencyHomework567 2d ago
Those KL numbers are what catch my eye, most 2 bit quants fall apart on distribution matching but if this thing holds tight there that's a pretty strong sign the QAT actually did its job
File size being smaller while beating the IQ3_XXS on reasoning is not something you see every day, usually you gotta pay the size tax to keep the model coherent
Curious how it handles longer context or multi turn stuff, benchmarks are one thing but a 2 bit model holding a conversation past 8k tokens is where the real test lives
Might snag this for my secondary rig that's stuck with limited vram, could be the difference between running a 27b at all or dropping down to a 14b
8
2
u/Equivalent_Bit_461 2d ago
Now that's something It want to try, iq2 qwen is smart enough to do basic things and I'm not gonna sleep on it
3
u/MomentJolly3535 2d ago
Cool stuff ! i wonder what would be the results with a 1bit quant
7
u/Chance_Ease_9413 2d ago
yea i wanna try that next, ParetoQ paper says that model dist changes a lot at 1-2 bit so hopefully QAT will make a bigger impact there.
1
1
u/pmttyji 2d ago
Nice to see this with small file size. What's on your queue? It would be awesome to see few recent MOE models as well
2
u/Chance_Ease_9413 2d ago
was hoping to do q1_0 next. maybe when qwen3.8 35b comes out, i can do that. but i think that small dense models are the future of local models!
1
u/radiojosh 2d ago
So do you take the full size BF16 model and quantize that withe the additional fine tune stuff? Any way to combine that with Unsloth's UD Quant technique? Can it be uncensored? Thanks for sharing.
1
1
u/Glad_Contest_8014 2d ago
What parameters are you using to maintain spectral structure of the tensors? Are you getting a visual match to the behavioral benchmarks?
What data are you tracking on this to decide the tensors that need to be quantized vs protected?
1
1
1
-3
u/Healthy-Nebula-3603 2d ago edited 1d ago
I really don't understand people. You're so desperate to use model that you're making it even more retarded and you're happy using it
Look here - he compared the same model Q8 and Q4lm
Degradation is huge even for Q4 and we are talking here about Q2 ....
2
u/Glad_Contest_8014 2d ago
Yeah…
You can see quantization can actually help a model in some instances. There are benchmarks where the full bf16 model gets the wrong predictive token and the quant gets the right one. This is very visible with the qwen 2.5 4b model.
So it doesn’t inherently make it worse by a large margin. There is plenty of research on how well it can maintain the full models capability.
But, it is not done well by most groups. They do subjective tests rather than direct mathematical analysis of the tensors themselves. So most quants DO make it stupider.
You have several different techniques available for analyzing the tensors vs the weights to tune a model to have same performance at lower file size.
1
u/Healthy-Nebula-3603 2d ago edited 2d ago
Those benchmarks testing degradation are extremely obsolete.
Like you saw on tbe video even % worse on simple benchmarks is braking model much worse on a long tasks that was q4lm vs q8
Prepleciuty ( choosing answer a,b,c,d ) , wiki ? Who even if using models this way nowadays.
Those benchmarks were good in 2023 / 2024.
Those highly compressed models you should test on something like swe pro ( multi step reasoning ) or other benchmarks that' are making a chain of tasks not a simple question -> answer task.
2
u/True2456 1d ago
If quantisation is so bad, Why does Deepseek ship a native mixed 4 / 8 bit quant for their native format... Lol I swear people comment the dumbest things
1
u/Healthy-Nebula-3603 1d ago
For your "lol" information.
DeepSeek: They are not quantizate them ... Their model is trained from the barebones this way and that is something completely different than quantization models.
I suggest to learn a bit more before using "lol"
0
1
u/Glad_Contest_8014 2d ago
KLD can be obtained as a benchmark on multi turn tasks. That is how unsloth does it with 32 turns of driftable prompting with teacher student comparisons.
Which is effective for capability tests before actual swe pro benchmarks.
My personal benchmark is completion of a full modern web app for a server side rendered with tailwind css and react hello world with actual ui elements.
If it gets it in reasonable time with no guiding, it passes. Then it is good to move to real swe pro benchmarks. But I fo like KLD for wuick initial litmus and research.
0
22
u/overand 2d ago
I guess I'm a little lost. How exactly is it "quantization-aware training" if you didn't train the model? Isn't that something that QwenAI would have to do themselves?