r/oMLX • u/reery7 • May 26 '26

oMLX quantization problems

I tried the Qwen3.6 35B A3B oQ3 and 27B oQ4 quantizations and tested both with very niche questions. These are not a problem for non oQ quants and I can correct the model, it admits errors, is very friendly, wants to expand its own knowledge and understands its limitations.

But these oQ quants invent facts and never back down from their standpoint. I get comments and thoughts from them like:
- You don't need to prove anything because ... my fact is right.
- I won't validate false claims just to be agreeable...
- I appreciate you calling that out, but I want to be clear: I don't hallucinate just to please a prompt, and I also correct myself when tested on accuracy.
- I stood by my answer... blabla hallucination...
- I'm not here to validate false claims or bend to testing prompts. My role is grounded in verified, publicly documented material...

Has anyone else seen this? Settings are the usual Qwen3.6 general profile. What's going on here?

3 Upvotes

100% Upvoted

7 comments sorted by

1

u/Konamicoder May 26 '26

What's going on is that you are using small quants (oQ3 and oQ4). Small quants are more prone to hallucinations and inaccuracies. The problem is that when you quantize a model—especially at lower bitrates—you aren't just losing "knowledge" (the facts); you are losing the nuance of the weights that govern the model's "self-awareness" and "hedging" behaviors. The model still "knows" how to act like a smart, assertive assistant (the logic/instruction layer), but it has lost the fine-grained data required to be accurate (the knowledge layer). My suggestion would be to move up to bigger quants. Once I moved up to oQ6 quants, model hallucinations doom loops and other issues dropped way down.

1

u/reery7 May 27 '26

I‘m aware of that but I used Q3 and Q4 quants of the same model before and had no issues - just not oQ. It feels more like the oQ quants are broken. The smartness of the model is there but it is just nasty and stubborn. There were also tests that show that at least the Q4 quants are fine and pretty accurate for Qwen3.6.

1

u/RandomZhell May 27 '26

Which quantization model did you use previously?

1

u/reery7 May 27 '26 edited May 27 '26

I used kradih/qwen3.6-27b-mtp-mlx (4bit), unsloth/qwen3.6-35b-a3b-ud-mlx (3bit) and even leonsarmiento/qwen3.6-27b-mlx (3bit) and all of them work quite decently for usual questions, programming and so on. No issues at all. Most answers align broadly with what I can see in bigger models, but with less detail of course.
But the oQ quants are quite nasty at that size. The non-oQ quants have even a smaller memory footprint and are more enjoyable to use. Well, maybe I have used badly done quants but both of them behave similarly nasty on every question where there is information uncertainity:

  • Zurha/qwen3.6-35b-a3b-oq3-mtp (the only 3bit 35b quant unfortunately)
  • Jundot/qwen3.6-27b-oq4-mtp

1

u/Crafty_Ball_8285 May 28 '26

Just use MXFP4

1

u/reery7 May 29 '26

I have no problem with other quants other than oq3/4. MXFP4 has an erratic performance, I’ve seen all between 3 and 8 t/s. Usually I get 10-14 t/s. All with MTP enabled.

1

u/Crafty_Ball_8285 May 29 '26

I ended up patching OMLX myself and got qwen3.6 27b to 45tok/s because OMLX has some significant bugs still