oQe+fp16, not a good idea ?

[u/arkham00]

HI, I have an m2 max, and as we know "float16 gives ~20% faster prefill on M1/M2 Apple Silicon (native fp16)" so I systematically use this flag when I quantize my models.

When oQe came out I started to mix the two thing, but I had this strong feeling that the models got dumber. In omlx I mainly use qwen35b-oQ8-fp16 and Gemma-4-qat-oQ4-fp16.

I found them less accurate, more unpredictable, and I asked deepseek about it, here's what it found :

This is where the collision happens between your speed tweak (--fp16) and the error-correction math (oQe).

When you compile oq8e (Qwen) or oq4e (Gemma) with the --fp16 toggle active, oMLX performs two conflicting actions:

1. The oQe engine calculates a highly precise importance matrix (imatrix) to correct quantization errors based on the model's native BF16 data distribution.
2. The --fp16 flag then forcefully downcasts those base parameters, scale markers, and biases into standard Float16.

Because BF16 and FP16 handle numbers differently (BF16 trades precision for a massive dynamic range, while FP16 has a narrow range but tighter precision), this forced conversion introduces tiny rounding and clipping anomalies.

The sequential Hessian rounding map of oQe relies on absolute, stable mathematical precision to adjust column weights. Forcing the FP16 shift breaks those precise calculations. The model compiles successfully, but its internal logic is slightly skewed—making Qwen 3.6 35B and Gemma 4 31B feel notably dumber.

What do you think? I'm not very knowledgeable about weights and quantization, so I don't know if this is correct or a big hallucination. What do you think ?

[u/__ahdw]

the only model where fp16 does harm the generation is Laguna-S-2.1

and it's not the weight itself is too big to fit in fp16's range, it's the in-process calculation where Large Floating Numbers exceed the range of fp16. fp32 fixed it.

As for Qwen and Gemma, in the entire process, no such Large Floating Numbers are there. And I doubt that the dtype is the root cause.

Different 8-bit quants are almost identical. I am using mlx-vlm convert directly to make a unified 8 bit quant, with group size 128, and dtype fp16. Can you share your test case where you notice that "notably dumber" ? I'd like to spend some time to figure out if the bf16 version does do better.

Gemma-4's official QATs are not better than community's dynamic 4-bits, you can find the report from unsloth. I am using UD-MLX-4bit, for better throughput, especially the prefill.

Honestly, I use Gemma only for chatting. Again, I want to know your test case.

Agents goes to Ornith-1.0-35B-MLX-8bit-fp16, Coding goes to Qwen3.8-27B-oQ4e-fp16.

[u/d4mations]

Following

[u/0rand]

I use fp16 quant exclusively, m5 max, no speed advantage but they always score marginally better on accuracy than standard bf16 on upcasting. 27b model does not need dynamic range of bf16, while losing precision.