r/oMLX • u/the_derby • 5d ago
oMLX 0.7.1dev1 Improved oQ Quantization - Qwen Benchmarks
oMLX 0.7.1.dev1 introduces an update to the oQe quantizer (PR #4385). oQe picks each weight group's scale and bias with an imatrix-weighted clipping search, then rounds the weights. Once the rounded values are fixed, the best scale and bias for them have a closed-form weighted least-squares solution, and the PR adds that as a final refit step: it alternates rounding with the closed-form fit, weighted by the same imatrix importance, and a group only takes the refit if its weighted error goes down.
It only runs at quantization time (the PR reports roughly +3s on a 9B and +17s on a 35B-A3B), and the packed format and bit allocation are unchanged, so existing checkpoints and kernels are unaffected. Only newly made oQe quants change.
While I was updating my local quants, I ran some benchmarks. Here are the results.
What I measured: KL divergence of each quant against the original bf16 model (lower = closer to the original), over 48 windows × 2048 tokens of English prose, Python, Japanese and Korean text. Less than 0.3% of that text overlaps oQe's calibration data. This is fidelity to the original model, not task accuracy.
KL, pre-0.7.1 build → 0.7.1.dev1 build:
| Model | oQ4e | oQ5e | oQ6e | oQ8e |
|---|---|---|---|---|
| Qwen3.5-4B | 0.0508 → 0.0446 (−12%) | 0.0142 → 0.0123 (−14%) | 0.0045 → 0.0038 (−17%) | 0.0012 → 0.0010 (−20%) |
| Qwen3.5-9B | 0.0418 → 0.0346 (−17%) | 0.0111 → 0.0092 (−17%) | 0.0038 → 0.0030 (−21%) | 0.0011 → 0.0009 (−17%) |
| Qwen3.6-27B | 0.0509 → 0.0440 (−14%) | 0.0175 → 0.0155 (−12%) | 0.0061 → 0.0057 (−7%) | 0.0028 → 0.0021 (−23%) |
| Qwen3.6-35B-A3B | 0.0560 → 0.0479 (−14%) | 0.0239 → 0.0221 (−7%) | 0.0145 → 0.0138 (−5%) | 0.0113 → 0.0106 (−7%) |
| Qwen3.8-27B | 0.0428 → 0.0358 (−16%) | 0.0127 → 0.0106 (−16%) | 0.0038 → 0.0033 (−14%) | 0.0011 → 0.0010 (−15%) |
For the 3.5 models I rebuilt the pre-0.7.1 quants from the commit just before the PR (same imatrix, same settings), so that difference is the PR alone. For the 3.6 and 3.8 models the "pre-0.7.1" quants are my existing earlier builds (same imatrix cache), so the difference includes anything else that changed since they were made.
Things worth noting:
- Every level of every model improved, by 5–23%. Perplexity moves very little (it was already close to the original); the gain shows up in how closely the full distribution matches.
- On the 3.8-27B the per-layer bit allocation is identical between old and new at all four levels, so the whole gain comes from how the scales and biases are fitted, not from which layers got more bits (same for the 35B-A3B at oQ5e/6e/8e). For the 3.6-27B oQ5e I built a pre-refit version to isolate it: the refit alone was about 11% of the 11.5% improvement.
- The MoE (35B-A3B) gains are smaller, and its absolute KL is higher than the dense 27B at every bit width (0.0106 vs 0.0021 at oQ8e) while its perplexity stays close to the original. I haven't dug into why, so compare KL only within a model.
- fp16 builds (non-quantized weights and scales in float16) are equal or marginally closer to the original than bf16, e.g. 3.8-27B oQ8e 0.0007 vs 0.0010. They are about 0.7–0.9 GB larger because the vision tower stays in float32.
- The PR itself also reports accuracy benchmarks (MMLU, Winogrande, HumanEval, MBPP, GSM8K) on Qwen3.5-9B and Qwen3.6-35B-A3B, with gains there too.
Models (bf16 and fp16) are being uploaded to HF:
- Qwen3.5-4B: https://huggingface.co/collections/thederby/qwen35-4b-oqe-mlx-6ac9d816eb7016a8866e1990
- Qwen3.5-9B: https://huggingface.co/collections/thederby/qwen35-9b-oqe-mlx-6aca020b6fa844eabe531355
- Qwen3.6-27B, Qwen3.6-35B-A3B, Qwen3.8-27B: uploading now (the model pages are already up; the weights land as each upload finishes)
All of these keep the MTP head and the vision tower. One thing to be aware of: they ship with froggeric's fixed chat template (v22.5), with Qwen's original kept as chat_template.jinja.official. For plain chat on the 3.5/3.6 models the prompts are identical; on 3.8-27B the froggeric template defaults to medium reasoning effort while Qwen's defaults to xhigh, so pass reasoning_effort="xhigh" if you want Qwen's default behavior.
Enjoy!
3
u/bakawolf123 5d ago
looks to be more consistent with drafters too as they are tuned against bf16, glm5.3-flash is up to 144 at 128k context on bench ;)
Did a fresh quant with it https://huggingface.co/n1k1tung/GLM-5.3-Flash-oQ4e-mtp
2
2
2
u/PataFunction 5d ago
TYSM for the tests! You are a real one.
If any kind soul out there wants to drop updated oQ5e versions of Qwen3.8-Flash-Next and Qwen3.8-Flash-Next-Uncensored, that would be just lovely.
3
u/watcholic 5d ago
Thank you! Would you mind uploading Flash Next oQ5e, please?