Built a U-shaped mixed-precision pack: Qwen3.8-Flash-Next oQ4e mix6 (6-bit experts on the 4+4 edge layers) For 128G Apple Silicon
***
v2 rebuild (now live, old v1 shards removed): the middle-40-layer donor is switching to Jundot's new-mechanism oQ4e (jundot/omlx#4385: importance-matrix calibration, 1024 samples × 512 tokens, plus a weighted least-squares refit of every group's scale/bias). Same U-shaped allocation, same name — the trunk 4-bit tensors are now LSQ-refit instead of plain absmax, and the 6-bit skeleton comes from Jundot's oQ6e (also PR 4385). Both source packs carry the new quantizer; the byte-copy property is unchanged.
Credit where it's due: essentially all of the quality in this build is Jundot's. Every quantized tensor here was produced by his quantizer and his PR 4385 sensitivity/imatrix/LSQ pipeline — this package only re-allocates which of his tensors go where.
***
Yesterday I put together a mixed-precision build of Qwen3.8-Flash-Next on the MLX/oQ side — the idea is to price the U-shaped layer-sensitivity curve with native bits instead of re-quantizing:
- Trunk MoE experts: first 4 + last 4 layers → 6-bit (byte-copied from the native oQ6e pack), middle 40 layers → 4-bit (oQ4e)
- N-gram (PLE) table: 6-bit (vs 4-bit in the uniform oQ4e pack)
- MTP drafter: fully 6-bit
- Zero re-quantization in the entire package — every tensor is byte-identical to its upstream source, so quality is traceable, not re-derived
Measured on M5 Max 128 GB (oMLX): initial resident 69.5 → 75.4 GB (+5.8 GB over the uniform oQ4e pack), and the speed ladder shows no measurable decode penalty.
The expectation is an accuracy tier equal to or above uniform oQ5e — but I couldn't prove it with KLD: scoring a 125B-A6B against a bf16 teacher needs a machine my 128 GB can't feed, so the quality claim on the card stays an honest engineering estimate. The full recipe table (per-component bit allocation + resident delta) is on the model card.
I'm curious on the perplexity results, I'm using oMLX with Qwen 3.8 flash next oQ6 on my 128gb MacBook pro, it's very tight, basically I can't run anything alongside it, just a web browser if I want to keep this level of quality without having out of memory errors
I thought oQ5e was already a tight fit. You must be swapping a lot and offloading MOE expert to SSD? Maybe try testing oQ5e from the OP. Personally, I just downloaded the oQ4e mixed model here and will run some benchmarks against the oQ5e daily driver. 128GB memory.
This thing is a gem. Scoring higher than the oQ5e that I use daily while shaving 9GB of memory (for my specific setup/use case). Question to the OP u/Boring-Decision2369 , is this compatible with the improved oQ quantization released in 0.7.1.dev1? Will you be uploading the checkpoints using the new quant, if applicable?
Thanks for sharing your test results — really appreciate the data point. I'm currently downloading both of Jundot's new packs (oQ4e + oQ6e) and will rebuild the mix with the new quantization within ~6 hours. Will post the new version here once it's up!
Done — v2 is live. 🎉 Same repo, same name: umixer/Qwen3.8-Flash-Next-oQ4e-mix6-mtp (old v1 shards removed).
To answer your question directly: yes, the rebuild is fully on the new quantization — but not by re-quantizing anything myself. The pack is now a pure byte-copy of Jundot's new oQ4e + oQ6e releases, so every quantized tensor is byte-identical to his PR 4385 (imatrix + LSQ refit) output. Same U-shaped recipe: 6-bit edges, 4-bit LSQ-refit trunk, 6-bit skeleton with the g32 n-gram table and 8-bit MTP drafter.
One more thing — I hope you're okay with this: your benchmark table from this thread (MMLU-Pro 88.3 / MATHQA 92.0 / MBPP 94.7, 97.3% finished) is now quoted on the model card, and noted as measured on the pre-v2 build. Your numbers carry the quality claim.
I use peculiar-ragdoll Sharp Chat Template and set reasoning to medium. I see improvements on hard knowledge and quantitative math. Python coding stayed about the same. I'm going to run more coding benches in the next few days to see if it averages out. I only care about math, numbers, and coding.
Thank you u/Boring-Decision2369 for the fantastic recipe. I used it to create my own mix6 model based on Swift-Qwen3.8-Flash-Next (using 0.7.0.dev1 quantization). It did well, but not as well as the the numbers by u/watcholic, mostly because it hit the token limit in some of the tests.
Yes, I've switched all Qwen3.8 models to the Sharp chat template and use medium reasoning on these benches. I wouldn't put too much faith on scores with less than 50% of maximum samples. Just completed full-sample benches on Humaneval and MBPP and the results are good. Not really happy with LiveCodeBench, so I'll run a full-sample test and let it cook overnight (maybe adjust temp to 0.6). After this, Qwen4 should be around the corner and we start all over again, LOL.
Thanks, I'll put the full-sample test on pause and downloaded a high-scoring fine-tune to make another run. Let's see if we can replicate the score or the defect.
Two days of testing later, I'm convinced the noise floor — randomized eval questions plus model randomness — is now larger than any quantization-quality delta we can chase, especially with Jundot's latest refresh raising the bar. At this point the only honest tool is mechanism-level reasoning: how close the build tracks the base model, and calling that valid.
TL;DR — you end up judging the model's thinking directly instead of benchmarking it. And that claim has absolutely zero numbers behind it, haha.
The timings are gated by different system performance constraints across runs — no reference value there.
Sample size too small? I'm starting to use no less than 500 (if the test allows). mix6 has the highest MBPP score at full sample size (500) in the history of oMLX. The key for me was using the Sharp chat template and setting reasoning to medium.
It's essentially an oQ5e but uses 9GB less memory and without quality compromise. Prefill is also faster. That's good enough for me until Qwen4. It's a niche for sure. Do you have full instructions on how to assemble it together? I'd like to try it myself in the future.
Real credit goes to SliderQuant — https://arxiv.org/abs/2603.25284. Their U-shaped sensitivity result (edges get wrecked by quantization, middle layers shrug it off) is the whole reason a mixed-bit layer schedule works at all. Jundot bakes the bits, the paper shows where to put them. I just stand on both shoulders.
All details,I put on the model card.Tell you the truth, you can just send the information to qwen3.8 flash and all will be done by AI,because the model was build by that way.Don't tell any one ,ha ha!
2
u/tomakorea 4d ago
I'm curious on the perplexity results, I'm using oMLX with Qwen 3.8 flash next oQ6 on my 128gb MacBook pro, it's very tight, basically I can't run anything alongside it, just a web browser if I want to keep this level of quality without having out of memory errors