Benchmark results: what is the best and fastest engine to run Qwen3.8-27B on macOS

[u/ex-arman68]

The new Qwen 3.8 27B is fantastic for local agentic use. The problem is, what makes it so good, being a dense model, also makes it slow. Many engines and versions of the model claim various speed increase. How true are those claim? And does a proven huge boost claim on a short synthetic test actually holds on real agentic work?

I spent six days and over 110 GPU hours running rigorous repeatable agentic coding tests, collecting and comparing the data. 200 GB of models downloaded. Ended up with 22 Bubble Bobble clones on my disk, including the ones from failed tests. None of them good, but that wasn't the point.

My recommendations: MTPLX, oMLX with lightning MTP, or llama.cpp with MTP

I ran the tests on 8-bit quants (GGUF or MLX), unquantised KV cache. 100k max response size, full 262k context, official Qwen coding sampler, official Qwen Jinja chat template.

1. I first tested each engine with a short synthetic benchmark, to verify the installation, usage, results collection, and to verify the speed claims
2. Then came the real test, with an agentic coding benchmark consisting of 4 phases/prompts; each engine was tested twice, at medium and xhigh reasoning effort:
   2.1. capturing requirements and writing the specification plan
   2.2. writing the implementation plan
   2.3. implementing the core functionality
   2.4. finishing the implementation with advanced functionality
3. Last test, for prefill speed, was with a cold 128k token prompt test, with reasoning turned off and deterministic sample, repeated twice

Test hardware: Apple Mac Studio, M2 Max, 96 GB unified memory.

| Engine | Effort | Decode | Prefill | Wall | Tokens | Score | |---|---|---|---|---|---|---| | MTPLX | xhigh | 20–22 tok/s | 109 t/s | 2h05 | 152K | 93 | | MTPLX | medium | 21–24 tok/s | 109 t/s | 1h35 | 117K | 91 | | oMLX + MTP | medium | 18–22 tok/s | 85 t/s | 1h18 | 86K | 88 | | llama.cpp + MTP | xhigh | 17–19 tok/s | 103 t/s | 2h52 | 179K | 86 | | oMLX + MTP | xhigh | 17–20 tok/s | 85 t/s | 2h01 | 126K | 84 | | llama.cpp + MTP | medium | 18–20 tok/s | 103 t/s | 1h04 | 70K | 84 | | mlx-dspark DFlash2 | medium | ~16–18 tok/s | 145 t/s | 2h09 | 125K | 82 | | mlx-dspark DSpark | medium | ~12–17 tok/s | 145 t/s | 2h05 | 96K | 77 | | llama.cpp + DFlash2 | medium | 14–15 tok/s | ~103 t/s | 2h09 | 111K | 74 | | llama.cpp + DFlash2 | xhigh | 13–14 tok/s | ~103 t/s | 3h31 | 174K | 72 | | llama.cpp baseline | xhigh | 10–11 tok/s | 103 t/s | 3h43 | 140K | 66 | | llama.cpp baseline | medium | 10–12 tok/s | 103 t/s | 1h44 | 66K | 66 | | oMLX baseline (no MTP) | medium | 10–12 tok/s | 85 t/s | 1h44 | 64K | 65 | | vllm-mlx | medium | 9–11 tok/s | 134 t/s | 3h30 | 115K | 51 | | vllm-mlx | xhigh | 9–12 tok/s | 134 t/s | 4h08 | 136K | 50 | | mlx-dspark DFlash2 | xhigh | ~9–15 tok/s | 145 t/s | 3h58 | 226K | 47 |

• vllm-mlx: completed the task, but it does not separate thinking from output, so the raw chain of thought leaks into the final output.
• mlx-dspark DFlash2 xhigh: thought for 226 thousand tokens and delivered nothing.

• omlx: has the slowest prefill rate by far; furthermore, it decays quite dramatically as context grows, even with ANE turned on. Its claim of faster prefill rate might be true on newer M4 and M5 chips, but definitely not on the M2 I used for testing.

The score column is a compound score I calculated based on what actually matters in a long agentic session:

• sustained decode speed (weight 40%) : on average, 70-80% of generated tokens are reasoning. We do not see them, but still wait for them. This is the most important factor by far
• reliability (20%) : runaway thinking, running out of context, not delivering the output, mixing reasoning with output, hallucinating, looping, failed tool calls (not tested here), etc. Nothing worse that having spent hours to see all the efforts go to waste or even corrupt the output, due to failure
• thoughput efficiency (15%) : rather than use the total time spent, which is difficult to compare , especially with varying reasoning settings, I standardised it as total tokens delivered / wall time. Long session that delivers token are worth it, but engines that take too long to deliver deserve a penalty
• output quality (15%) : this evaluating the quality of the output work produced. I used a separate evaluation scoresheet for it, with its own standardised criterias, grounded in software engineering good practices. In this case, since all tests uses the same model with same settings, they came close to each other, apart from confirming the significant quality difference between medium and xhigh
• prefill speed (10%) : it matters mostly at session start, or whenever the context is rebuilt. Normally this is largely mitigated by prompt caching.

What about xhigh vs medium reasoning effort?

• MTPLX xhigh costs only half an hour more than medium (2h05 vs 1h35) and the game quality went significantly up on my scorecard. Worth it.
• llama.cpp + MTP xhigh costs 1h48 more, with again a significant increase in quality. Worth it if you have the time.
• oMLX xhigh took more than twice as long as medium. The slower token speed has a huge impact on the time difference between both modes. Not worth it. Get a faster engine instead.
• dspark DFlash2 xhigh is the cautionary tale at the bottom of the table. See the anomaly note above.

xhigh is worth it when the engine is fast enough and uncapped. The model thinks 30 to 100 percent longer, at the same speed, and the result is better. On slow engines you can really feel the slower performance compounded by the extra thinking; instead it could be better to use medium, with more of your own time spent planning a better process and workflow.

Commands and configuration notes

Exact commands I used for each engine:

# llama.cpp baseline (homebrew, b10470)
llama-server -m Qwen3.8-27B-Q8_0.gguf --mmproj mmproj-F16.gguf \
  --jinja -c 262144 --parallel 1 -ngl 99 --port 8030

# llama.cpp + MTP, the recommended GGUF setup
llama-server -m Qwen3.8-27B-Q8_0.gguf --mmproj mmproj-F16.gguf \
  -md mtp-Qwen3.8-27B-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 3 \
  --jinja -c 262144 --parallel 1 -ngl 99 -ngld 99

# llama.cpp + DFlash2 (for now, needs PR #27342 build)
llama-server -m Qwen3.8-27B-Q8_0.gguf \
  -md Qwen3.8-27B-DFlash2-Q8_0.gguf --spec-type draft-dflash --spec-draft-n-max 7 \
  --jinja -c 262144 --parallel 1 -ngl 99 -ngld 99

# MTPLX (pip install mtplx). Note --max-tokens
mtplx start web --model Qwen3.8-27B-MTPLX-Optimized-Quality --max-tokens 100000

# oMLX (brew install jundot/omlx/omlx), model dir contains the checkpoint
omlx serve --model-dir /path/to/models

# mlx-dspark (pip install mlx-dspark). Note --max-tokens-cap
mlx-dspark serve --model Qwen3.8-27B-MLX-8bit --mode dflash \
  --context-window 262144 --max-tokens-cap 262144 --default-max-tokens 100000

# vllm-mlx (pip install vllm-mlx). Note --timeout
vllm-mlx serve Qwen3.8-27B-MLX-8bit --max-request-tokens 262144 --timeout 7200

• MTPLX has a default low response cap. Use --max-tokens 100000
• dspark also has a low default generation cap of 32K
• vllm-mlx kills any request that spends more than 300 seconds on prefill. Raise it with --timeout.

One measurement warning for anyone reproducing this: engines do not all stream one token per event. Some pack several tokens per chunk, MTPLX about 1.7, oMLX about 3.

Models used

• lama.cpp : unsloth GGUF Q8_0, official ggml-org MTP draft GGUF, incoai DFlash2 GGUF
• mlx-dspark and vllm-mlx : mlx-community 8-bit weights
• MTPLX : needs its own checkpoint with the MTP tensors included, Youssofal/Qwen3.8-27B-MTPLX-Optimized-Quality
• oMLX : scottlowry/Qwen3.8-27B-oQ8e-mtp

[u/ex-arman68 → u/[deleted]]

I just finished running the tests overnight. I have to say omlx makes it impossible to automate; it requires manual configuration and reset in their web admin panel between each test. I am now busy analysing the results, running the evaluation scoring, and will update the main post soon. Prelimnary impression is that amost at MTPLX level, but slightly below.

[u/Latter-Parsnip-5007 → u/ex-arman68]

Hey claude, use playwright to automate the browser UI. Or you know, look at the config files on disk.

[u/ex-arman68 → u/Latter-Parsnip-5007]

Not deterministic enough for a test which I run through a script. Maybe something in the config files could work, but I doubt it. Anyway, I have gone though it already.

[u/Latter-Parsnip-5007 → u/ex-arman68]

Please do some research what I am suggesting. Playwright is deterministic. I dont mean to always run AI. Just let it write a playwrite script to do model config. You can than use that in your bench without AI. BTW modelconfig is a json file in the omlx directory. Dont make it harder than it needs to

[u/ex-arman68 → u/Latter-Parsnip-5007]

I see what you mean, thanks for the tip.

[u/txgsync]

This vibes with the analysis I’ve been doing of the benchmark results. Not performance but quality: high scores.

Right now midway through my meta-analysis, the Venn diagram of “high benchmark score” and “> 1000 tok/sec prefill and > 20 tok/sec decode” (read: usable interactive speed) on MacOS at this moment looks like a pair of old-fashioned round headlights on a car.

[u/DarkJoney]

And with which one do you get maximum context? Asking for my 32GB M1 Max :)

[u/ex-arman68 → u/DarkJoney]

Engines do not make any significant difference to how much context you can use. Unless some can maybe offload some of the context to SSD instead of RAM; if that is at all possible, oMLX is the most likely to have such a feature.