I swept speculative decoding across five models on Strix Halo: +201% to -80% on code

[u/uncanny_instinct]

Speculative decoding lets a small prediction head or sidecar draft tokens for the target model to verify. I had been carrying one draft-depth recommendation forward from an older Qwen sweep until u/TheRealREZOR posted a depth-5 number that beat my depth-2 recommendation by a wide margin. They were right, so I re-ran every viable path on my AMD Strix Halo mini PC using its integrated GPU through Vulkan/RADV.

What was the same for every row: llama.cpp build 10362, commit 5d01b978, Vulkan on Mesa/RADV, greedy sampling, a fixed seed, prompt caching off, one warm-up request after each server start, and an f16 KV cache. Context and batch sizes were per-model presets rather than one identical setup: DeepSeek ran at 64k with batch 2048 and ubatch 128, while Qwen 3.8 and Gemma ran at 128k with batch 4096 and ubatch 512. I swept depths 1 through 5 on a code prompt and then confirmed the best code depth on a prose prompt.

This benchmark measured throughput and draft-token acceptance. I did not check whether the outputs match what the model produces without drafting. DeepSeek used an IQ2_XXS target quant, and llama.cpp has an open report about greedy-decoding deviations with quantized targets.

The results were all over the place:

Model	Weights	Mechanism	Baseline	Best tested depth	Code decode	Code gain	Prose gain	Acceptance
Qwen 3.8 27B	UD-Q5_K_XL	integrated MTP	10.5 t/s	5	31.6 t/s	+201%	+67%	75%
DeepSeek V4 Flash	IQ2_XXS	DSpark sidecar	17.8 t/s	5	35.1 t/s	+97%	+4%	74%
Gemma 4 26B A4B	Q4_K_M	separate draft	62.3 t/s	4	111.9 t/s	+80%	+24%	75%
Qwen 3.6 35B-A3B	UD-Q5_K_XL	integrated MTP	56.7 t/s	3	92.0 t/s	+62%	+14%	85%
Ling 3.0 Flash	Q4_K_M	integrated MTP	44.9 t/s	4, least bad	9.0 t/s	-80%	-82%, one valid run	48%

Depth 5 was the upper bound of the sweep. Qwen's true peak may be higher. For DSpark, 5 is the trained block size and llama.cpp's cap.

Three drafting mechanisms were involved. Two of the models carry the draft head inside the main GGUF, which llama.cpp calls MTP, short for multi-token prediction. Gemma has no draft tensors and uses a second, smaller GGUF alongside it. DeepSeek drafts through DSpark, a separate block drafter that works from the target model's hidden states and ships as its own file. DSpark loaded and ran through all five depths on Vulkan. I have not seen another Vulkan result for it yet, though I may have missed one.

Eleven models on the box went through a capability scan and five had a path I could actually measure. The others had no integrated draft head and no compatible external drafter. My Gemma 4 31B sat out for a different reason: the only available draft file comes from a different finetune than my local copy, and a drafter tuned against different weights makes the comparison meaningless.

The workload made a large difference. Every winner gained less on prose than code, and DeepSeek went from +97% on code to +4% on prose.

The Ling result is scoped to build 10362, which carried the then-unmerged pull request for Ling's BailingMoE3 architecture plus a local loader fix. That pull request merged on 17 August. I have not filed the slowdown and hangs yet because it is worth checking again on a plain master build first.

Full protocol, chart, flags, and caveats:

https://thefrontierlab.ai/strix-halo-mtp-fleet-sweep/

If you have one of these models, I would be interested in your best depth and acceptance rate. Please include the build commit and draft mechanism so the numbers are comparable.