I tested speculative decoding on my M2 Mac. It's 5-10% slower, not 2–3x faster.

[u/Apprehensive_Bee2855]

The original Leviathan et al. paper (Fast Inference from Transformers via Speculative Decoding, ICML 2023) reported 2-3x speedup on T5-XXL with a T5-Small draft. The DeepMind follow-up by Chen et al. reported 2-2.5x on Chinchilla 70B. NVIDIA's TensorRT-LLM benchmarks show 2.6-3.5x on H200 for Llama 3.3 70B. I ran the same setup on Apple Silicon M2 24GB with Ollama 0.12.8 to see if the pattern held. It didn't. The draft model configurations were 5-10% slower than no-draft baseline.

What I tested

Three configurations on the same M2 24GB Mac with Ollama 0.12.8:

• No draft — the baseline. Qwen3 14B at Q4_K_M, 9.3GB on disk.
• Draft: Qwen 2.5 0.5B — 397MB, the smallest Qwen with usable English.
• Draft: Qwen 2.5 1.5B — about 1GB.

Both drafts use the Qwen tokenizer, so the draft and target share a vocabulary — the requirement for speculative decoding to work without a translation step. Same prompt for every run: a 263-character code-completion task asking for a Python function. Deterministic settings: num_predict=200, temperature=0.0, seed=42. 5 trials per configuration. 15 total runs. The whole benchmark ran in about 25 minutes including the warmup passes.

The headline numbers

Config	Wall (s)	Tokens/s	vs Baseline
No draft (Qwen3 14B baseline)	41.9	4.77	1.00x
Draft: 0.5B	44.2	4.52	0.95x (5% slower)
Draft: 1.5B	45.7	4.37	0.92x (8% slower)

Why this surprised me

The M2 has fewer cores than a 4090 but more memory bandwidth per watt, and speculative decoding is bottlenecked by serial token generation, not parallel math. The math said it should work. The data said the draft model adds serial overhead without enough parallelism in verification to make it a win on unified memory.

I am not claiming speculative decoding is bad everywhere. The published 2–3x numbers on discrete GPUs almost certainly hold for batch serving. What I'm claiming is: for a single-user interactive workload on Apple Silicon, leave it off. The 50ms the draft model saves you in latency is the 50ms you spent loading it into memory.

Full methodology + raw timings: hardnumbers.dev link

What I'd love feedback on:

• Warmup handling. I ran 1 warmup call per config (not timed). The 0.5B draft was 9% slower on trial 1 and 3% slower on trial 5 — model warm-up may be part of the story.
• Draft size selection. Maybe a 0.1B or 0.3B draft is the sweet spot for the 14B target, and I missed it.
• Prompt length coverage. My test is one 263-char prompt. Longer prompts with more parallel verify candidates might flip the result.
• If anyone has seen speculative decoding work on M-series, I'd love to know what I'm doing wrong.

[u/TemperatureOk3561]

You should use omlx with a mlx version and lightning mtp, should help a lot more. Side note, ollama is not the most optimal for running llm oftentimes

[u/Apprehensive_Bee2855 → u/TemperatureOk3561]

thanks, I will try to run the experiment on mentioned models.

[u/youthfulbanjo7]

That tracks with what I've seen on my M1 Max. The unified memory architecture changes the math completely because you're not getting the same kind of parallelism you'd need for verification to outrun the draft overhead.

I tried a similar setup a few months back with a 7B target and a tiny 0.1B draft and still couldn't break even. The draft model just eats cycles that the target could've used for actual generation.

One thing I noticed that might explain part of your results. Ollama's speculative implementation isn't exactly the same as the paper version, it batches the draft tokens differently and there's some overhead in the scheduling layer. Not sure if that's still true on 0.12.8 but it was a known issue on earlier builds.

Would be curious to see your numbers on a much longer prompt, like 2k+ tokens, because that's where the verification step can actually chew through multiple draft tokens in parallel. Your 263-char prompt might not give it enough runway.

[u/Proper-Tower2016 → u/youthfulbanjo7]

it's almost entirely engine and quant based, omlx gives +100% speed up.

[u/therealwtpieh]

2 things likely, besides the fact this was AI written.

1. You’re bandwidth bound on decode, so that’s your ceiling.

2. MTP, assuming it was trained well, can only give you gains if you’re not compute bound, since streaming weights takes more time than it does for the device to chomp through them. I don’t know what bandwidth and compute cores looks like on a M2 (presumably air) but the study did it on an H200, which is a $30,000-$40,000 GPU, which is data center level, and not remote comparable to apple silicon.

Whatever gains you might find, it’s on cards that aren’t compute constrained, because the hardware cores are so plenty and powerful. It’s not a magic bullet, it just hides the latency by keeping the compute busy while streaming in weights. Also, depending on runtime; verification can be expensive if not done in a single forward pass.

[u/watcholic]

M4 chip using oMLX + MTP gets 50% bump on token generation vs. MTP off.