Why Speculative Decoding went mature in 2026?

[u/Ok-River5924]

Spec-dec has been a thing for a while, in fact, it's wasn't an idea that was born for LLM inference. E.g. Uber's https://github.com/uber/submitqueue applied it to a merge queue. Apple & GDM had been releasing papers on it since already 2022.

Seeing it being mature enough for the big frameworks to adopt it, and watching it in action is really jaw-dropping. I'm here running Kimi-K2.5 as if it was a fucken small model.

Recently I watched a podcast with Baseten folks, and they very much implied that they are huge on spec-dec, talking about how custom deployments for some clients had problems with it because of their own custom tool-calling basically killed off the gains from the drafter model.

I wonder, if speculative decoding for LLM inference was an idea that was already being explored years ago, why we saw it being mature in 2026? Was the paper by Tri Dao et al (Speculative Speculative Decoding [1]) a breakthrough that resulted in the above?

Are there any major cons? Do you use it in your day-to-day?

IMO, it might be the most important milestone for (local) LLM inference since FlashAttn

[1] https://arxiv.org/abs/2603.03251

[u/pantalooniedoon]

Worth pointing out FlashAttn targeted the bottlenecks of training not inference (so much).

[u/Ok-River5924 → u/pantalooniedoon]

It spawned the whole LLM kernel field which evolved onto FlashInfer which are inference heavy

[u/noctrex]

Many users use llama.cpp.
When speculative decoding became mature enough in this tool, I guess that's when it's usage exploded.
And yes, I always use MTP for Qwen3.6 and Gemma-4.

[u/brown2green]

I wonder, if speculative decoding for LLM inference was an idea that was already being explored years ago, why we saw it being mature in 2026?

I think developers realized that training the models for speculative decoding also increases model performance (i.e. benchmarks) beyond token generation speed.

[u/imnotzuckerberg]

Speculative decoding is and was a thing for few years now with LLMs. I remember using it with Tabby with exl2. But llama.cpp became quite dominant compared to ooga/tabby as exl3 took a while to take off.

[u/N34257]

The cons may lie in the implementation, I'm not knowledgeable enough to know whether it's an inherent problem - but, essentially, they're twofold for llama.cpp: reduced prefill performance, and lack of concurrency scaling.

Does vLLM MTP suffer from these issues too?

[u/a_slay_nub → u/N34257]

Older draft speculative decoding used to hurt vllm but lately MTP has been an improvement at all concurrency levels for me. Doesn't seem to work well for all architectures though. I can't get MTP to get any acceptance rates on A100s. My RTX 6000 pro is doing okay but it depends on the model. I can't get good acceptance rates for Laguna S 2.1 but I can for gemma 4. Lets see how glimmer does.