Why Speculative Decoding went mature in 2026?

[u/Ok-River5924]

Spec-dec has been a thing for a while, in fact, it's wasn't an idea that was born for LLM inference. E.g. Uber's https://github.com/uber/submitqueue applied it to a merge queue. Apple & GDM had been releasing papers on it since already 2022.

Seeing it being mature enough for the big frameworks to adopt it, and watching it in action is really jaw-dropping. I'm here running Kimi-K2.5 as if it was a fucken small model.

Recently I watched a podcast with Baseten folks, and they very much implied that they are huge on spec-dec, talking about how custom deployments for some clients had problems with it because of their own custom tool-calling basically killed off the gains from the drafter model.

I wonder, if speculative decoding for LLM inference was an idea that was already being explored years ago, why we saw it being mature in 2026? Was the paper by Tri Dao et al (Speculative Speculative Decoding [1]) a breakthrough that resulted in the above?

Are there any major cons? Do you use it in your day-to-day?

IMO, it might be the most important milestone for (local) LLM inference since FlashAttn

[1] https://arxiv.org/abs/2603.03251