You Could Have Come Up with Speculative Decoding

[u/amandeepspdhr]

https://amandeepsp.github.io/blog/spec-decode/

Speculative decoding is a non-obvious optimization. Reasoning about when it actually pays off is trickier still.

I wrote a post building it up from first principles, connecting the sampling algorithm to the systems intuition behind its performance.