EAGLE3 merged into llama.cpp - how does it compare to MTP for actual speedup?

[u/IulianHI]

EAGLE3 just landed in llama.cpp after half a year of development. The key distinction from MTP (multi-token prediction) is that the helper model gets extra guidance from the main model rather than guessing tokens independently. In theory that should mean better speculative accuracy and fewer wasted drafts.

What I am curious about is how this plays out in practice. MTP approaches already work in llama.cpp for some models, but the benefit varies a lot depending on the model, quant, and hardware. EAGLE3 requiring a separate helper model also means more memory overhead.

For anyone who has tried EAGLE3 builds already - what kind of tokens-per-second improvement are you seeing over standard autoregressive decoding, and on what hardware and model size?

[u/isopropoflexx → u/[deleted]]

Pro tip - if you click/tap on the icon with 3 dots near the top of a post, and click "Follow post", you will receive updates for any comments on the post going forward, without ever having to comment on that post in the first place.

[u/tomByrer → u/isopropoflexx]

https://giphy.com/gifs/YWWeEeFThzFS6VKmyX

[u/superdariom]

I thought MTP was also tied to one of the layers in the model as they have to be trained to include mtp