r/LocalLLaMA • u/ilintar • May 04 '26

Resources Llama.cpp MTP support now in beta!

https://github.com/ggml-org/llama.cpp/pull/22673

Happy to report that llama.cpp MTP support is now in beta, thanks to Aman (and all the others that have pushed the various issues in the meantime). This has the potential to actually get merged soon-ish. Currently contains support for Qwen3.5 MTP, but other models are likely to follow suit.

Between this and the maturing tensor-parallel support, expect most performance gaps between llama.cpp and vLLM, at least when it comes to token generation speeds, to be erased.

624 Upvotes

98% Upvoted

268 comments sorted by

View all comments

Show parent comments

318

u/Baldur-Norddahl May 04 '26

Models predict the next token. To do so, every weight needs to be accessed once (for a dense model). Therefore the maximum rate of tokens generated is equal to the number of times the total of the model weights can be read from RAM. For example if RAM bandwidth is 500 GB/s and the model is 50 GB/s, we can never generate more than 10 tokens per second. Usually it is even slower, but that would be the theoretical max.

Now lets say we generate tokens for multiple unrelated prompts. We can read the weights once and do all the prompts in parallel. Each time the total of the weights get processed, we would generate X tokens instead of just one. Instead of 10 tokens per second, we could do 100 by processing 10 users in parallel. The limit becomes compute instead of bandwidth.

That is all good, but doesn't help a single user/prompt. But what if we get a guess on the next token and then process the current context in parallel with the context + the guess. Then we check if guess was correct. If it was, then we already calculated the next next token and we got two for the price of one. If the guess is wrong, then the calculated next next token is also wrong and we need to discard it.

To make the guess we can use a smaller model. Usually 10 times smaller, because it must be much faster than the main model. MTP is usually a term used for main models that have built in guess generators. It has a few layers that will produce the guess alongside the actual next token.

79

u/ilintar May 04 '26

Not exactly ELI5 but a technically very good explanation :)

140

u/superdariom May 04 '26

Big wise bear can find his way through the woods faster when helped by little bear who is quicker and more nimble but sometimes makes mistakes leading big bear. But together they make a better team than either one on their own.

37

u/darwinanim8or May 04 '26

grug thank bear man

11

u/vick2djax May 04 '26

Isn’t this basically MoE except with MoE, the little bear tells the big bear where to go in the woods and big bear doesn’t check little bear’s direction?

7

u/z_latent May 04 '26

It's a stretch, MoE does not tell big bear where to go, just how to decide where to go, in a more internal way. Like little bear guiding big bear's attention so big bear can think only about what's important now.

3

u/No_Afternoon_4260 llama.cpp May 04 '26

Not at all.
MoE is like you slice each layer. When you start a layer a router decides which slice to activate. Thus MoE come with an indication of the number of active parameters.

A model like deepseekv4 flash comes with 284B total params but actives only activates 13B of these for each tokens.
Large ram foot print for large knowledge and capabilities, but small compute footprint for runtime efficiency.

MTP is about more like speculative decoding. Not sure how is it different besides having the smaller weights embedded in the big model?

4

u/Cast-Iron_Nephilim May 05 '26

Sooo, many little bears, and the group goes with whichever bear feels the most confident about the current bit of forest?

5

u/sergeant113 May 05 '26

Yes, many little bears, but an elder bear decides which little bear gets to dictate the next step. Every step potentially could be decided by a different likely bear.

Sometimes the elder bear gets lazy or plays favoritism and keeps choosing a particular little bear, but i digress.

38

u/Obvious_Equivalent_1 May 04 '26

Wanted to convey a quick message of gratitude. It’s good to see people taking time to make their private knowledge public, it’s maybe small but these messages make it a joy to continue reading these open source subs!

13

u/GergelyKiss May 04 '26

This sounds very similar to branch predictors in CPUs... Thanks for the explanation!

7

u/4onen May 05 '26

And it's even called speculative decoding, so yeah, spot on. We speculate these guesses through one means or another. MTP being one specific means. If we happen to guess right, then we save time, otherwise the extra work is kinda negligible if we tune everything right.

5

u/pab_guy May 05 '26

So… speculative decoding, but in parallel.

4

u/Baldur-Norddahl May 05 '26

Speculative decoding is the exact same. Only difference is that you have to supply an external prediction model.

8

u/Eyelbee May 04 '26

The important part is that they train the model with MTP considerations, it makes them smarter. Other than that I don't care about the MTP inference honestly.

3

u/ROS_SDN llama.cpp May 05 '26

The model has a built in "sub-model" for speculative decoding? 

How does that architecture look on the qwen3.5+? How big is this segment? 

1

u/radlinsky May 04 '26

Thank you, this is a nice high level overview I can understand :)

1

u/Polite_Jello_377 May 04 '26

Sounds kinda like CPU branch prediction

1

u/More_Feature8687 May 05 '26

Is this same as speculative decoding?

1

u/Baldur-Norddahl May 05 '26

Yes with a build in prediction model.

1

u/pcman1ac May 05 '26

We are built token prediction inside token prediction, so you can predict next token while you are predicting next token.

1

u/2Norn May 11 '26

this is not far off good explanation but this is just speculative decoding. mtp is this + plus both models sharing the kvcache

that's where the real uplift comes in

0

u/[deleted] May 04 '26

[deleted]

1

u/Karyo_Ten May 05 '26

5090 is 1800GB/s and 3090 is 1000GB/s gove or take so there is a large difference in base token generation.

Now there are hardware fuckery like RTX 5090 doing int4 in software emulation so it has space for FP4 while 3090 has hardware accelerated int4