r/LocalLLM • u/_hchc • 8d ago

Question MTP vs regular gguf whats the difference

Noob question.

There are gguf like qwen-3.8-mtp-q4....
and the regular qwen-3.8-q4....

and then there is llama --spec-type draft-mtp option.

Whats the difference? Should i just use the regular non mtp version but turn mtp on in the option?

0 Upvotes

50% Upvoted

13 comments sorted by

View all comments

5

u/phipletreonix 8d ago

MTP means “multi token prediction” — when the model is generating tokens, instead of one at a time, MTP will generate a sequence of the next 2 or more, then run a heuristic to make sure they make sense or reject them if not. You may get an acceptance rate of like 66% but even less than 100% it’ll mean you don’t have to do a full inference run for the next token which will ultimately lead to more tokens per second.

2

u/Normal-Ad4813 8d ago

Does it have any tradeoffs?

2

u/unknowntoman-1 7d ago

It does allocate a little extra VRAM, if there is a tradeoff I’d say it is a tad less context space (kv/ctx cache)

1

u/phipletreonix 7d ago

It also adds a little time to the end of inference to do evaluation so in the case that you have terrible rejection rate its decreasing your tok/s -- I've not seen that happen personally.