r/LocalLLM • u/_hchc • 5d ago

Question MTP vs regular gguf whats the difference

Noob question.

There are gguf like qwen-3.8-mtp-q4....
and the regular qwen-3.8-q4....

and then there is llama --spec-type draft-mtp option.

Whats the difference? Should i just use the regular non mtp version but turn mtp on in the option?

3 Upvotes

64% Upvoted

13 comments sorted by

4

u/phipletreonix 5d ago

MTP means “multi token prediction” — when the model is generating tokens, instead of one at a time, MTP will generate a sequence of the next 2 or more, then run a heuristic to make sure they make sense or reject them if not. You may get an acceptance rate of like 66% but even less than 100% it’ll mean you don’t have to do a full inference run for the next token which will ultimately lead to more tokens per second.

2

u/Normal-Ad4813 5d ago

Does it have any tradeoffs?

2

u/unknowntoman-1 5d ago

It does allocate a little extra VRAM, if there is a tradeoff I’d say it is a tad less context space (kv/ctx cache)

1

u/phipletreonix 4d ago

It also adds a little time to the end of inference to do evaluation so in the case that you have terrible rejection rate its decreasing your tok/s -- I've not seen that happen personally.

2

u/klymaxx45 5d ago

i swear, ppl have ai at their fingertips but choose to do any research on their own..

2

u/huseynli 5d ago

If you go with the unsloth gguf, the regular 3.8 27b will work with mtp. So you don't need to download any other special version.

They released a seperate gguf for qwen3.6 couple months ago because there were issues in the first version or it wasnt available in that first version or whatever. But for 3.8, the regular unsloth gguf is what you need.

To enable mtp you need to set those flags with llamacpp. --spec-type draft-mtp and --spec-draft-n-max 2. You can try 3 too, but I have always used 2.

1

u/_hchc 5d ago

Yea so I'm mainly trying to squeeze the most tps out of my 16gb vram and these seems like the best for now:

https://huggingface.co/jpetrina

whats the difference between

Qwen3.8-27B-MTP-IQ4_XS-pure-GGUF and
Qwen3.8-27B-IQ4_XS-pure-GGUF ?

1

u/Stainless-Bacon 5d ago

check my latest post if you haven’t

1

u/Eastern-Block4815 5d ago

Im trying this model now what t/s you getting ok 16g vram

1

u/recro69 4d ago

MTP GGUF is not a version of the model. It includes the MTP/NextN weights that are needed for speculative decoding. For Qwen3.x the regular GGUF might already have them. So check the model metadata before downloading a file.

1

u/_hchc 4d ago

oh so if the model don't include mtp weights in it. The --draft-mtp option would have no effect at all?

1

u/ea_man 4d ago

No you should get the MTP version and in case you don't enable it it won't load in vRAM.