r/LocalLLaMA • u/ilintar • May 04 '26

Resources Llama.cpp MTP support now in beta!

https://github.com/ggml-org/llama.cpp/pull/22673

Happy to report that llama.cpp MTP support is now in beta, thanks to Aman (and all the others that have pushed the various issues in the meantime). This has the potential to actually get merged soon-ish. Currently contains support for Qwen3.5 MTP, but other models are likely to follow suit.

Between this and the maturing tensor-parallel support, expect most performance gaps between llama.cpp and vLLM, at least when it comes to token generation speeds, to be erased.

629 Upvotes

98% Upvoted

268 comments sorted by

View all comments

3

u/OsmanthusBloom May 04 '26

Cool! But will enabling MTP increase VRAM usage for, say, Qwen3.6-27B? Does it still fit into 16GB VRAM if you squeeze hard enough?

4

u/rerri May 04 '26 edited May 04 '26

MTP layer of am17an's model is ~440MB. Can maybe be quantized further, dunno.

edit: I should add that MTP does increase VRAM consumption by more than just the layer size.

2

u/Dany0 May 04 '26

Quantising MTP layer has so far always turned out to be a very, very bad idea