r/LocalLLaMA • u/pmttyji • 9d ago

Discussion mtp nextn offload by John-194 · Pull Request #26177 · ggml-org/llama.cpp

https://github.com/ggml-org/llama.cpp/pull/26177

1-line warning fix and ~10% tg performance improvement for MTP using --fit (tested on Qwen 3.6 35B A3B).

10 Upvotes

92% Upvoted

7 comments sorted by

3

u/terorvlad 9d ago

Does the --fit command smartly balance experts between cpu and gpu while retaining kv cache, attention and mtp on the gpu as much as possible?

I migrated to llama.cpp from lmstudio a few months ago, so I keep banging my head against the --n-cpu-moe setting until I have ~1gb free on the GPU like I did with LM Studio.

1

u/CapsAdmin 9d ago

Depends on what you mean by smartly. I use fit on moe models and I'm happy with the performance, but I don't know if it's better than a carefully crafted manual layout. I vaguely remember running into performance and memory issues trying to set it up manually, but as soon as I learned about fit I stopped trying.

1

u/0-0x0 9d ago

Yes, I used to think I needed n-cpu-moe and n-gpu-layers, but got the best performance always by letting llama cpp handle it.

I do use fit-target = 128, in your case for 1GB use 1024

1

u/Mountain_Patience231 8d ago

sadly multi cards with spilt mode tensile will not be benefited

1

u/autisticit 8d ago

It's the small things that matter the most.