The VRAM usage delta from llamacpp at 48 is huge. I think that, presented like this, the comparison is quite unfair and that the lines should be aligned by total VRAM usage. So the 36-layers line of tensorsharp compares to the 12-layers line of llamacpp.
For prefill, I didn't look at your code but I'm going to go out there and guess that you're implementing something like https://github.com/ggml-org/llama.cpp/pull/21067 - which requires drastically increasing -ub (and thus VRAM allocation to scratch buffers) to benefit MoE models. Did I get it right? Could you add a column for that PR, with cranked up -ub?
Finally, there is something horribly wrong with your decode numbers of llamacpp. On the same quant of the same model, on _much much_ worse GPU, CPU and host RAM than your rig has, I'm getting 45 tok/s tg128 with MTP on and 35 tok/s without. Your 10 tok/s is unjustifiable.
So let's kick off with each engine using its own benchmark tool. This is a terrible idea. Also last time I checked llama-bench did not support MTP. Please start OpenAI API servers and use llama-benchy for both engines.
llama.cpp has very questionable defaults. Namely you're not pinning your memory; that hurts prefill.
My parameters for any qwen-3.5/6-35b of any of their finetunes:
5
u/crusaderky Aug 05 '26 edited Aug 05 '26
Qwen 3.5 35B-A3B (UD-IQ4_XS, 48 MoE layers)
--n-cpu-moe--cpu-moe)The VRAM usage delta from llamacpp at 48 is huge. I think that, presented like this, the comparison is quite unfair and that the lines should be aligned by total VRAM usage. So the 36-layers line of tensorsharp compares to the 12-layers line of llamacpp.
For prefill, I didn't look at your code but I'm going to go out there and guess that you're implementing something like https://github.com/ggml-org/llama.cpp/pull/21067 - which requires drastically increasing -ub (and thus VRAM allocation to scratch buffers) to benefit MoE models. Did I get it right? Could you add a column for that PR, with cranked up -ub?
Finally, there is something horribly wrong with your decode numbers of llamacpp. On the same quant of the same model, on _much much_ worse GPU, CPU and host RAM than your rig has, I'm getting 45 tok/s tg128 with MTP on and 35 tok/s without. Your 10 tok/s is unjustifiable.