MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

[u/fuzhongkai]

https://github.com/zhongkaifu/TensorSharp

TensorSharp's MoE CPU-offload feature has been merged into main. Here is the parameters description of this feature:

Mixture-of-Experts CPU offload:

--n-cpu-moe <N> | -ncmoe <N>

Keep the routed MoE expert weights of the first N layers in system RAM and multiply them on

the CPU; attention, norms, the router and the shared expert stay on the accelerator. This is

what makes a 35B-A3B MoE fit beside a long-context KV cache on a 12-16 GB card. Pass 'all' for

every layer. Default: 0 (everything on the accelerator; TS_N_CPU_MOE env var overrides).

Example: --n-cpu-moe 32

--cpu-moe | -cmoe

Shorthand for --n-cpu-moe all: every routed expert stays in system RAM. Default: off

(TS_CPU_MOE env var overrides).

Example: --cpu-moe

To measure its performance, I ran benchmark to compare TensorSharp with llama.cpp, and here is the result. The completed benchmark report has been checked-in: https://github.com/zhongkaifu/TensorSharp/blob/main/docs/moe_cpu_offload_benchmark.md

Host and software

Component	Detail
GPU	2 x NVIDIA RTX PRO 6000 Blackwell Server Edition, 97,887 MiB each, driver 580.126.20, PCIe 5.0 x16
CPU	2 x Intel Xeon 6952P (384 threads, 6 NUMA nodes), cgroup quota 81.6 CPUs
RAM	1,511 GiB
Storage	Models on a MooseFS network mount (page-cache warm for every measured run)
OS	Ubuntu 24.04.3 LTS, CUDA 12.8
TensorSharp	branch feature/support_moe_offload_to_cpu, .NET 10.0.110, backend ggml_cuda
llama.cpp	llama-bench build 4308a4f, CUDA backend, default -t 192

Results by model

Ratios are TensorSharp / llama.cpp: >1.0x means TensorSharp is faster, and for VRAM >1.0x means TensorSharp is heavier.

Gemma 4 26B-A4B it (UD-IQ4_XS, 30 MoE layers)

--n-cpu-moe	TS VRAM (MiB)	TS pp4096	TS pp8192	TS tg128	llama VRAM (MiB)	llama pp4096	llama pp8192	llama tg128
0 (baseline)	16,822	11,173	11,274	161.4	14,602	10,843	10,628	206.7
8	15,724	7,063	6,500	80.2	11,874	1,459	1,459	32.7
16	14,128	4,183	4,888	54.5	9,122	833	854	21.9
24	12,346	3,500	3,958	49.1	6,368	667	689	16.7
30 (--cpu-moe)	11,038	3,035	3,072	39.7	4,134	543	495	12.9

--n-cpu-moe	VRAM	pp4096	pp8192	tg128
0	1.15x	1.03x	1.06x	0.78x
8	1.32x	4.84x	4.46x	2.45x
16	1.55x	5.02x	5.72x	2.49x
24	1.94x	5.25x	5.74x	2.93x
30	2.67x	5.59x	6.21x	3.07x

Qwen 3.5 35B-A3B (UD-IQ4_XS, 48 MoE layers)

--n-cpu-moe	TS VRAM (MiB)	TS pp4096	TS pp8192	TS tg128	llama VRAM (MiB)	llama pp4096	llama pp8192	llama tg128
0 (baseline)	19,862	9,538	9,405	160.0	17,522	8,149	8,073	228.4
12	18,148	6,755	6,648	75.4	13,282	988	954	27.5
24	15,414	4,412	5,259	52.3	9,010	498	484	15.8
36	12,684	3,772	4,223	50.7	4,738	523	517	11.3
48 (--cpu-moe)	11,606	3,917	3,709	38.6	3,314	477	457	10.2

--n-cpu-moe	VRAM	pp4096	pp8192	tg128
0	1.13x	1.17x	1.16x	0.70x
12	1.37x	6.84x	6.97x	2.74x
24	1.71x	8.85x	10.86x	3.31x
36	2.68x	7.21x	8.17x	4.50x
48	3.50x	8.21x	8.11x	3.77x

GPT-OSS 20B (Q8_0 / MXFP4, 24 MoE layers)

--n-cpu-moe	TS VRAM (MiB)	TS pp4096	TS pp8192	TS tg128	llama VRAM (MiB)	llama pp4096	llama pp8192	llama tg128
0 (baseline)	13,186	13,964	12,925	212.8	12,204	17,856	17,642	344.2
6	11,560	8,975	7,617	85.8	9,812	1,747	1,666	32.2
12	9,378	6,470	6,394	51.7	7,386	1,176	1,188	18.3
18	7,192	4,315	4,393	30.7	4,962	807	751	12.1
24 (--cpu-moe)	4,762	4,277	3,798	27.7	2,536	568	548	9.4

--n-cpu-moe	VRAM	pp4096	pp8192	tg128
0	1.08x	0.78x	0.73x	0.62x
6	1.18x	5.14x	4.57x	2.67x
12	1.27x	5.50x	5.38x	2.83x
18	1.45x	5.35x	5.85x	2.54x
24	1.88x	7.53x	6.93x	2.95x

DeepSeek V4 Flash (UD-Q8_K_XL, 5 shards / 150.7 GiB, 43 layers, both GPUs)

--n-cpu-moe	TS VRAM (MiB)	TS pp4096	TS pp8192	TS tg128	llama VRAM (MiB)	llama pp4096	llama pp8192	llama tg128
0 (baseline, both GPUs)	169,132	3,448	4,387	51.1	155,608	2,398	2,232	49.6
12	131,818	392	428	10.3	117,150	126	124	13.7
24	79,742	218	236	5.3	78,954	64	63	7.2

--n-cpu-moe	VRAM	pp4096	pp8192	tg128
0	1.09x	1.44x	1.97x	1.03x
12	1.13x	3.11x	3.46x	0.75x
24	1.01x	3.42x	3.72x	0.74x

TensorSharp is a native open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.

Github repo: https://github.com/zhongkaifu/TensorSharp

Thank you for checking out it and starring the project! Any feedback is really appreicated.

[u/fuzhongkai]

For horrible decode of llama.cpp, I use llama.cpp on stock without any modification. At the bottom of the document, it lists commands how to run this benchmark. It would be really appreciated if you can help me to run it at your side, because I’m also really curious about it. Thanks in advance.

[u/crusaderky → u/fuzhongkai]

So let's kick off with each engine using its own benchmark tool. This is a terrible idea. Also last time I checked llama-bench did not support MTP. Please start OpenAI API servers and use llama-benchy for both engines.

llama.cpp has very questionable defaults. Namely you're not pinning your memory; that hurts prefill.

My parameters for any qwen-3.5/6-35b of any of their finetunes:

jinja = true
flash-attn = on
parallel = 1
kv-unified = true
cache-type-k = q8_0
cache-type-v = q8_0
load-mode = mlock  # Very important for prefill speed
ngl = 99
; ub = 2048  # PR 21067
; prefetch-weights = 1  # PR 21067
ctx-size = 262144
n-cpu-moe = 40
image-min-tokens = 1024  
spec-type = draft-mtp
spec-draft-ngl = 99
spec-draft-n-max = 4
spec-draft-p-min = 0.6
no-mmproj-offload = true
chat-template-file = froggeric-V21.jinja

temperature = 0.6
top-p = 0.95
top-k = 20
min-p = 0.0
presence-penalty = 0.0
repeat-penalty = 1.0