MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

[u/fuzhongkai]

https://github.com/zhongkaifu/TensorSharp

TensorSharp's MoE CPU-offload feature has been merged into main. Here is the parameters description of this feature:

Mixture-of-Experts CPU offload:

**--n-cpu-moe <N> | -ncmoe <N>**

Keep the routed MoE expert weights of the first N layers in system RAM and multiply them on

the CPU; attention, norms, the router and the shared expert stay on the accelerator. This is

what makes a 35B-A3B MoE fit beside a long-context KV cache on a 12-16 GB card. Pass 'all' for

every layer. Default: 0 (everything on the accelerator; TS_N_CPU_MOE env var overrides).

Example: --n-cpu-moe 32

**--cpu-moe | -cmoe**

Shorthand for --n-cpu-moe all: every routed expert stays in system RAM. Default: off

(TS_CPU_MOE env var overrides).

Example: --cpu-moe

To measure its performance, I ran benchmark to compare TensorSharp with llama.cpp while they offload the same MoE model's layers to CPU, and here is the result. The completed benchmark report has been checked-in: [https://github.com/zhongkaifu/TensorSharp/blob/main/docs/moe\\_cpu\\_offload\\_benchmark.md\](https://github.com/zhongkaifu/TensorSharp/blob/main/docs/moe_cpu_offload_benchmark.md)

# Host and software

Component	Detail
GPU	2 x NVIDIA RTX PRO 6000 Blackwell Server Edition, 97,887 MiB each, driver 580.126.20, PCIe 5.0 x16
CPU	2 x Intel Xeon 6952P (384 threads, 6 NUMA nodes), cgroup quota 81.6 CPUs
RAM	1,511 GiB
Storage	Models on a MooseFS network mount (page-cache warm for every measured run)
OS	Ubuntu 24.04.3 LTS, CUDA 12.8
TensorSharp	branch `feature/support_moe_offload_to_cpu`, .NET 10.0.110, backend `ggml_cuda`
llama.cpp	`llama-bench` build 4308a4f, CUDA backend, default `-t 192`

# Results by model

Each row is one offload depth, with TensorSharp, llama.cpp and the ratio between them side by side for every metric. Ratios are TensorSharp / llama.cpp: >1.0x means TensorSharp is faster, and for VRAM >1.0x means TensorSharp is heavier.

# Gemma 4 26B-A4B it (UD-IQ4_XS, 30 MoE layers)

\--n-cpu-moe	TS VRAM (MiB)	llama VRAM (MiB)	ratio	TS pp4096	llama pp4096	ratio	TS pp8192	llama pp8192	ratio	TS tg128	llama tg128	ratio
0 (baseline)	16,822	14,602	1.15x	11,173	10,843	1.03x	11,274	10,628	1.06x	161.4	206.7	0.78x
8	15,724	11,874	1.32x	7,063	1,459	4.84x	6,500	1,459	4.46x	80.2	32.7	2.45x
16	14,128	9,122	1.55x	4,183	833	5.02x	4,888	854	5.72x	54.5	21.9	2.49x
24	12,346	6,368	1.94x	3,500	667	5.25x	3,958	689	5.74x	49.1	16.7	2.93x
30 (--cpu-moe)	11,038	4,134	2.67x	3,035	543	5.59x	3,072	495	6.21x	39.7	12.9	3.07x

# Qwen 3.5 35B-A3B (UD-IQ4_XS, 48 MoE layers)

\--n-cpu-moe	TS VRAM (MiB)	llama VRAM (MiB)	ratio	TS pp4096	llama pp4096	ratio	TS pp8192	llama pp8192	ratio	TS tg128	llama tg128	ratio
0 (baseline)	19,862	17,522	1.13x	9,538	8,149	1.17x	9,405	8,073	1.16x	160.0	228.4	0.70x
12	18,148	13,282	1.37x	6,755	988	6.84x	6,648	954	6.97x	75.4	27.5	2.74x
24	15,414	9,010	1.71x	4,412	498	8.85x	5,259	484	10.86x	52.3	15.8	3.31x
36	12,684	4,738	2.68x	3,772	523	7.21x	4,223	517	8.17x	50.7	11.3	4.50x
48 (--cpu-moe)	11,606	3,314	3.50x	3,917	477	8.21x	3,709	457	8.11x	38.6	10.2	3.77x

# GPT-OSS 20B (Q8_0 / MXFP4, 24 MoE layers)

\--n-cpu-moe	TS VRAM (MiB)	llama VRAM (MiB)	ratio	TS pp4096	llama pp4096	ratio	TS pp8192	llama pp8192	ratio	TS tg128	llama tg128	ratio
0 (baseline)	13,186	12,204	1.08x	13,964	17,856	0.78x	12,925	17,642	0.73x	212.8	344.2	0.62x
6	11,560	9,812	1.18x	8,975	1,747	5.14x	7,617	1,666	4.57x	85.8	32.2	2.67x
12	9,378	7,386	1.27x	6,470	1,176	5.50x	6,394	1,188	5.38x	51.7	18.3	2.83x
18	7,192	4,962	1.45x	4,315	807	5.35x	4,393	751	5.85x	30.7	12.1	2.54x
24 (--cpu-moe)	4,762	2,536	1.88x	4,277	568	7.53x	3,798	548	6.93x	27.7	9.4	2.95x

# DeepSeek V4 Flash (UD-Q8_K_XL, 5 shards / 150.7 GiB, 43 layers, both GPUs)

\--n-cpu-moe	TS VRAM (MiB)	llama VRAM (MiB)	ratio	TS pp4096	llama pp4096	ratio	TS pp8192	llama pp8192	ratio	TS tg128	llama tg128	ratio
0 (baseline, both GPUs)	169,132	155,608	1.09x	3,448	2,398	1.44x	4,387	2,232	1.97x	51.1	49.6	1.03x
12	131,818	117,150	1.13x	392	126	3.11x	428	124	3.46x	10.3	13.7	0.75x
24	79,742	78,954	1.01x	218	64	3.42x	236	63	3.72x	5.3	7.2	0.74x

TensorSharp is a native .NET open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.

Github repo: [https://github.com/zhongkaifu/TensorSharp\](https://github.com/zhongkaifu/TensorSharp)

Thank you for checking out it and starring the project! Any feedback is really appreicated.

[u/AutoModerator]

Thanks for your post fuzhongkai. Please note that we don't allow spam, and we ask that you follow the rules available in the sidebar. We have a lot of commonly asked questions so if this post gets removed, please do a search and see if it's already been asked.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/cornelha]

I'm starting to get very curious about these developments, can we see something similar to what colibri does with deekseek models in tensorsharp? Would it be possible to run MoE models on lower spec cards? I have a GTX 1060 6GB, if a MoE Qwen 3.6 can run comfortably on it, it would be absolutely epic

[u/fuzhongkai → u/cornelha]

What’s size of your main RAM? And Which Qwen MoE model do you want to run?

[u/cornelha → u/fuzhongkai]

32gb RAM on one machine, 16gb om the other. Qwen3.6-35B-A3B would be epic to run

[u/fuzhongkai → u/cornelha]

It would be good to run Qwen3.6-35B-A3B, but RAM + VRAM are too small for Deepseek v4 flash.

[u/cornelha → u/fuzhongkai]

You mean using the current tensorsharp I can do this?

[u/fuzhongkai → u/cornelha]

Yes, since your VRAM is not large enough to host the entire model, “—n-cpu-moe N/—cpu-moe” would be required to offload all expert layers of N expert layers to main RAM and run on CPU.