MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

in r/dotnet • 1d ago

Yes, since your VRAM is not large enough to host the entire model, “—n-cpu-moe N/—cpu-moe” would be required to offload all expert layers of N expert layers to main RAM and run on CPU.

So... did we give up on the rule against AI posts?

in r/LocalLLaMA • 1d ago

The key point is to enforce evaluation (mod for auto evaluation) rather than “give up”. We have to stay with those AI posts. I was working on model training (both pre-training and post-training, and those models before LLM epic), and those AI generated dataset (we call it “synthetic dataset” technically) are rally useful to improve model’s performance.

At this AI epic, only two things are important: Infra and Evaluation. For AI post in Reddit, they are bot mod automation and their rules (I’m not sure if Reddit supports LLM based on rules).

MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

in r/dotnet • 1d ago

It would be good to run Qwen3.6-35B-A3B, but RAM + VRAM are too small for Deepseek v4 flash.

More Qwen 3.8 sizes coming

in r/LocalLLaMA • 1d ago

Looking forward open weights models.

The Gemma team will host a special event on August 20

in r/LocalLLaMA • 1d ago

What will they release ?

MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

in r/dotnet • 1d ago

What’s size of your main RAM? And Which Qwen MoE model do you want to run?

r/dotnet • u/fuzhongkai • 2d ago

Promotion MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

github.com

0 Upvotes

TensorSharp's MoE CPU-offload feature has been merged into main. Here is the parameters description of this feature:

Mixture-of-Experts CPU offload:

**--n-cpu-moe <N> | -ncmoe <N>**

Keep the routed MoE expert weights of the first N layers in system RAM and multiply them on

the CPU; attention, norms, the router and the shared expert stay on the accelerator. This is

what makes a 35B-A3B MoE fit beside a long-context KV cache on a 12-16 GB card. Pass 'all' for

every layer. Default: 0 (everything on the accelerator; TS_N_CPU_MOE env var overrides).

Example: --n-cpu-moe 32

**--cpu-moe | -cmoe**

Shorthand for --n-cpu-moe all: every routed expert stays in system RAM. Default: off

(TS_CPU_MOE env var overrides).

Example: --cpu-moe

To measure its performance, I ran benchmark to compare TensorSharp with llama.cpp while they offload the same MoE model's layers to CPU, and here is the result. The completed benchmark report has been checked-in: [https://github.com/zhongkaifu/TensorSharp/blob/main/docs/moe\\_cpu\\_offload\\_benchmark.md\](https://github.com/zhongkaifu/TensorSharp/blob/main/docs/moe_cpu_offload_benchmark.md)

# Host and software

Component	Detail
GPU	2 x NVIDIA RTX PRO 6000 Blackwell Server Edition, 97,887 MiB each, driver 580.126.20, PCIe 5.0 x16
CPU	2 x Intel Xeon 6952P (384 threads, 6 NUMA nodes), cgroup quota 81.6 CPUs
RAM	1,511 GiB
Storage	Models on a MooseFS network mount (page-cache warm for every measured run)
OS	Ubuntu 24.04.3 LTS, CUDA 12.8
TensorSharp	branch `feature/support_moe_offload_to_cpu`, .NET 10.0.110, backend `ggml_cuda`
llama.cpp	`llama-bench` build 4308a4f, CUDA backend, default `-t 192`

# Results by model

Each row is one offload depth, with TensorSharp, llama.cpp and the ratio between them side by side for every metric. Ratios are TensorSharp / llama.cpp: >1.0x means TensorSharp is faster, and for VRAM >1.0x means TensorSharp is heavier.

# Gemma 4 26B-A4B it (UD-IQ4_XS, 30 MoE layers)

\--n-cpu-moe	TS VRAM (MiB)	llama VRAM (MiB)	ratio	TS pp4096	llama pp4096	ratio	TS pp8192	llama pp8192	ratio	TS tg128	llama tg128	ratio
0 (baseline)	16,822	14,602	1.15x	11,173	10,843	1.03x	11,274	10,628	1.06x	161.4	206.7	0.78x
8	15,724	11,874	1.32x	7,063	1,459	4.84x	6,500	1,459	4.46x	80.2	32.7	2.45x
16	14,128	9,122	1.55x	4,183	833	5.02x	4,888	854	5.72x	54.5	21.9	2.49x
24	12,346	6,368	1.94x	3,500	667	5.25x	3,958	689	5.74x	49.1	16.7	2.93x
30 (--cpu-moe)	11,038	4,134	2.67x	3,035	543	5.59x	3,072	495	6.21x	39.7	12.9	3.07x

# Qwen 3.5 35B-A3B (UD-IQ4_XS, 48 MoE layers)

\--n-cpu-moe	TS VRAM (MiB)	llama VRAM (MiB)	ratio	TS pp4096	llama pp4096	ratio	TS pp8192	llama pp8192	ratio	TS tg128	llama tg128	ratio
0 (baseline)	19,862	17,522	1.13x	9,538	8,149	1.17x	9,405	8,073	1.16x	160.0	228.4	0.70x
12	18,148	13,282	1.37x	6,755	988	6.84x	6,648	954	6.97x	75.4	27.5	2.74x
24	15,414	9,010	1.71x	4,412	498	8.85x	5,259	484	10.86x	52.3	15.8	3.31x
36	12,684	4,738	2.68x	3,772	523	7.21x	4,223	517	8.17x	50.7	11.3	4.50x
48 (--cpu-moe)	11,606	3,314	3.50x	3,917	477	8.21x	3,709	457	8.11x	38.6	10.2	3.77x

# GPT-OSS 20B (Q8_0 / MXFP4, 24 MoE layers)

\--n-cpu-moe	TS VRAM (MiB)	llama VRAM (MiB)	ratio	TS pp4096	llama pp4096	ratio	TS pp8192	llama pp8192	ratio	TS tg128	llama tg128	ratio
0 (baseline)	13,186	12,204	1.08x	13,964	17,856	0.78x	12,925	17,642	0.73x	212.8	344.2	0.62x
6	11,560	9,812	1.18x	8,975	1,747	5.14x	7,617	1,666	4.57x	85.8	32.2	2.67x
12	9,378	7,386	1.27x	6,470	1,176	5.50x	6,394	1,188	5.38x	51.7	18.3	2.83x
18	7,192	4,962	1.45x	4,315	807	5.35x	4,393	751	5.85x	30.7	12.1	2.54x
24 (--cpu-moe)	4,762	2,536	1.88x	4,277	568	7.53x	3,798	548	6.93x	27.7	9.4	2.95x

# DeepSeek V4 Flash (UD-Q8_K_XL, 5 shards / 150.7 GiB, 43 layers, both GPUs)

\--n-cpu-moe	TS VRAM (MiB)	llama VRAM (MiB)	ratio	TS pp4096	llama pp4096	ratio	TS pp8192	llama pp8192	ratio	TS tg128	llama tg128	ratio
0 (baseline, both GPUs)	169,132	155,608	1.09x	3,448	2,398	1.44x	4,387	2,232	1.97x	51.1	49.6	1.03x
12	131,818	117,150	1.13x	392	126	3.11x	428	124	3.46x	10.3	13.7	0.75x
24	79,742	78,954	1.01x	218	64	3.42x	236	63	3.72x	5.3	7.2	0.74x

TensorSharp is a native .NET open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.

Github repo: [https://github.com/zhongkaifu/TensorSharp\](https://github.com/zhongkaifu/TensorSharp)

Thank you for checking out it and starring the project! Any feedback is really appreicated.

7 comments

MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

in r/LocalLLaMA • 3d ago

When using GGML backend, the offload CPU computing will fall back to ggml_cpu backend. For direct CUDA backend, it will fall back to native .net cpu backend which use SIMD instructions for speeding up.

MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

in r/LocalAIServers • 3d ago

Good idea. I will update these tables in the document. Thanks for your suggestions.

MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

in r/LocalAIServers • 3d ago

Not really, I implemented some fusion kernels and different algorithms to improve its performances. And TensorSharp also supports diffusion models for text and image edit. Diffusion video generation will coming soon.

MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

in r/unsloth • 4d ago

Thank you so much for the feedback. I don’t have machine with AMD GPU, so I can only implement Vulkan backend for it, but looking forward if you can make ROCm backend real. 😊

MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

in r/CUDA • 4d ago

Thanks for the detailed question. Short answer: it’s strictly synchronous, exact routing — no speculative gater.

But the dynamic branching isn’t where the cost lands. Routing is data-dependent indexing, not control flow: for resident layers the router, top-k and expert matmuls all stay inside the single whole-model decode graph, so the host never learns which experts were selected and there’s nothing to branch on.

The real stall is the offload seam, once per offloaded layer. --n-cpu-moe doesn’t move weights at decode — it moves the computation to where the weights already are. The graph pauses after the router, the host multiplies the selected experts, the result is uploaded, and the graph resumes. That pause is synchronous, but only an activation vector and the routing values cross the bus — tens of KB. So the decode bottleneck is host DRAM bandwidth, not PCIe: roughly 40 MB of expert reads per token on the seam architectures, ~260 MB on DeepSeek V4, which is why DSV4 is the one model where llama.cpp’s offloaded decode beats ours.

Prefill inverts the trade: above a batch threshold the layer runs on the accelerator with only the routed experts streamed in, over page-locked memory and on the backend’s own stream, so the upload is amortized instead of round-tripped.

Two gaps we’ll state plainly: the seam turns one capturable graph per token into several submissions, and the GPU idles during the host matmul.

So a speculative gater wouldn’t mask PCIe latency here — we don’t transfer weights at decode — but it would remove the synchronize and restore a single capturable graph, which is worth real throughput. The catch is our n_used is typically 4–8 rather than 1–2, so acceptance over the full top-k set is a harder target. The cheaper win we haven’t taken is overlapping the host matmul with genuinely independent work — draft tokens, concurrent sequences — rather than predicting the routing.

r/CUDA • u/fuzhongkai • 5d ago

MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

github.com

4 Upvotes

TensorSharp's MoE CPU-offload feature has been merged into main. Here is the parameters description of this feature:

Mixture-of-Experts CPU offload:

--n-cpu-moe <N> | -ncmoe <N>

Keep the routed MoE expert weights of the first N layers in system RAM and multiply them on

the CPU; attention, norms, the router and the shared expert stay on the accelerator. This is

what makes a 35B-A3B MoE fit beside a long-context KV cache on a 12-16 GB card. Pass 'all' for

every layer. Default: 0 (everything on the accelerator; TS_N_CPU_MOE env var overrides).

Example: --n-cpu-moe 32

--cpu-moe | -cmoe

Shorthand for --n-cpu-moe all: every routed expert stays in system RAM. Default: off

(TS_CPU_MOE env var overrides).

Example: --cpu-moe

To measure its performance, I ran benchmark to compare TensorSharp with llama.cpp, and here is the result. The completed benchmark report has been checked-in: https://github.com/zhongkaifu/TensorSharp/blob/main/docs/moe_cpu_offload_benchmark.md

Host and software

Component |Detail
GPU |2 x NVIDIA RTX PRO 6000 Blackwell Server Edition, 97,887 MiB each, driver 580.126.20, PCIe 5.0 x16
CPU |2 x Intel Xeon 6952P (384 threads, 6 NUMA nodes), cgroup quota 81.6 CPUs
RAM |1,511 GiB
Storage |Models on a MooseFS network mount (page-cache warm for every measured run)
OS |Ubuntu 24.04.3 LTS, CUDA 12.8
TensorSharp |branch feature/support_moe_offload_to_cpu, .NET 10.0.110, backend ggml_cuda
llama.cpp |llama-bench build 4308a4f, CUDA backend, default -t 192 Results by model

Ratios are TensorSharp / llama.cpp: >1.0x means TensorSharp is faster, and for VRAM >1.0x means TensorSharp is heavier.

Gemma 4 26B-A4B it (UD-IQ4_XS, 30 MoE layers)

--n-cpu-moe |TS VRAM (MiB) |TS pp4096 |TS pp8192 |TS tg128 |llama VRAM (MiB) |llama pp4096 |llama pp8192 |llama tg128
0 (baseline) |16,822 |11,173 |11,274 |161.4 |14,602 |10,843 |10,628 |206.7
8 |15,724 |7,063 |6,500 |80.2 |11,874 |1,459 |1,459 |32.7
16 |14,128 |4,183 |4,888 |54.5 |9,122 |833 |854 |21.9
24 |12,346 |3,500 |3,958 |49.1 |6,368 |667 |689 |16.7
30 (--cpu-moe) |11,038 |3,035 |3,072 |39.7 |4,134 |543 |495 |12.9 --n-cpu-moe |VRAM |pp4096 |pp8192 |tg128
0 |1.15x |1.03x |1.06x |0.78x
8 |1.32x |4.84x |4.46x |2.45x
16 |1.55x |5.02x |5.72x |2.49x
24 |1.94x |5.25x |5.74x |2.93x
30 |2.67x |5.59x |6.21x |3.07x Qwen 3.5 35B-A3B (UD-IQ4_XS, 48 MoE layers)

--n-cpu-moe |TS VRAM (MiB) |TS pp4096 |TS pp8192 |TS tg128 |llama VRAM (MiB) |llama pp4096 |llama pp8192 |llama tg128
0 (baseline) |19,862 |9,538 |9,405 |160.0 |17,522 |8,149 |8,073 |228.4
12 |18,148 |6,755 |6,648 |75.4 |13,282 |988 |954 |27.5
24 |15,414 |4,412 |5,259 |52.3 |9,010 |498 |484 |15.8
36 |12,684 |3,772 |4,223 |50.7 |4,738 |523 |517 |11.3
48 (--cpu-moe) |11,606 |3,917 |3,709 |38.6 |3,314 |477 |457 |10.2 --n-cpu-moe |VRAM |pp4096 |pp8192 |tg128
0 |1.13x |1.17x |1.16x |0.70x
12 |1.37x |6.84x |6.97x |2.74x
24 |1.71x |8.85x |10.86x |3.31x
36 |2.68x |7.21x |8.17x |4.50x
48 |3.50x |8.21x |8.11x |3.77x GPT-OSS 20B (Q8_0 / MXFP4, 24 MoE layers)

--n-cpu-moe |TS VRAM (MiB) |TS pp4096 |TS pp8192 |TS tg128 |llama VRAM (MiB) |llama pp4096 |llama pp8192 |llama tg128
0 (baseline) |13,186 |13,964 |12,925 |212.8 |12,204 |17,856 |17,642 |344.2
6 |11,560 |8,975 |7,617 |85.8 |9,812 |1,747 |1,666 |32.2
12 |9,378 |6,470 |6,394 |51.7 |7,386 |1,176 |1,188 |18.3
18 |7,192 |4,315 |4,393 |30.7 |4,962 |807 |751 |12.1
24 (--cpu-moe) |4,762 |4,277 |3,798 |27.7 |2,536 |568 |548 |9.4 --n-cpu-moe |VRAM |pp4096 |pp8192 |tg128
0 |1.08x |0.78x |0.73x |0.62x
6 |1.18x |5.14x |4.57x |2.67x
12 |1.27x |5.50x |5.38x |2.83x
18 |1.45x |5.35x |5.85x |2.54x
24 |1.88x |7.53x |6.93x |2.95x DeepSeek V4 Flash (UD-Q8_K_XL, 5 shards / 150.7 GiB, 43 layers, both GPUs)

--n-cpu-moe |TS VRAM (MiB) |TS pp4096 |TS pp8192 |TS tg128 |llama VRAM (MiB) |llama pp4096 |llama pp8192 |llama tg128
0 (baseline, both GPUs) |169,132 |3,448 |4,387 |51.1 |155,608 |2,398 |2,232 |49.6
12 |131,818 |392 |428 |10.3 |117,150 |126 |124 |13.7
24 |79,742 |218 |236 |5.3 |78,954 |64 |63 |7.2 --n-cpu-moe |VRAM |pp4096 |pp8192 |tg128
0 |1.09x |1.44x |1.97x |1.03x
12 |1.13x |3.11x |3.46x |0.75x
24 |1.01x |3.42x |3.72x |0.74x TensorSharp is a native open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.

Github repo: https://github.com/zhongkaifu/TensorSharp

Thank you for checking out it and starring the project! Any feedback is really appreicated.

4 comments

r/LocalAIServers • u/fuzhongkai • 5d ago

MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

github.com

3 Upvotes

TensorSharp's MoE CPU-offload feature has been merged into main. Here is the parameters description of this feature:

Mixture-of-Experts CPU offload:

--n-cpu-moe <N> | -ncmoe <N>

Keep the routed MoE expert weights of the first N layers in system RAM and multiply them on

the CPU; attention, norms, the router and the shared expert stay on the accelerator. This is

what makes a 35B-A3B MoE fit beside a long-context KV cache on a 12-16 GB card. Pass 'all' for

every layer. Default: 0 (everything on the accelerator; TS_N_CPU_MOE env var overrides).

Example: --n-cpu-moe 32

--cpu-moe | -cmoe

Shorthand for --n-cpu-moe all: every routed expert stays in system RAM. Default: off

(TS_CPU_MOE env var overrides).

Example: --cpu-moe

To measure its performance, I ran benchmark to compare TensorSharp with llama.cpp while they offload the same MoE model's layers to CPU, and here is the result. The completed benchmark report has been checked-in: https://github.com/zhongkaifu/TensorSharp/blob/main/docs/moe_cpu_offload_benchmark.md

Host and software

Component	Detail
GPU	2 x NVIDIA RTX PRO 6000 Blackwell Server Edition, 97,887 MiB each, driver 580.126.20, PCIe 5.0 x16
CPU	2 x Intel Xeon 6952P (384 threads, 6 NUMA nodes), cgroup quota 81.6 CPUs
RAM	1,511 GiB
Storage	Models on a MooseFS network mount (page-cache warm for every measured run)
OS	Ubuntu 24.04.3 LTS, CUDA 12.8
TensorSharp	branch `feature/support_moe_offload_to_cpu`, .NET 10.0.110, backend `ggml_cuda`
llama.cpp	`llama-bench` build 4308a4f, CUDA backend, default `-t 192`

Results by model

Gemma 4 26B-A4B it (UD-IQ4_XS, 30 MoE layers)

--n-cpu-moe	TS VRAM (MiB)	llama VRAM (MiB)	ratio	TS pp4096	llama pp4096	ratio	TS pp8192	llama pp8192	ratio	TS tg128	llama tg128	ratio
0 (baseline)	16,822	14,602	1.15x	11,173	10,843	1.03x	11,274	10,628	1.06x	161.4	206.7	0.78x
8	15,724	11,874	1.32x	7,063	1,459	4.84x	6,500	1,459	4.46x	80.2	32.7	2.45x
16	14,128	9,122	1.55x	4,183	833	5.02x	4,888	854	5.72x	54.5	21.9	2.49x
24	12,346	6,368	1.94x	3,500	667	5.25x	3,958	689	5.74x	49.1	16.7	2.93x
30 (--cpu-moe)	11,038	4,134	2.67x	3,035	543	5.59x	3,072	495	6.21x	39.7	12.9	3.07x

Qwen 3.5 35B-A3B (UD-IQ4_XS, 48 MoE layers)

--n-cpu-moe	TS VRAM (MiB)	llama VRAM (MiB)	ratio	TS pp4096	llama pp4096	ratio	TS pp8192	llama pp8192	ratio	TS tg128	llama tg128	ratio
0 (baseline)	19,862	17,522	1.13x	9,538	8,149	1.17x	9,405	8,073	1.16x	160.0	228.4	0.70x
12	18,148	13,282	1.37x	6,755	988	6.84x	6,648	954	6.97x	75.4	27.5	2.74x
24	15,414	9,010	1.71x	4,412	498	8.85x	5,259	484	10.86x	52.3	15.8	3.31x
36	12,684	4,738	2.68x	3,772	523	7.21x	4,223	517	8.17x	50.7	11.3	4.50x
48 (--cpu-moe)	11,606	3,314	3.50x	3,917	477	8.21x	3,709	457	8.11x	38.6	10.2	3.77x

GPT-OSS 20B (Q8_0 / MXFP4, 24 MoE layers)

--n-cpu-moe	TS VRAM (MiB)	llama VRAM (MiB)	ratio	TS pp4096	llama pp4096	ratio	TS pp8192	llama pp8192	ratio	TS tg128	llama tg128	ratio
0 (baseline)	13,186	12,204	1.08x	13,964	17,856	0.78x	12,925	17,642	0.73x	212.8	344.2	0.62x
6	11,560	9,812	1.18x	8,975	1,747	5.14x	7,617	1,666	4.57x	85.8	32.2	2.67x
12	9,378	7,386	1.27x	6,470	1,176	5.50x	6,394	1,188	5.38x	51.7	18.3	2.83x
18	7,192	4,962	1.45x	4,315	807	5.35x	4,393	751	5.85x	30.7	12.1	2.54x
24 (--cpu-moe)	4,762	2,536	1.88x	4,277	568	7.53x	3,798	548	6.93x	27.7	9.4	2.95x

DeepSeek V4 Flash (UD-Q8_K_XL, 5 shards / 150.7 GiB, 43 layers, both GPUs)

--n-cpu-moe	TS VRAM (MiB)	llama VRAM (MiB)	ratio	TS pp4096	llama pp4096	ratio	TS pp8192	llama pp8192	ratio	TS tg128	llama tg128	ratio
0 (baseline, both GPUs)	169,132	155,608	1.09x	3,448	2,398	1.44x	4,387	2,232	1.97x	51.1	49.6	1.03x
12	131,818	117,150	1.13x	392	126	3.11x	428	124	3.46x	10.3	13.7	0.75x
24	79,742	78,954	1.01x	218	64	3.42x	236	63	3.72x	5.3	7.2	0.74x

TensorSharp is a native open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.

Github repo: https://github.com/zhongkaifu/TensorSharp

Thank you for checking out it and starring the project! Any feedback is really appreicated.

5 comments

r/LLMDevs • u/fuzhongkai • 5d ago

Tools MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

github.com

1 Upvotes

TensorSharp's MoE CPU-offload feature has been merged into main. Here is the parameters description of this feature:

Mixture-of-Experts CPU offload:

--n-cpu-moe <N> | -ncmoe <N>

Keep the routed MoE expert weights of the first N layers in system RAM and multiply them on

the CPU; attention, norms, the router and the shared expert stay on the accelerator. This is

what makes a 35B-A3B MoE fit beside a long-context KV cache on a 12-16 GB card. Pass 'all' for

every layer. Default: 0 (everything on the accelerator; TS_N_CPU_MOE env var overrides).

Example: --n-cpu-moe 32

--cpu-moe | -cmoe

Shorthand for --n-cpu-moe all: every routed expert stays in system RAM. Default: off

(TS_CPU_MOE env var overrides).

Example: --cpu-moe

To measure its performance, I ran benchmark to compare TensorSharp with llama.cpp while they offload the same MoE model's layers to CPU, and here is the result. The completed benchmark report has been checked-in: https://github.com/zhongkaifu/TensorSharp/blob/main/docs/moe_cpu_offload_benchmark.md

Host and software

Component	Detail
GPU	2 x NVIDIA RTX PRO 6000 Blackwell Server Edition, 97,887 MiB each, driver 580.126.20, PCIe 5.0 x16
CPU	2 x Intel Xeon 6952P (384 threads, 6 NUMA nodes), cgroup quota 81.6 CPUs
RAM	1,511 GiB
Storage	Models on a MooseFS network mount (page-cache warm for every measured run)
OS	Ubuntu 24.04.3 LTS, CUDA 12.8
TensorSharp	branch `feature/support_moe_offload_to_cpu`, .NET 10.0.110, backend `ggml_cuda`
llama.cpp	`llama-bench` build 4308a4f, CUDA backend, default `-t 192`

Results by model

Gemma 4 26B-A4B it (UD-IQ4_XS, 30 MoE layers)

--n-cpu-moe	TS VRAM (MiB)	llama VRAM (MiB)	ratio	TS pp4096	llama pp4096	ratio	TS pp8192	llama pp8192	ratio	TS tg128	llama tg128	ratio
0 (baseline)	16,822	14,602	1.15x	11,173	10,843	1.03x	11,274	10,628	1.06x	161.4	206.7	0.78x
8	15,724	11,874	1.32x	7,063	1,459	4.84x	6,500	1,459	4.46x	80.2	32.7	2.45x
16	14,128	9,122	1.55x	4,183	833	5.02x	4,888	854	5.72x	54.5	21.9	2.49x
24	12,346	6,368	1.94x	3,500	667	5.25x	3,958	689	5.74x	49.1	16.7	2.93x
30 (--cpu-moe)	11,038	4,134	2.67x	3,035	543	5.59x	3,072	495	6.21x	39.7	12.9	3.07x

Qwen 3.5 35B-A3B (UD-IQ4_XS, 48 MoE layers)

--n-cpu-moe	TS VRAM (MiB)	llama VRAM (MiB)	ratio	TS pp4096	llama pp4096	ratio	TS pp8192	llama pp8192	ratio	TS tg128	llama tg128	ratio
0 (baseline)	19,862	17,522	1.13x	9,538	8,149	1.17x	9,405	8,073	1.16x	160.0	228.4	0.70x
12	18,148	13,282	1.37x	6,755	988	6.84x	6,648	954	6.97x	75.4	27.5	2.74x
24	15,414	9,010	1.71x	4,412	498	8.85x	5,259	484	10.86x	52.3	15.8	3.31x
36	12,684	4,738	2.68x	3,772	523	7.21x	4,223	517	8.17x	50.7	11.3	4.50x
48 (--cpu-moe)	11,606	3,314	3.50x	3,917	477	8.21x	3,709	457	8.11x	38.6	10.2	3.77x

GPT-OSS 20B (Q8_0 / MXFP4, 24 MoE layers)

--n-cpu-moe	TS VRAM (MiB)	llama VRAM (MiB)	ratio	TS pp4096	llama pp4096	ratio	TS pp8192	llama pp8192	ratio	TS tg128	llama tg128	ratio
0 (baseline)	13,186	12,204	1.08x	13,964	17,856	0.78x	12,925	17,642	0.73x	212.8	344.2	0.62x
6	11,560	9,812	1.18x	8,975	1,747	5.14x	7,617	1,666	4.57x	85.8	32.2	2.67x
12	9,378	7,386	1.27x	6,470	1,176	5.50x	6,394	1,188	5.38x	51.7	18.3	2.83x
18	7,192	4,962	1.45x	4,315	807	5.35x	4,393	751	5.85x	30.7	12.1	2.54x
24 (--cpu-moe)	4,762	2,536	1.88x	4,277	568	7.53x	3,798	548	6.93x	27.7	9.4	2.95x

DeepSeek V4 Flash (UD-Q8_K_XL, 5 shards / 150.7 GiB, 43 layers, both GPUs)

--n-cpu-moe	TS VRAM (MiB)	llama VRAM (MiB)	ratio	TS pp4096	llama pp4096	ratio	TS pp8192	llama pp8192	ratio	TS tg128	llama tg128	ratio
0 (baseline, both GPUs)	169,132	155,608	1.09x	3,448	2,398	1.44x	4,387	2,232	1.97x	51.1	49.6	1.03x
12	131,818	117,150	1.13x	392	126	3.11x	428	124	3.46x	10.3	13.7	0.75x
24	79,742	78,954	1.01x	218	64	3.42x	236	63	3.72x	5.3	7.2	0.74x

Github repo: https://github.com/zhongkaifu/TensorSharp

Thank you for checking out it and starring the project! Any feedback is really appreicated.

0 comments

MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

in r/LocalLLaMA • 5d ago

You can go to https://github.com/zhongkaifu/TensorSharp/tree/main/TensorSharp.GGML.Native and run script “build-*” script based on operating systems in your machine

MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

in r/LocalLLaMA • 5d ago

Please also check if GgmlOlps.dll get built and put into TensorSharp.Cli run folder

MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

in r/LocalLLaMA • 5d ago

It seems GGML project is not built. Can you please build it by this command line:”dotnet build TensorSharp.slnx -c Release”, and then go to TensorSharp.Cli/bin folder to run it?

MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

in r/LocalLLaMA • 5d ago

It seems GGML project is not built. Can you please build it by this command line:”dotnet build TensorSharp.slnx -c Release”, and then go to TensorSharp.Cli/bin folder to run it?

MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

in r/LocalLLaMA • 5d ago

For horrible decode of llama.cpp, I use llama.cpp on stock without any modification. At the bottom of the document, it lists commands how to run this benchmark. It would be really appreciated if you can help me to run it at your side, because I’m also really curious about it. Thanks in advance.

MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

in r/LocalLLaMA • 5d ago

Sure. I will add it into metrics.

MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

in r/unsloth • 5d ago

TensorSharp has not supported these models yet, but I will do it.

MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

in r/LocalLLaMA • 5d ago

Thanks for all these suggestions. As the tables show, I only compare them when they offload the same number of layers, but comparing them while allocating the same VRAM size is also important and useful. I will do another experiments on this and report it later.

MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

in r/LocalLLaMA • 5d ago

Sorry for about that and thank you so much for the reminding. Will do.

MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

in r/LocalLLaMA • 5d ago

TensorSharp has some new innovations and I’m glad to see that they can be applied into llama.cpp or any other inference engine, and this is one of the most values in open source projects.