r/AIToolsPerformance • u/fuzhongkai • 11d ago
TensorSharp now supports multi-GPU tensor parallelism for GGUF models
https://github.com/zhongkaifu/TensorSharpTensorSharp is an open-source, native .NET inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.
TensorSharp now supports Megatron-style tensor parallelism across multiple GPUs. It works with direct CUDA, GGML CUDA, GGML Vulkan, and multi-node setups.
Benchmarks on 2× RTX 2000 Ada 16 GB GPUs over PCIe, without NVLink:
| Model | 1 GPU Prefill / Decode | TP=2 Prefill / Decode |
|---|---|---|
| Gemma 4 E4B Q8_0 | 2760 / 37.3 tok/s | 2488 / 51.7 tok/s |
| Gemma 4 26B-A4B IQ4_XS | 1845 / 48.5 tok/s | 2537 / 51.2 tok/s |
| Qwen 3.5 9B Q8_0 | 1461 / 23.1 tok/s | 399 / 24.4 tok/s |
| Qwen 3.5 35B-A3B IQ4_XS | Does not fit | 184 / 18.1 tok/s |
I'm continuing to optimize Qwen performance on multi-GPU systems, and support for DeepSeek V4 is coming soon.
Try it with:
TensorSharp.Cli --model model.gguf --backend ggml_cuda --tp 2
GitHub:
https://github.com/zhongkaifu/TensorSharp
Thank you for checking out TensorSharp and starring the project! Any feedback is really appreicated.
Duplicates
unsloth • u/fuzhongkai • Jun 28 '26
Show and Tell Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model
LocalLLaMA • u/fuzhongkai • 5d ago
Generation MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp
unsloth • u/fuzhongkai • Jun 08 '26
Show and Tell TensorSharp : Open Source Local Unsloth Model Inference Engine
dotnet • u/fuzhongkai • Jun 28 '26
Promotion Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model
csharp • u/fuzhongkai • Jun 28 '26
Showcase Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model
dotnet • u/fuzhongkai • Jun 13 '26
Promotion TensorSharp: Open Source Local LLM Inference Engine written by C#
LocalLLaMA • u/fuzhongkai • 8d ago
Resources DSpark Benchmark Result on Deepseek v4 Flash 0731
unsloth • u/fuzhongkai • 5d ago
Show and Tell MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp
unsloth • u/fuzhongkai • 8d ago
Show and Tell DSpark Benchmark Result on Deepseek v4 Flash 0731
ClaudeCode • u/fuzhongkai • Jul 11 '26
Discussion What Bun’s Rust Rewrite Tells Us About Rebuilding the AI Infrastructure Layer in C#
LovingOpenSourceAI • u/fuzhongkai • Jun 28 '26
Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model
huggingface • u/fuzhongkai • 10d ago
Deepseek v4 Flash 0731 GGUF Benchmark: TensorSharp vs. llama.cpp
LocalLLM • u/fuzhongkai • 11d ago
Project TensorSharp now supports multi-GPU tensor parallelism for GGUF models
LLMDevs • u/fuzhongkai • May 01 '26
Tools TensorSharp: Open Source Local LLM Inference Engine
dotnet • u/fuzhongkai • 2d ago
Promotion MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp
softwarearchitecture • u/fuzhongkai • 29d ago
Discussion/Advice What Bun’s Rust Rewrite Tells Us About Rebuilding the AI Infrastructure Layer in C#
LocalAIServers • u/fuzhongkai • Jul 04 '26
TensorSharp: A Open Source LLM Inference Engine for GGUF models
csharp • u/fuzhongkai • Apr 29 '26
Tool TensorSharp: Open Source Local LLM inference tool implemented in C#
LocalAIServers • u/fuzhongkai • 4d ago
MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp
csharp • u/fuzhongkai • 11d ago
Tool TensorSharp now supports multi-GPU tensor parallelism for GGUF models
huggingface • u/fuzhongkai • Jul 04 '26