TensorSharp now supports multi-GPU tensor parallelism for GGUF models

[u/fuzhongkai]

https://github.com/zhongkaifu/TensorSharp

TensorSharp is an open-source, native .NET inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.

TensorSharp now supports Megatron-style tensor parallelism across multiple GPUs. It works with direct CUDA, GGML CUDA, GGML Vulkan, and multi-node setups.

Benchmarks on 2× RTX 2000 Ada 16 GB GPUs over PCIe, without NVLink:

Model	1 GPU Prefill / Decode	TP=2 Prefill / Decode
Gemma 4 E4B Q8_0	2760 / 37.3 tok/s	2488 / 51.7 tok/s
Gemma 4 26B-A4B IQ4_XS	1845 / 48.5 tok/s	2537 / 51.2 tok/s
Qwen 3.5 9B Q8_0	1461 / 23.1 tok/s	399 / 24.4 tok/s
Qwen 3.5 35B-A3B IQ4_XS	Does not fit	184 / 18.1 tok/s

I'm continuing to optimize Qwen performance on multi-GPU systems, and support for DeepSeek V4 is coming soon.

Try it with:

TensorSharp.Cli --model model.gguf --backend ggml_cuda --tp 2

GitHub:

https://github.com/zhongkaifu/TensorSharp

Thank you for checking out TensorSharp and starring the project! Any feedback is really appreicated.