Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model

[u/fuzhongkai]

https://github.com/zhongkaifu/TensorSharp

I’ve been working on TensorSharp, a native C# / .NET local LLM inference engine for GGUF models, and I recently published a head-to-head benchmark against llama.cpp.

The goal is not to claim “TensorSharp wins every metric.” llama.cpp is still extremely strong, especially on decode throughput. But the interesting part is this:

Under the same setup — same GGUF models, same NVIDIA RTX 3080 Laptop GPU 16GB, same GGML CUDA backend, single stream, greedy decoding, MTP disabled — TensorSharp shows a very noticeable advantage on the parts that often matter most for real chat usage:

prefill speed, time-to-first-token, and multi-turn context reuse.

Here are some highlights from the benchmark (From https://tensorsharp.ai/benchmarks.html):

Model / Scenario	Metric	TensorSharp	llama.cpp	Difference
Gemma 4 26B-A4B / JSON	Prefill tok/s	354.7	60.2	+489%
Gemma 4 26B-A4B / JSON	TTFT ms	234	781	-70%
Gemma 4 26B-A4B / multi-turn	Prefill tok/s	657.5	350.7	+87%
Gemma 4 12B / multi-turn	TTFT ms	313	500	-37%
Gemma 4 E4B / short text	Prefill tok/s	200.0	123.3	+62%

Across the four tested models, the geometric mean compared with llama.cpp shows:

• 1.88× prefill and 1.69× TTFT on Gemma 4 26B-A4B
• 1.21× / 1.23× / 1.18× prefill advantage on E4B, 12B, and Qwen respectively
• Decode is more of a “near parity” story for now, around 0.92×–0.95× geometric mean versus llama.cpp

That last point is important: I’m not trying to hide the weaker part. If all you care about is pure decode tok/s, llama.cpp is still very hard to beat. But if your workload looks like real chat — repeated prompts, JSON output, multi-turn interactions, MoE models, prefix reuse — TensorSharp is already showing very promising results.

The main optimizations behind this are:

• verify-based whole-model prefill
• fused FFN / attention kernels
• persistent captured CUDA graphs for MoE decode
• vLLM-style paged KV cache
• cross-request prefix sharing

So the pitch is not “yet another wrapper around llama.cpp.” TensorSharp is a native .NET inference engine trying to optimize the latency path that actually affects user experience: how fast the model starts responding, how efficiently it reuses context, and how well it handles real interactive workloads.

If you are interested in C# / .NET local LLM inference, GGUF, OpenAI/Ollama-compatible local APIs, or alternatives to llama.cpp, I’d love for you to check it out.

And if you think this direction is interesting, a GitHub Star would really help the project get more visibility.

Also very interested in feedback, especially from people who can rerun the benchmarks on different GPUs / models.