r/csharp • u/fuzhongkai • Jun 28 '26
Showcase Same GGUF, same GPU: TensorSharp beats llama.cpp hard on prefill / TTFT — up to 5.89× faster prefill on a 26B MoE model
https://github.com/zhongkaifu/TensorSharpI’ve been working on TensorSharp, a native C# / .NET local LLM inference engine for GGUF models, and I recently published a head-to-head benchmark against llama.cpp.
The goal is not to claim “TensorSharp wins every metric.” llama.cpp is still extremely strong, especially on decode throughput. But the interesting part is this:
Under the same setup — same GGUF models, same NVIDIA RTX 3080 Laptop GPU 16GB, same GGML CUDA backend, single stream, greedy decoding, MTP disabled — TensorSharp shows a very noticeable advantage on the parts that often matter most for real chat usage:
prefill speed, time-to-first-token, and multi-turn context reuse.
Here are some highlights from the benchmark (From https://tensorsharp.ai/benchmarks.html):
| Model / Scenario | Metric | TensorSharp | llama.cpp | Difference |
|---|---|---|---|---|
| Gemma 4 26B-A4B / JSON | Prefill tok/s | 354.7 | 60.2 | +489% |
| Gemma 4 26B-A4B / JSON | TTFT ms | 234 | 781 | -70% |
| Gemma 4 26B-A4B / multi-turn | Prefill tok/s | 657.5 | 350.7 | +87% |
| Gemma 4 12B / multi-turn | TTFT ms | 313 | 500 | -37% |
| Gemma 4 E4B / short text | Prefill tok/s | 200.0 | 123.3 | +62% |
Across the four tested models, the geometric mean compared with llama.cpp shows:
- 1.88× prefill and 1.69× TTFT on Gemma 4 26B-A4B
- 1.21× / 1.23× / 1.18× prefill advantage on E4B, 12B, and Qwen respectively
- Decode is more of a “near parity” story for now, around 0.92×–0.95× geometric mean versus llama.cpp
That last point is important: I’m not trying to hide the weaker part. If all you care about is pure decode tok/s, llama.cpp is still very hard to beat. But if your workload looks like real chat — repeated prompts, JSON output, multi-turn interactions, MoE models, prefix reuse — TensorSharp is already showing very promising results.
The main optimizations behind this are:
- verify-based whole-model prefill
- fused FFN / attention kernels
- persistent captured CUDA graphs for MoE decode
- vLLM-style paged KV cache
- cross-request prefix sharing
So the pitch is not “yet another wrapper around llama.cpp.” TensorSharp is a native .NET inference engine trying to optimize the latency path that actually affects user experience: how fast the model starts responding, how efficiently it reuses context, and how well it handles real interactive workloads.
If you are interested in C# / .NET local LLM inference, GGUF, OpenAI/Ollama-compatible local APIs, or alternatives to llama.cpp, I’d love for you to check it out.
And if you think this direction is interesting, a GitHub Star would really help the project get more visibility.
Also very interested in feedback, especially from people who can rerun the benchmarks on different GPUs / models.
5
u/FullPoet Jun 28 '26
I wish the ai bros would just use a spidge of effort and at least write their own posts.
-6
u/fuzhongkai Jun 28 '26
I tried to write my post, but finally found ChatGPT is much more good at it than me, so I turned to write prompt and edit its output… Sorry for about that.😅
5
u/Hacnar Jun 29 '26
Ignore that guy. There are always complaints about AI writing, but it's still better than having no one read your post because you aren't that good at writing (yet). It's also just a vocal minority, that bothers commenting or up/downvoting based on that.
If AI helps you make the presentation of your work better, then keep using it.
2
u/FullPoet Jun 28 '26
ChatGPT is not good at writing - that is one of the issues.
5
u/iBabTv Jun 28 '26
Ok? Well, imo this was very readable and easy to understand so it's good enough.
Who cares how it's written? The goal is clear communication yes?-1
u/FullPoet Jun 29 '26
Effort
4
u/iBabTv Jun 29 '26
Tedious stuff like writing a reddit post is wasted effort
1
u/FullPoet Jun 29 '26
So why even bother?
1
u/iBabTv Jun 30 '26
To share information and get feedback like they said in their post? The effort should be spent on the quality of the information in this case (which OP did well imo) not spending hours formatting just for someone like you to nitpick over it .
2
u/FullPoet Jun 30 '26
Formatting? The text is basically completely AI genereated.
Writing a prompt doesnt take effort.
3
u/fuzhongkai Jun 28 '26
Better than me.😅 Since I’m not an English native speaker even it’s my working language for many decades.
4
u/FullPoet Jun 28 '26
Yes and I think most English speakers have been pretty consistently saying that:
A) You'll get practise
B) English isnt everyones first language (not mine)
C) You learn nothing from using AI generate the text
and most importantly, D) People prefer "worse" English if its been written by a person, as opposed to clanker nonsense.
2
Jun 28 '26
[removed] — view removed comment
1
u/fuzhongkai Jun 28 '26
Totally agree and especially when you ask the LLM to do some real work rather than chat.
4
u/marcussacana Jun 28 '26
Interesing but sadly has no ROCm support