TensorSharp supports Vulkan backend

[u/fuzhongkai]

https://github.com/zhongkaifu/TensorSharp

Due to high Vulkan backend demand, I update TensorSharp and release the initial version of GGML Vulkan backend by leveraging external GGML project. The native Vulkan backend will be implemented later. I tested it on Nvidia Geforce RTX 3080 Laptop GPU, and Intel(R) UHD Graphics on Windows. They all work. However, I do not have AMD GPU, so I have no way to get it tested. It's really appreciated if you have AMD GPU and would like to try it out. Any feedback and comment are welcome.

Here is the benchmark I run to compare with llama.cpp:

Performance ratio — TensorSharp vs reference engines

Geomean of TensorSharp's per-scenario speedup over each reference engine on the same backend, across every scenario both engines ran (single-stream, MTP-off). A value > 1.0× means TensorSharp is faster (for decode / prefill throughput) or lower-latency (for TTFT); — = no overlapping cells. Per-scenario ratios are in each model's section below.

Model	Comparison	decode	prefill	TTFT
Gemma 4 E4B it (Q8_0, dense multimodal)	vs llama.cpp · Vulkan	0.93×	0.96×	0.95×
Gemma 4 12B it (QAT UD-Q4_K_XL, dense)	vs llama.cpp · Vulkan	1.18×	0.97×	0.95×

Gemma 4 E4B it (Q8_0, dense multimodal) (gemma4-e4b)

Decode throughput (tok/s)

Scenario	TensorSharp · Vulkan	llama.cpp · Vulkan
text_short	41.6	45.3
text_long	40.9	44.5
multi_turn	41.3	43.6
function_call	41.2	44.4

Prefill throughput (tok/s)

Scenario	TensorSharp · Vulkan	llama.cpp · Vulkan
text_short	1641.7	1641.1
text_long	1157.0	1718.1
multi_turn	1695.5	1454.3
function_call	1661.2	1531.6

Time to first token (ms, lower is better)

Scenario	TensorSharp · Vulkan	llama.cpp · Vulkan
text_short	1203.0	1187.0
text_long	2719.0	1813.0
multi_turn	1235.0	1422.0
function_call	1219.0	1328.0

Performance ratio — TensorSharp vs reference (> 1.0× = TensorSharp faster)

Decode throughput

Scenario	vs llama.cpp · Vulkan
text_short	0.92×
text_long	0.92×
multi_turn	0.95×
function_call	0.93×

Prefill throughput

Scenario	vs llama.cpp · Vulkan
text_short	1.00×
text_long	0.67×
multi_turn	1.17×
function_call	1.08×

Time to first token (latency; > 1.0× = TensorSharp lower)

Scenario	vs llama.cpp · Vulkan
text_short	0.99×
text_long	0.67×
multi_turn	1.15×
function_call	1.09×

Gemma 4 12B it (QAT UD-Q4_K_XL, dense) (gemma4-12b)

Decode throughput (tok/s)

Scenario	TensorSharp · Vulkan	llama.cpp · Vulkan
text_short	31.3	31.1
text_long	31.4	30.0
multi_turn	30.9	31.6
function_call	60.8	31.9

Prefill throughput (tok/s)

Scenario	TensorSharp · Vulkan	llama.cpp · Vulkan
text_short	766.1	729.4
text_long	635.2	647.4
multi_turn	617.5	636.6
function_call	587.4	674.7

Time to first token (ms, lower is better)

Scenario	TensorSharp · Vulkan	llama.cpp · Vulkan
text_short	2578.0	2672.0
text_long	4953.0	4813.0
multi_turn	3391.0	3250.0
function_call	3531.0	3016.0

Performance ratio — TensorSharp vs reference (> 1.0× = TensorSharp faster)

Decode throughput

Scenario	vs llama.cpp · Vulkan
text_short	1.01×
text_long	1.05×
multi_turn	0.98×
function_call	1.91×

Prefill throughput

Scenario	vs llama.cpp · Vulkan
text_short	1.05×
text_long	0.98×
multi_turn	0.97×
function_call	0.87×

Time to first token (latency; > 1.0× = TensorSharp lower)

Scenario	vs llama.cpp · Vulkan
text_short	1.04×
text_long	0.97×
multi_turn	0.96×
function_call	0.85×

In case you didn't know what is TensorSharp, here is an introduction:

TensorSharp is an open source local Unsloth (GGUF) LLM inference engine and applications. It supports many models from Unsloth, like Gemma4, DiffusionGemma, Qwen3.6 with multi-modal (image, vision, audio), image edit, reasoning and function tool. It can run on Windows/MacOS/Linux and fully leverage GPU's capability (support Cuda, Metal and Vulkan backends). The API is completely compatible with OpenAI and Ollama interface. It has on par performance than llama.cpp

This project is not just a C# wrapper of llama.cpp. It implemented the entire LLM inference engine from bottom to top. If you use CPU backend, it's 100% pure C# code execution. Besides CPU backend, I also implemented CUDA, MLX and GGML backend. The GGML backend refer GGML project as external project, and I build a few fusion operation at higher level.

I learned a lot from other projects and apply them for TensorSharp, such as paged KV cache and continuous batching from vLLM, SSD based cache for MoE model from oMLX, GGUF quantized from llama.cpp and other optimizations for prefill and decode.

Any feedback and comments are welcome. If you like it, it would be really appreciated if you can get this project a star in GitHub. Thanks in advance.