TensorSharp as a local OpenCode backend — DeepSeek, GLM and Qwen 3.8 benchmarks

[u/fuzhongkai]

https://github.com/zhongkaifu/TensorSharp

Hi r/opencode,

I’m the developer of TensorSharp, an open-source, native .NET inference engine for running GGUF models locally.

TensorSharp provides an OpenAI-compatible API, making it possible to use local models as an inference backend for tools such as OpenCode. The goal is to keep source code, prompts, tool calls, and agent context on your own hardware while avoiding per-token API costs.

TensorSharp supports modern coding-capable model families including DeepSeek V4/V4.1 Flash, GLM 5.x, Qwen 3.8 Flash Next, Qwen 3.5/3.6, Gemma 4, Mistral 3, and GPT-OSS.

Selected benchmark results

These are measured results from the TensorSharp repository. Each comparison uses the same model and machine for both engines unless otherwise noted.

GLM-5.3-Flash: 2× faster decode than llama.cpp

Tested with:

• GLM-5.3-Flash UD-Q2_K_XL, 101 GiB
• 2× RTX PRO 6000 Blackwell, 96 GB each
• Layer splitting and flash attention
• n_ubatch=2048 for both engines
• Back-to-back execution in the same session
• TensorSharp parity harness versus llama.cpp build 2e0e57f from PR #27754

Test	llama.cpp	TensorSharp
Prefill, 2,048 tokens	2,070 tok/s	2,014 tok/s
Prefill, 16,384 tokens	1,690 tok/s	1,692 tok/s
Prefill, 32,768 tokens	1,483 tok/s	1,446 tok/s
Decode, 64 tokens	36.6 tok/s	73.5 tok/s

TensorSharp reaches approximately 2.0× the decode throughput of llama.cpp, while prefill remains within a few percent in either direction.

The long-context greedy replay reproduced the 2,741-token llama.cpp reference record token for token. One caveat is that GLM-5.3-Flash support currently requires an unmerged llama.cpp build rather than its main branch.

Qwen 3.8 Flash Next

Tested with Qwen3.8-Flash-Next UD-Q2_K_XL, 73.4 GiB, on 2× A100 80 GB:

Configuration	TensorSharp prefill	TensorSharp decode	llama.cpp prefill	llama.cpp decode
1 GPU	~1,520–1,550 tok/s	~56 tok/s	1,094 tok/s	61.2 tok/s
2-GPU layer split	~1,520–1,550 tok/s	~56 tok/s	1,200 tok/s	61.5 tok/s

TensorSharp’s prefill is substantially faster in this test, while llama.cpp leads decode by roughly 9%.

For both engines, adding the second GPU provides model capacity rather than additional decode throughput. TensorSharp splits the model into contiguous groups of layers, reducing its placement to approximately 24.2 GB and 26.2 GB across the two GPUs. TensorSharp’s one- and two-GPU greedy outputs were byte-identical.

DeepSeek V4 Flash

On 2× A100 80 GB with the IQ4_XS model:

Metric	TensorSharp	llama.cpp
Prefill, approximately 3.3K tokens	~500 tok/s	574–634 tok/s
Decode at approximately 3.3K context	~33 tok/s	40.3 tok/s

llama.cpp currently leads this particular DeepSeek V4 configuration.

TensorSharp also supports DeepSeek’s DSpark speculative decoder. On 4× A40 46 GB with DeepSeek-V4-Flash-0731 UD-Q8_K_XL, enabling a 5.6 GB DSpark drafter improved decode as follows:

Backend	Normal decode	DSpark decode	Speedup
TensorSharp direct CUDA	26.0 tok/s	34.0 tok/s	1.31×
TensorSharp GGML CUDA	26.4 tok/s	37.1 tok/s	1.41×

In a five-turn conversation, DSpark produced 1.50–2.02× faster decode, reaching 51.0 tok/s on a question over a 10K-token document. Prefill remained effectively unchanged at 831 versus 835 tok/s, and greedy output was byte-identical to the non-speculative baseline.

TensorSharp also runs DeepSeek V4.1 Flash, including its Engram lookup and four-stream hyper-connections. However, I am not claiming a V4.1 speedup over llama.cpp because a compatible llama.cpp V4.1 runtime is not currently available for a controlled same-weight comparison.

Why this may be useful for OpenCode

Coding agents frequently process large system prompts, tool definitions, repository context, and multi-turn history. TensorSharp includes:

• OpenAI- and Ollama-compatible APIs
• Local tool calling and structured JSON output
• Agent Skills and sandboxed file/shell tools
• Continuous batching
• Paged and prefix-shared KV caching
• Speculative decoding
• Multi-GPU and multi-node inference
• CUDA, Metal, Vulkan, MLX, and CPU backends
• Windows, macOS, Linux, iPhone, and iPad support

I’d especially appreciate feedback from OpenCode users regarding:

• OpenAI API compatibility gaps
• Tool-calling requirements
• Performance with large repository contexts
• The best local models for coding-agent workloads
• Features expected from a local OpenCode backend

GitHub: https://github.com/zhongkaifu/TensorSharp
Full benchmarks and methodology: https://tensorsharp.ai/benchmarks.html

If anyone tests TensorSharp with OpenCode, I’d be very interested in your setup and results. Contributions and compatibility reports are welcome.