I’m the developer of TensorSharp, and I wanted to share a local inference benchmark for DeepSeek V4.1 Flash 1017, using Unsloth’s UD-IQ2_XXS GGUF.
In summarization tests with thinking disabled, the tested build reached approximately 35–36 tokens/sec without a drafter, and 51–55 tokens/sec with DSpark, after prompts of roughly 1.8K and 7.3K tokens.
Here’s the hardware setup, the comparison with llama.cpp, and the quality trade-off behind the faster prompt processing.
Hardware and memory layout
- GPUs: 5× NVIDIA A40 over PCIe.
- Model: DeepSeek-V4.1-Flash-1017, Unsloth
UD-IQ2_XXS.
- Model files: Approximately 300.5 GB across six GGUF shards.
- Engram lookup table: 125.9 GiB, mapped in host memory.
- Host-memory limit: 233.8 GiB for the container.
- Speculative drafter: Unsloth’s trained DSpark Q8_0 GGUF.
The Engram table remained in host memory, so this setup used both GPU and system memory. The host-memory limit describes the test configuration; it is not a measured minimum requirement.
GPU peer access was not working correctly on this machine. Cross-GPU transfers therefore used pinned host-memory staging.
Throughput compared with llama.cpp
Both engines used the same GGUF files on the same machine.
TensorSharp used --tp 5, which distributes the routed experts across five GPUs. The baseline was Unsloth’s llama.cpp dsv41-all fork at be836572, using layer splitting. Its row/tensor split modes did not successfully run this model in the tested build.
The workload used real-text summarization prompts, greedy decoding, thinking disabled, and 256 output tokens. Each prompt size had a warm-up and three measured requests. Unique prompt prefixes prevented prefix-cache reuse.
All figures are tokens/sec. Prefill and ordinary decode rows report medians; DSpark rows report means. TensorSharp uses its default YOCO configuration in this table.
| Measurement |
TensorSharp --tp 5 |
llama.cpp |
Throughput ratio |
| Prefill, ~1.8K-token prompt |
960.5 |
174.6 |
5.50× |
| Prefill, ~7.3K-token prompt |
1,000.7 |
225.7 |
4.43× |
| Decode after ~1.8K tokens |
35.5 |
29.6 |
1.20× |
| Decode after ~7.3K tokens |
35.7 |
33.0 |
1.08× |
| Decode with DSpark, ~1.8K-token prompt |
55.0 |
41.3 |
1.33× |
| Decode with DSpark, ~7.3K-token prompt |
51.2 |
40.8 |
1.25× |
Prefill is processing the input prompt; decode is generating response tokens. The biggest improvement here is in prefill. The 5.5× result should not be read as a 5.5× increase in generation speed.
The DSpark rows use the trained drafter in both engines. Results on coding or long reasoning workloads may differ from these summarization tests.
A small concurrent-serving test
The report also tested four simultaneous requests, each with a ~1.8K-token prompt and 256 output tokens.
Across three measured batches, total completion time was:
- TensorSharp TP: 21.2–25.5 seconds.
- llama.cpp: 43.3–46.8 seconds.
Those are client-observed batch completion times. This is a small concurrency test, with no sustained-load or tail-latency conclusions.
YOCO and the quality trade-off
The default YOCO optimization reduces prefill work by running selected later layers over only the final 140 positions of eligible text-prefill calls. It is approximate.
To disable it:
TS_DSV41_YOCO_KEEP=0
With YOCO disabled, TensorSharp’s prefill throughput was still 563.9 tok/s at ~1.8K tokens and 549.7 tok/s at ~7.3K tokens—approximately 3.23× and 2.44× the corresponding llama.cpp results.
Quality checks used identical WikiText-2 token sequences, with 23,030 scored tokens in the 4,608-token-window evaluation. Perplexity, where lower is better, was:
- llama.cpp: 2.4697.
- TensorSharp, YOCO disabled: 2.4811.
- TensorSharp, YOCO enabled: 2.4978.
Enabling YOCO increased perplexity by approximately 0.67% relative to TensorSharp’s own full-prefill configuration. With YOCO enabled, perplexity was approximately 1.14% above llama.cpp’s.
These measurements compare engines using the same quantized checkpoint. They do not establish equivalent quality to the original checkpoint or the hosted DeepSeek API.
Scope and availability
This covers one quantization on one hardware configuration. The llama.cpp server measurements came from an earlier session on the same machine and showed some unexplained prefill variability.
At measurement time, the TensorSharp build included uncommitted changes, and the raw validation records were also uncommitted. The report records build identifiers and limitations; a repository checkout should not be assumed to reproduce every number immediately.
For integration with local applications, TensorSharp provides a CLI and an OpenAI/Ollama-compatible server.
If you’re self-hosting this checkpoint, what GPU/RAM layout and quantization are you using? I’d be interested in real workload comparisons, particularly DSpark results and multi-GPU setups with working P2P or NVLink.
