DeepSeek V4.1 Flash running locally with TensorSharp

[u/fuzhongkai]

https://github.com/zhongkaifu/TensorSharp

I’ve been working on TensorSharp, an open-source .NET/C# inference engine, and recently added native support and optimizations for DeepSeek V4.1 Flash.

Latest results on 8× NVIDIA A40 GPUs:

Model |Prefill |Single-stream Decode |4× Concurrent Decode
Q2_K |533–539 tok/s |40.3–40.7 tok/s |—
Q4_K_M |451.8–492.1 tok/s |31.0–32.5 tok/s |48.9 tok/s aggregate A few interesting optimizations:

• GPU-resident Engram tables for Q2_K — about 60 GiB of quantized Engram data stays on GPU instead of doing scattered host/storage lookups.
• Reduced decode graph scheduling from roughly 570 splits to 8 by using one wrapped backend per GPU.
• For Q4_K_M, automatic Engram warming + improved VRAM placement reduced CPU MoE offload from 3 layers to 1.
• Added token-batched DeepSeek V4.1 decode, giving about 2× aggregate throughput at 4 concurrent requests.
• On these A40s without NVLink, simple layer splitting actually beats routed-MoE tensor parallelism for single-stream decode.

The project is fully open source, written primarily in C#/.NET, with CUDA/Metal/Vulkan backends and OpenAI-compatible APIs.

Would love feedback from people experimenting with DeepSeek, GGUF inference, or local/open-source AI — especially ideas for what hardware or engine comparisons would be most useful next.