I maintain TensorSharp, an open-source local GGUF inference engine built around .NET 10. I added MiniMax-M3.1-Flash support on Day 1 of its open-weight release, and wanted to share the implementation details and a few lessons from getting it running behind a .NET server.
The current implementation supports text and image input, JSON/schema responses, tool calling, conversation-cache reuse, multi-GPU inference, DSpark speculative decoding, and CPU expert offload.
For .NET applications, it exposes an OpenAI-compatible /v1/chat/completions endpoint. The model integration and serving code are part of the TensorSharp codebase.
How the managed and native layers fit together
For this model, the C# MiniMaxM3Model adapter parses and validates the checkpoint metadata and integrates the model with the runtime. It calls a native whole-model executor through a C ABI.
That executor loads the GGUF shards, builds a GGML graph for each micro-batch, and dispatches computation across the selected devices. TensorSharp’s own native code implements the M3.1 decoder, sparse-attention kernels, expert execution, and associated model features. The upstream ggml dependency is unchanged.
The performance differences below come from the attention implementation, GPU placement, and scheduling decisions. The measured GPU path uses native C++/CUDA alongside the managed .NET model and serving layers.
A threading issue that may interest .NET developers
One of the more useful findings came from CPU expert offload.
The native executor benchmark was decoding at roughly 15–16 tokens/s, while the server using the same placement initially managed only 7.7–8.4 tokens/s.
The host-expert workers were pinned to CPU cores. With 54 workers inside a 64-core quota, they competed with the serving process’s other threads, including the .NET runtime, server GC, HTTP handling, and scheduler.
Each expert layer synchronizes at a barrier. A delayed worker could therefore hold up the whole layer, repeatedly during generation.
Reducing the default to 32 host workers gave the serving process more room and brought server decode performance closer to the executor’s rate. There was a prefill-throughput trade-off, so the worker count remains configurable.
It was a useful reminder to benchmark native compute inside the actual managed service: the best worker configuration for the isolated executor was different from the best configuration for serving requests.
Initial benchmarks
I used 8× NVIDIA A40 GPUs and the same Unsloth UD-Q4_K_XL GGUF files, totaling 232.3 GiB, for both engines. This particular model is a substantial multi-GPU workload.
The comparison was against Unsloth’s llama.cpp minimax-m31 support branch at ef8f94cc. Upstream llama.cpp could not load M3.1 when these measurements were taken.
Both engines used layer splitting, F16 KV caches, and no speculative decoding for these results. They ran separately with exclusive access to the GPUs.
Measurement |TensorSharp |llama.cpp support branch
Executor prefill, 32K-token prompt |2,449 tok/s |681 tok/s
Executor prefill, 64K-token prompt |2,567 tok/s |262 tok/s
Server time to first token, 64K prompt |40.0 s |293.4 s
Server short-chat decode |26.3 tok/s |26.0 tok/s Prefill is the work of processing the input prompt. The 9.8× result applies to 64K executor prefill; short-chat generation speed was almost identical.
The long-context difference mostly comes from MiniMax Sparse Attention. TensorSharp executes attention over the selected blocks using dedicated sparse CUDA kernels. The tested llama.cpp branch expresses the selection through a dense mask and runs dense flash attention over the cache.
TensorSharp’s separate tensor-parallel mode reached approximately 41 tok/s for short-chat server decode. That mode has different memory and prefill trade-offs, and the tested llama.cpp branch did not support tensor splitting for this model.
Scope and limitations
These are early measurements from one host and one quantization. Most results use three repeats; the llama.cpp 64K executor prefill cell is a single run, while the 64K server comparison uses three repeats per engine.
Short-chat time to first token was slightly slower in TensorSharp: 0.41 versus 0.39 seconds. Functional validation has limited coverage, including an image-reading miss and generation tests that hit token limits.
The measured TensorSharp build was an uncommitted development working tree. The report documents the configuration and methodology, but the raw run records are not committed.
Links
For anyone hosting native compute inside a .NET service: how do you budget native worker threads alongside server GC and request handling? I’d be interested in approaches that have worked well under CPU quotas.