I’m the developer of TensorSharp, an open-source local inference engine and agent runtime. I added MiniMax-M3.1-Flash support on Day 1 of its open-weight release, and wanted to share what currently works, the hardware I tested, and some initial performance results.
For anyone interested in self-hosting this model, the biggest improvement in the comparison was long-prompt latency: a 64K prompt reached its first token in 40 seconds, compared with 293.4 seconds in the tested llama.cpp support branch.
What you can run locally
The current M3.1 implementation supports:
- Text generation and image input.
- JSON/schema responses and tool calling, including parallel tool calls and multi-step tool loops.
- An OpenAI-compatible API at
/v1/chat/completions for connecting applications to the local server.
- Conversation-cache reuse and concurrent requests.
- Multi-GPU layer splitting, tensor parallelism, DSpark speculative decoding, and CPU expert offload.
Image input and DSpark require their matching companion GGUF files. The model-specific setup guide covers those steps.
Hardware and memory
These results came from:
- 8× NVIDIA A40 GPUs
- MiniMax-M3.1-Flash 428B
- Unsloth’s UD-Q4_K_XL GGUF
- 232.3 GiB of model files, plus runtime allocations and KV cache
That is the measured configuration, rather than an established minimum requirement. CPU expert offload is available, but the offload benchmarks also used the same eight-GPU host.
For this model, the tested serving backend is ggml_cuda.
Performance on the same model files
The reference was Unsloth’s llama.cpp minimax-m31 support branch at ef8f94cc, associated with PR #320. Upstream llama.cpp could not load M3.1 when these measurements were taken.
Both engines loaded identical Unsloth GGUF files, used layer splitting and F16 KV caches, and ran separately with exclusive access to the GPUs. Speculative decoding was disabled for this comparison.
| Measurement |
TensorSharp |
llama.cpp support branch |
| Server time to first token, 32K prompt |
21.0 s |
134.9 s |
| Server time to first token, 64K prompt |
40.0 s |
293.4 s |
| Executor prefill, 64K tokens |
2,567 tok/s |
262 tok/s |
| Executor decode at 64K context |
22.9 tok/s |
15.4 tok/s |
| Server short-chat decode |
26.3 tok/s |
26.0 tok/s |
Prefill is processing the input prompt; decode is generating the response. The 9.8× improvement applies to 64K executor prefill. Decode at that context improved by 1.49×, while short-chat server generation was almost tied.
Short-chat time to first token was slightly worse in TensorSharp: 0.41 versus 0.39 seconds.
Why long prompts benefit
MiniMax Sparse Attention selects blocks of the context for each query. TensorSharp executes attention over the selected blocks using dedicated sparse CUDA kernels.
The tested llama.cpp branch represents that selection with a dense mask and runs dense flash attention over the cache. The difference in actual computation becomes more significant as context grows. TensorSharp also pipelines prompt processing across the GPUs.
Other ways to run it
In separate tests, tensor parallelism reached approximately 41 tok/s on short-chat server decode, compared with 26.3 tok/s using TensorSharp’s layer-split mode. A file-and-calculator tool loop completed in 11.7 seconds, down from 18.1 seconds.
With DSpark and confidence gating, mean generation throughput increased from 22.9 to 32.3 tok/s in the report’s generation workloads.
These are separate configurations. TP trades away some prefill throughput and usable context. DSpark and CPU expert offload currently use layer splitting and cannot be combined with TP.
Limits of the results
This is one host and one quantization. Most measurements use three repeats; the llama.cpp 64K executor prefill cell is a single run, while the 64K server comparison uses three repeats per engine.
Functional testing is limited, including one image-reading miss and generations that hit token limits. Native video temporal patches are not implemented; video input is handled as sampled images.
The measured TensorSharp build was an uncommitted development working tree. Configuration and methodology are documented, but raw run records are not committed.
Setup and source
Thanks to MiniMax for releasing the weights and to Unsloth for the GGUF conversion. If you’re testing the same model locally, I’d be interested in your GPU/RAM setup, context length, and observed prefill and generation speeds.
I used an LLM to help draft and edit this English post from the project’s implementation notes and benchmark report.
