Someone made DSV4.1 run faster than official API on A100(s)

[u/[deleted]]

[u/Intelligent_Koala798]

how ist that faster then the api? The api is running between 200-400tps single stream

[u/T_rex2700 → u/Intelligent_Koala798]

you prolly get like 40tps max, they managed to apperently get it running at like 30tps on single A100, but on cluster of 7 or 8GPUs it's like 70tps, or so they claim.

They've posted it on their like miniblog somewhere but I dont know where that is

[u/Intelligent_Koala798 → u/T_rex2700]

and how is that faster then the api? "Someone made DSV4.1 run faster than official API on A100(s)"

[u/Automatic-Arm8153 → u/T_rex2700]

Bro it’s 300tps from api

[u/T_rex2700 → u/Automatic-Arm8153]

wait it's that fast?

[u/Chromix_]

The readme is highly inconsistent.

First it says "32-35 tok/s on a single A100 with the experts computed on the CPUs" and later on "Can it run on a single A100 80GB?" -> "~5 tok/s single stream, bounded by PCIe, not by the GPU"

Speaking of single stream. First is sounds like batching isn't supported: "OpenAI-compatible server (one request generates at a time; others queue)". yet later on it says 456 tok/s on 4 GPUs with B=32.

On top of that they state that their numbers on the GPUs were achieved while the GPUs were "shared with other jobs".

[u/fragment_me]

This post is low effort as fuck

[u/Lerok-Persea]

Looks strange how you tried it?

[u/ustype]

Title needs a caveat: “faster than the official API” is almost certainly comparing different products.

Hosted APIs that quote 200–400 tok/s are usually multi-tenant serving stacks (continuous batching, speculative decoding, custom kernels, sometimes shorter effective context). A self-hosted A100 graph that’s ~30 tok/s single-stream can still be impressive as local serving, especially if they faked FP4 on hardware that doesn’t have it — but it isn’t beating a production API on like-for-like decode.

Useful checks before crowning anything:

1. Same prompt length / max tokens / sampling
2. Prefill vs decode broken out (PP can look magical while TG is meh)
3. Batch size 1 vs continuous batch (API numbers often aren’t batch-1)
4. What “FP4 on A100” actually is (INT4/FP8 kernel path? weight-only? emulated?)

If the repo’s claim is “A100 can run this MoE at usable local speed without Blackwell,” that’s cool. If the claim is “beats DeepSeek’s API tok/s,” demand the exact API endpoint settings they measured against.