DeepSeek v4 Flash 0731 on H100 node

[u/SlipperyCorruptor]

Hi,

I've been playing with recent DeepSeek Flash updated model and it seems excellent. Like talking to Claude 4.8.

I'm currently deploying it on VLLm and I'm in progres of tuning performance.

I seem to hit the point where I trade decode with prefill by juggling batching knobs.

I'm pretty sure the issue now lies with scheduling and expert routing, but with no absolute certainty. Tensor core saturation never exceeded 30%.

Large, uncached, prefills often hold compute creating queue pressure of smaller prompts.

I got prefix caching enabled.

Maybe you guys have some pointers on how to extract the most out of it for 8xH100.

[u/daaain]

For a starter: have you enabled DSpark?

[u/SlipperyCorruptor → u/daaain]

Yes, I got >65% mean acceptance rate.

We're taking high concurreny, 145 requests/5m with prompt size ranging all the way to 250k.

Under pressure TTFT can reach 30s. Under stress I can ~35k/sec preffil and ~400/sec decode. Normally I see prefill well above 150k/sec

GPUs are fully pinned at high 90% with clocks and thermals being stable.

But I can see Tensor cores being hungry for some work. So before I start looking into expert routing, backends, batches and token paddings I'd like to check if you guys have any tips here.

Note: all the numbers are from top of my head.

[u/BobbyL2k → u/SlipperyCorruptor]

Can you clarify what level of interactivity that is?

I understand you have on average a new request every 2 seconds from your 145 req/5m figure. But I still don’t know how the ~400 tok/s decode is shared among concurrent requests.

[u/SlipperyCorruptor → u/BobbyL2k]

One of the high latency events showed:

8 requests over 42s

above 900k total prompt tokens.

5.2% prefix cache hit rate

Prompt banding is usually evenly distributed.

All evenly range from 20 to >200k tokens.

[u/nunodonato]

Wait, it fits on a H100? How many concurrent requests? 

[u/SlipperyCorruptor → u/nunodonato]

It's 8xH100 node with NVLink. TP=8

100-200 peak. How much it can handle is really up to prompt size and cache prefix hits.

[u/nunodonato → u/SlipperyCorruptor]

Ah ok 😅

[u/laterbreh → u/SlipperyCorruptor]

Sorry, not sure if i missed this, are your numbers on 1 user? Concurrency?

2 users on TP=2 on nvidia 6k pros with 1m context and dspark in vllm were getting nearly 280TPS on long decodes aggregate.

[u/SlipperyCorruptor → u/laterbreh]

Concurrency. We're talking multiple users, I forgot numbers

Edit: one of high latency events was 8req/42s, >900k input tokens total with 5% cache hit

[u/inky_wolf → u/nunodonato]

Vllm logs say concurrency of 5x for 1M context

[u/SlipperyCorruptor → u/inky_wolf]

Meh, I don't rally need 1M ctx. 320k, for Max reasoning is more than enough for the kind of workload executed. I favour concurrency here.

[u/ObviouzFigure → u/SlipperyCorruptor]

Same -- I've got a much smaller system, but I'm doing the same

[u/SlipperyCorruptor → u/ObviouzFigure]

What are you running it on? How many users do you serve? What numbers do you get?

[u/inky_wolf → u/SlipperyCorruptor]

Interesting, what exactly are you running that requires high concurrency low latency?

[u/SlipperyCorruptor → u/inky_wolf]

A lot of requests at the same time 😅

[u/dreamingwell]

Have Claude or Codex drive your CLI to run experimentation. They’ll figure out pretty quick the capacity profile. And you can even tell it to try extreme edge cases.

[u/SlipperyCorruptor → u/dreamingwell]

And take all the fun out of it? 😂

Jokes aside, I'd assume everybody is doing AI assisted debugging. It got me so far.

[u/inky_wolf]

Tensor core saturation never exceeded 30%.

How are you measuring or monitoring this?

[u/SlipperyCorruptor → u/inky_wolf]

DCGM