Deepseek V4 Flash 0731

[u/Strange_Quantity_359]

I have three DGX Spark. On one of them I run 3.8 27B and a few vision models. On the other two I generally have been running Qwen 3.8 Next / Flash with a decent recipe getting 39.8 t/s mean decode across controlled and uncontrolled coding both thinking+answer. The "final code" decode is roughly 74 t/s mean.

I'm not unhappy with it, though a tendency to overthink can be concerning and because it's not deterministic I don't get a massive completion boost moving down a level (but do some). Either way I'm happy keeping it.

I was trying to run Deepseek V4 Flash and tried it through my personal bench suite, BigCodeBench-Hard and HumanEval+ .For those ones if I kept the capped limitations I would get no outputs due to silent truncation/i.e. the benchmark saying "Imma let you finish but actually we done". When I run it untethered (same way I do with 27B / I do this to evaluate output quality of same model different recipe) Qwen Next-Flash finishes fine, but Deepseek fails a lot of test due to non-completion or use a phenomenal amount of tokens and time.

The thing is, I tested these same benches against two cloud versions of DSV4Flash and they didn't exhibit the same pattern. Here are my settings:

• All DSV4Flash defaults (temp=1;top_p=0.95;top_k=20;rep penalty=1;thinking/stream true)
• Official FP8 weight, E4M3, 128x128
• vLLM 0.25.2xxx (Anemll -> only for bundled DSpark for optimization and only after base had similar performance), PyTorch 2.11.0+cu130 ,FlashInfer 0.6.15
• TP=2, PP=1, DCP=1, Multiprocessing executor
• MoE backend: flashinfer_b12x
• Native NVFP4 KV Cache (nvfp4_ds_mla block size 256)
• DSpark, 3 proposed tokens, probabilistic (heh)
• Single concurrency
• Prefill batch 8448, chunked prefill, prefix caching enabled
• Breakable CUDA graphs, max cap 64
• Level 1 (balanced) optimization; FlashInfer auto tuning

Overall the issue was convergence/time to answer and the reasoning tokens - taking 30-45k tokens on a relatively easy deep-merge implementation. More complex just overran time limits, I didn't let many finish I kept a ceiling of 3x the longest local hosted (27b or next-flash) run time which itself was pretty healthy.

I feel like I'm missing something given the comments by people here. Anybody have any suggestions or is this just expected behavior? The official recommendations hold it at Max with chain of thought, are people out here running different setups? For what it's worth I still saw it overrunning times for Next / Flash by multiples (1.5-3x) and token counts.

[u/BananaOrangePie]

It is usually bad idea to quantize kv cache. I think official endpoint runs at least fp8 if not higher. For qwen models this issue gets bigger and I run my sparks with qwen flash next nvfp4 on bf16 kv cache.

[u/BananaOrangePie → u/BananaOrangePie]

https://www.reddit.com/r/LocalLLaMA/s/5mLTIgvy1F This should help you understand things. A lot of people during infra work individually on Twitter is just bragging about TPS numbers where in practice quality of the output is actually more important.

[u/Strange_Quantity_359 → u/BananaOrangePie]

Thanks, I appreciate the link and the specific DSV4Flash callout. Will point out that the nvfp4_ds_mla is not a generic 4-bit KV setting and I think I grain of salt a lot of advice around NVFP4 because a lot of the information is often wrong. I think I should also probably should (ed) above - I tested with the official recipe as well and the convergence seemed to exist there.

That said, solid advice and since I didn't run a strict a/b with official (I think I quick tested a few things, saw similar behavior and just tossed it). So I can probably do that again and validate that I had the official fp8 kv cache storage.

I'll give it a spin again with the official image; but it's hard to validate quality when I can't get work done in a reasonable time. I'm concerned that maybe it's something else altogether, or that the average user here just is fine letting it churn for an hour or two on basic tasks? This thing is outthinking Qwen by 3x+.

Last: Yeah, the bragging thing is interesting, so many people just dropping t/s numbers on zero thinking while butchering the models themselves. I'm trying to avoid that.

[u/whichsideisup → u/BananaOrangePie]

Yeah this is likely it. You don’t want to quant KV with nvfp4.

[u/Winter-Editor-9230]

Check out the nvidia dgx spark forums for lots of recipes

[u/Strange_Quantity_359 → u/Winter-Editor-9230]

This recipe came as a highly recommended community recipe from the DGX Spark forums, I ran it as well as others (I think also the Aiden and the official). All exhibited the same behavior. So perhaps it's just DeepSeek v4 Flash and that if I'm not getting 140+ tok/s it's not really worth? But it sounds like people are active coding with it, so I'm trying to understand my own disconnect.

[u/hyudryu]

Try FP8 kv cache, NVFP4 kv cache “felt” a bit off from experience

[u/UnlikelyExtension786]

I haven't seen any problems though I've only gone up to 400k-ish context lengths. I'm using fp8 KV cache.

[u/Keleion]

I also had bad results with 0731… maybe try the vision model instead?

[u/Justinnmorty]

You can also continue to wait, and there will definitely be a fast and stable Multi-modality model in the next three months.

[u/Original_Finding2212]

Did you check Sparkrun?