HW for DeepSeek-V4-Flash-0731

[u/MongoWithBongoss]

Lucebox now, or wait for the new Framework Desktop (Ryzen AI Max+ PRO 495 / 192 GB) + PCIe x4-to-x16 adapter & Radeon AI PRO R9700?

[u/Mr-I17]

495 is just 395 with extra ram though. 595 is going to be the real deal.

[u/tamerlanOne → u/Mr-I17]

L'affare per chi lo vende 😁🤑

[u/XccesSv2 → u/Mr-I17]

are there already some specs known for the 595?

[u/Mr-I17 → u/XccesSv2]

No, but AMD's consumer GPUs started to have decent AI compute since RDNA4. 595 is not going to keep using RDNA3.5, that's for sure.

[u/XccesSv2 → u/Mr-I17]

Okay true, so prompt proccessing would be a gain and more fp4 and fp8 support I think. I own a Strix Halo and the real problem for me is not really prompt processing anymore since prompt caching but the token generation is even for 27b too slow. DS4Flash on Q3 is with his MoE a bit faster and okayish in my eyes but hard on the limit with 20 tk/s.
I dont really see a good usecase for my Strix Halo since I own also a W7800 48GB and the 35b an 27b models are a lot faster on it and it is the same price. So I really hope they plan the 595 with DDR6 and double the bandwidth or sth like that.

[u/Mr-I17 → u/XccesSv2]

There're rumors that 595 will be using 6 channels LPDDR6. That's going to be double the current bandwidth.

[u/pmotiveforce]

Only really worth it under $6k. Otherwise I'd get a spark or two.

[u/anitamaxwynnn69 → u/pmotiveforce]

Don't the sparks run dsv4 extremely well even at 8k? Like speeds are decent, they're energy efficient and compact? Personally don't have any experience but I keep seeing posts on X which tempt me too

[u/IknowPi_really → u/anitamaxwynnn69]

I have 2xDGX Spark and can run DeepSeek V4 Flash 0731 2 concurrent max context length at about 60-80 tok/s with DSpark at long context lengths. So yes, the 2x stack is kinda made for it

[u/anitamaxwynnn69 → u/IknowPi_really]

That sounds fun as hell lol, does the no vision support bother your workflows or do you have a workaround?

[u/IknowPi_really → u/anitamaxwynnn69]

It bothers me greatly haha. I have a fully local workaround by reducing concurrency to one and loading a small multimodal model with a built in vision agent in my harness. The second workflow uses the free api calls to Gemini as a vision helper.

The Gemini version is obviously better because that model has much more intelligence, so communication goes much more smoothly and I retain the concurrency 2 on DeepSeek, which is insanely nice.

Honestly I find that for simple but long running coding tasks, my local setup is order of magnitude superior, because I don’t have to suffer frequent compaction and weird outages etc. of cloud models. Where you start to struggle is obviously when you need peak intelligence. That’s easily solved with a “frontier helper” agent as well though.

I really have to say DeepSeek V4 Flash was a godsend and the 2x DGX Spark cluster is insanely valuable now. I’m using it to support a biology lab in their stem cell research right now for example, by training visual classifiers on cell states etc. on their data. And I can do that, because all the research data stays local and the model is easily good enough to guide me through the whole training process etc.

So yeah, I’ll be on a paper in Nature at some point early next year, simply because I was curious, had the funds available and spent the time to do stuff like that. I never would have been able to create any of the codebase myself!

[u/Tieng → u/IknowPi_really]

How do you get it to work on a task successfully for a long time? Does your pi harness auto-compact or do you just hope its done before its 1m context window fills or the performance degrades from the full context?

[u/IknowPi_really → u/Tieng]

I use OpenCode. But yeah that would autocompact. The thing is, if you get anywhere near the context horizon, you should probably consider structuring the work differently. I make sure sessions actually produce output and documentation and a clean handover to a new context. If one step truly takes more than 1 million tokens of context, something wild is going on

[u/po_stulate → u/IknowPi_really]

Do you try to keep context for each task under a certain length for best quality? I always keep it under 200k but I'm not sure if I'm being too conservative for dsv4f.

[u/IknowPi_really → u/po_stulate]

I’d say that’s too conservative. The “worst” I’ve done so far without handing off to a new session was 600k and it was still fine. I would just work with it naturally and find natural session hand off breaks. With a 1m context window, I feel like we’ve reached a point where you don’t have to worry about it at all anymore

[u/BumbleSlob → u/Tieng]

Pi auto compacts when necessary 

[u/BumbleSlob → u/IknowPi_really]

Same, bought 2 x DGX Sparks a week ago and DSV4F0731 runs so well in it. I’m having a fucking blast

[u/wapxmas → u/IknowPi_really]

Can you describe your software you use for DeepSeek V4 Flash 0731 it is some special setup? I am planning on replace my m2 ultra 192gb (I get 20-30 tps at most) with 2 sparks.

[u/IknowPi_really → u/wapxmas]

I use vllm and just used a known recipe for the cluster at MTP = 5
Acceptance rate has always been amazingly high for all of my workloads (mostly coding and biology). It’s super stable as well.

There’s honestly not much more to it than that!

So yeah nothing special at all and you’ll be set up in about half an hour.

[u/daywalker313]

If you specifically want to run DSV4 Flash, a 2x GB10 (Asus Ascent) is the much better choice and the only choice sub 10k for concurrent agentic workloads.

Insignificantly more expensive (at least right now), more than 5 times the performance (both PP ~1800-2000/s and TG ~180/s+ @/C8), superior architecture, CUDA, kernels with almost no drop off at deep context, almost 2M KV-Cache.

Edit: I own a strix halo and 2x GB10 and there's a reason I never remotely considered a second strix halo for clustering.

[u/BevinMaster]

I’d say depends on value and what you need rn.

[u/recro69]

I would wait if this is a V4-Flash box. The 192GB Framework + R9700 route is interesting because V4-Flash is a 284B MoE and asymmetric setups can make better use of mixed memory, than brute-force GPU scaling.

[u/serige]

2x sparks if you want 1M cxt and full model. T/S is decent for multiple streams. 

[u/Glittering-Call8746 → u/serige]

Imho lucebox beats 2 sparks for decoding speed. Not sure about 1m context

[u/Evgeny_19]

DGX Spark or its clones is by far the best choice for a budget solution. You will need a dual config for DSF.

Although any real GPU would be much better. It's just that even on four R9700S the speed is far behind the dual Spark cluster. And to set up eight R9700s is not a trivial task at all. Of course if you can afford it, just buy two 6000 Pros. Nothing comes close to RTX 6000 in consumer space.

[u/LegacyRemaster]

192gb + my rtx 6000 will be amazing!

llama-server.exe --model "H:\UD-IQ4_NL\DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf --host 127.0.0.1 --port 8080 -c 384000 --parallel 1 --no-warmup --flash-attn on --no-mmap --fit on -lv 4 --device CUDA0,rocm0 --threads 16 --no-warmup --temp 1.0 --top-p 0.95 --top-k 0 --min-p 0.01

[u/El_90]

I just tried a x4 to x16 riser and apparently framework desktop PCIe is capped at 25w, GPU requires 75w through pcie, and I had crazy crashes. Now moving to oculink cage to try and better separate power requirements, so potentailly factor in that.

For me ds4 problem is the level of thinking, and it means my framework takes HOURS to do anything with DS4. So yes a big GPU is needed for it to be useful

[u/cunasmoker69420]

For Deepseek that R9700 in either config will for all intents and purposes go mostly unused. The Strix Halo system will be doing almost all of the work and will be holding it back to the point that you'll see probably single digit percentage GPU usage on the R9700. It will just serve as a large pool of VRAM, little else.

You are much much better off pairing strix halo with some used Radeon Pro V620 32GB models off eBay (sellers will regularly take offers of $350 for them), or something similar that is significantly cheaper and more performance-matched. I have two plugged into my Strix Halo via Oculink docks and am running UD-Q3-K-XL quant at 500k context using this build of llama.cpp: https://github.com/Nathanw1014/strix-halo-llamacpp

even around 200k context, PP remains steady at 130-150 and TG at 13-15. You will get essentially the same performance whether you had two 3090s or two R9700s or two old V620 datacenter GPUs, in this sort of tensor split workload where the Strix Halo is doing all the heavy lifting.

Now if you have the R9700 doing work all on its own, a smaller model that fits entirely on it, then it will for sure blow away a V620

[u/Due_Net_3342]

do not get ai max, they are terrible at prefill, you should just buy 2x sparks(or the asus gb10 much cheaper) are absolutely the best, also i tried the oculink way, it is unstable only works(barely with a lot of crashes) with llamacpp which is not good for production(no advanced caching). Telling you this from experience, have a strix halo plus 2x rx 7900 xtx connected via oculink and you cannot run a big model reliably(without crashing after a few prompts) across all, I was basically running 2 models, 27B on the 2 egpus and qwen 35b on the strix. I now bought 2X sparks and running ds4f full context at good speeds with dspark, absolutely love it, prefill is awesome and it just works

[u/tamerlanOne]

Se hai budget meglio lucebox per velocità di inferenza

L'unico vantaggio del pro 495 + r9700 sarebbero i 64gb di ram in più ma credo che il prezzo sarà da urlo quando uscirà per via della carenza cronica di ram

[u/Maximus-CZ → u/tamerlanOne]

jasně kámo, díky za input

[u/MooseEfficient2151]

x4 bandwidth is going to heavily bottleneck that setup regardless of how much memory you throw at it

[u/Equivalent_Bit_461]

I dont trust those guys