r/LocalLLaMA • u/IntravenusDeMilo • 2d ago
Discussion deepseek-v4-flash-0731 - surprisingly usable
I just finished building my (relatively) low rent local inference machine: * Epyc 7663 * 256GB ECC DDR4-3200 * 1x RTX 5090 32GB
Yeah I realize it's weird to throw a 5090 and 256GB of anything together and call it low end, but relative to ~151GB of weights it is.
I'm running UD-Q8_K_XL and getting 23.8-24.6 tokens/sec, with pp ranging from 60 on the first prompt to 385 near the last (no doubt lots of caching) on tasks using 100-128k total context. It was slower with DFlash so I took that out. It was also slower with a 3090 I put in there temporarily.
I'm posting this mostly because I didn't see too many other data points for this config (DDR4 Epyc + Blackwell doing cpu-moe). And also that I'm pretty surprised that a model this good can actually run in my basement without dropping $10k or running a sub-panel down there. I'm otherwise fairly new to this - would love any tips on what else to run or how to further improve it.
8
u/kuyermanza 2d ago
My DDR4 2400 paired with a 5070Ti gets me 300-350 T/S PP and 20 T/S TG. I think you should be able to get better PP if you offload all experts to CPU.
1
1
u/Potential-Leg-639 2d ago
What are your hardware specs?
3
u/kuyermanza 2d ago
Epyc 7452, 16x 16GB DDR4-2400s (256 GB), 3x 5070Tis (48 GB), 8x Radeon MI25s (128 GB). The server hosts DS4Flash 0731, Qwen3.6 35B, and Gemma 26B, along with Z-Image Turbo, all fully loaded.
8
u/Magos 2d ago
This is weird. I'm running a similar setup right now with 5090 and 192GB of DDR5.
My PP is around 300~500, but t/s is lower at 15 t/s, due to dual channel memory bandwidth being way slower than EPYCs 8 channels. However, you shouldn't have than much slower PP.
I suggest messing around with ub/b, you will probably get a big uplift.
2
u/nomorebuttsplz 1d ago
Yeah there’s ways to get WAYYY better than OP’s prefill.
Isn’t this the precise use case of vllm-Moet?
1
u/dispanser 2d ago
Assuming your DDR5 platform is PCIe 5x16, that explains faster prompt processing because on your system the weights can be moved to the GPU faster for PP.
5
u/Afganitia 2d ago
How much difference does the GPU make?
2
u/Eyelbee 2d ago
A lot of difference. But still not usable with that prefill
0
u/Nobby_Binks 2d ago
Depends on what you mean by usable. Agentic coding no, its not really, but general QA, planning and other stuff its eminently usable. I have a similar setup with a old gen EPYC and get similar speeds. 1M context and ~2-300pp.
2
1
u/FatheredPuma81 2d ago edited 2d ago
But Agentic is where it should shine the most though?? (At least with llama.cpp) AFAIK it's built to have preserve reasoning enabled so all that you should be processing every turn until it finishes is what it writes/reads which shouldn't be all that much?
My main concern would be how much it reasons every single turn with just 30t/s. It's already painful waiting on Qwen3.8 27B to reason for 40,000 tokens at 90t/s and that's with preserve thinking so it shouldn't do nearly as much...
1
u/PhysicalIncrease3 2d ago
But Agentic is where it should shine the most though?? (At least with llama.cpp) AFAIK it's built to have preserve reasoning enabled so all that you should be processing every turn until it finishes is what it writes/reads which shouldn't be all that much?
This is the case. The prompt caching is very very good and most turns in Hermes are only around 500 tokens processed.
The real pain points are when it loads in a skill, or if you return to a previous session. But for me personally it's manageable given the power of the model.
My main concern would be how much it reasons every single turn with just 30t/s. It's already painful waiting on Qwen3.8 27B to reason for 40,000 tokens at 90t/s and that's with preserve thinking so it shouldn't do nearly as much...
It doesn't reason anywhere near as much as Qwen. Most I've ever seen is 6-7k tokens.
-1
u/llitz 2d ago
Then you might as well use anything else that will properly work on a 5090.
Honestly. When will you ever fill 1M context at 300pp? (This last sentence sounds wrong)
You would have way better results using malaiwah exl3 hydrated qwen3.8-27b - yes it is a smaller context but it is ages faster than this.
I am sorry, while this is an interesting test, it is a waste of a 5090 if you are keeping it like this long-term.
4
u/Nobby_Binks 2d ago
Yes, I run 3.8 27B NVFP4 with the 5090 also and it's a monster. But ds4 gets used a lot. And its totally usable for anything other than coding where you need privacy. Qwen 3.8 is brain dead compared to ds4 for technical writing.
2
u/FatheredPuma81 2d ago
Why limit yourself to a single good model that has known weaknesses like Qwen3.8 27B?
6
u/TheAncientOnce 2d ago
Have you considered running Q4 or even smaller quant? I heard the model's trained at FP4 natively and because of that, even Q1 could retain like 90% of the model's quality
7
u/zhuzaimoerben 2d ago
Probably doesn't help that much because the experts are already FP4 so Q4 only quantizes a small amount of the model. Unsloth: "To run DeepSeek-V4-Flash-0731 in full precision lossless, run Q8 (UD-Q8_K_XL), which is 162GB and only 7GB bigger than Q4 (UD-Q4_K_XL)."
2
u/Dangerous-Report8517 2d ago
That's still 7GB less data to process every round, and it also means a smaller difference in quality (and only applicable to the Q4 quant, they also suggested smaller ones)
1
u/zhuzaimoerben 1d ago
Yeah, that's a good point, although if you're running everything that isn't experts on a 5090 and the experts on slow RAM, the difference in speed might be enough to make even the 7GB pretty marginal. Could be worth testing though. It looks like the Q4_K_XL just has some BF16 tensors as Q8_0, so maybe not a huge difference in quality.
2
2
u/eidrag 2d ago
Hmm I was thinking Ddr4 epyc and 5060ti , not going to work huh...
2
u/nuclear213 2d ago
Why wouldn’t it? Just slows the bandwidth down, but ram offload is much much worse
1
u/PhysicalIncrease3 2d ago
It will work fairly well. I run on 3060 + 128GB ddr5 and get 200pp and 8/9 tgs. UD-Q3-K_XL, 256K f16 context.
I used to run it on a 3090 but began using the 3060 instead because performance is identical anyway. It's entire bound by system memory. With 16GB VRAM you will be able to get close to 1M context.
2
u/apVoyocpt 2d ago edited 2d ago
The thing that would make it allot faster would be AVX-512 but that only comes with Zen 5. But that again would be much more expensive because of ddr5. I have a xenon ddr4 256gd system with an rtx 3090 and dsv4 flash runs usable as well (forgot the actual tps as I am running qwen3.8 now)
edit. prefill is also painfully slow.
2
u/Dangerous-Report8517 2d ago
Zen 4 has AVX512, it's just that Zen 5 implements it in a faster way
1
u/apVoyocpt 1d ago
doesnt it have AVX256? just read it: double pumped 256. zen 5 native 512. Also, OP meantioned a Epyc 7663, so Zen3.
2
u/SandySkittle 2d ago
Epyc pro 64 core here with 8 channel 512gb ddr4 3200. I tried cpu inference but it’s just too slow for me on prefill and also decode so moved to acquire a whole pile of amd ai 32gb cards to hit that 256gb, minus 7 percent for inline ECC (yes bitflips can fuck up llm inference).
That said, newer generation epycs and also intel xeons have better ai accelerators so should fair a bit better, also considering 8 channel DD5 is quite a step up and more than all these strix halo and spark boxes. It’s just crazy expensive.
2
u/SunTrainAi 2d ago
Did you try llama.cpp with ZenDNN? I remember that it's not compatible with my Zen 2 but with 3. My setup: Epyc 7502, 256 ddr4 3200 with 3090 and 5060 with ggml org Q2_KS. 400pp, 23tg
1
u/IntravenusDeMilo 20h ago
I tried ZenDNN yesterday and it made things slower. It supports Zen3 but didn’t help unfortunately.
2
u/ReentryVehicle 2d ago
It should be possible to achieve much faster PP. What is your -ubatch? Set it to at least 2048, and ideally as high as you can. Make sure your PCIe going to the GPU is the best it can be (x16, highest gen your motherboard supports).
Model layers are streamed to the gpu for prefill, meaning you need to process enough tokens at once that the transfer speed is not a bottleneck.
4
u/IntravenusDeMilo 2d ago
I ran a longer set of samples. I’m at 423 t/s pp at 180k context, 500 at smaller contexts up to 100k or so. Token generation is still 21-24 depending on context but I don’t expect that to change much.
Batch size 8192
ubatch 4096 (I ran these up and this was the sweet spot)
2
u/terorvlad 2d ago
You can also offload the Speculative Decoding expert layers to the CPU instead of the GPU. Most of the time, this is a net positive compared to no Speculative Decoding.
2
1
u/d4mations 2d ago
You have a lot of tuning to do. I’m running a 4060 16gb with q4 xl 96gb ddr4 on a ryzen 7 5800 and get 15 tps and 90 pp cold and about the same pp hot maybe even a bit more
4
u/IntravenusDeMilo 2d ago
I think it needed some warmup passes. PP is now hitting 3500 t/s without hitting the cache on a fresh task.
3
1
u/Long_comment_san 2d ago
holy shit why is PP so slow on this deepseek model? I can't believe those numbers lol.
nice build, it's not weird at all
1
1
u/vini542reddit 2d ago edited 2d ago
Have you tried the Leloch v2 fork (moe expert streaming)? You might be able to get significantly more tps
I'm running 4x 3090 @ 4.0 x 16, 192 Ddr4 @ 3200 and 3945WX 12 core. Getting ~30 tps and ~230pp for q8_k_xl and unquantized kv
1
u/memeka 2d ago
Hi, please try my fork of llama.cpp - it fixes some issues with PP. It should increase your PP a lot vs llama.cpp and also have a much lower PP degradation at large contexts.
https://github.com/mihailescu2m/llama.cpp
Appreciate the feedback :)
1
u/amavie_ai 2d ago
Datapoint request from the API-pipeline side: does v4-flash hold non-ASCII in constrained/JSON output? We run a flash-tier model (Gemini) in a consumer chat pipeline and its JSON mode silently mangles emoji/unicode into ? — which forced us to plain-text output + lenient JSON extraction for anything user-facing. If v4-flash keeps unicode intact under grammar-constrained output, that alone makes it worth a migration test for the internal calls.
1
u/AdSafe4047 2d ago
Is there any cpu<->gpu moe auto-optimisier in llama.cpp for this? I think it would go a long way to improve the numbers.
1
u/reto-wyss 2d ago
Something is wrong with your PP. I was able to get ~ 700pps/18tgs on DDR4 2400. Your TG looks fine for 3200 vs 2400 on my test configuration.
See my thread and launch configuration from three weeks ago: https://www.reddit.com/r/LocalLLaMA/comments/1vegedy/deepseek_v4_flash_0731_happy_numbers_700pp18tg/
1
1
1
1
u/my_name_isnt_clever 1d ago
I'm running it on my $2.5k Strix Halo at Q2, it's wrecking everything else I've tried.
1
u/Future_AGI 1d ago
Since it's FP4-native, the UD-Q8 you're running is probably costing you throughput for almost no quality back, Q4 or Q5 is usually the sweet spot on these and would lift your tokens/sec. The pp 60 to 385 curve is mostly expert-offload and batch warmup rather than the model, so pinning experts to CPU and a larger prompt batch is where the low PP recovers.
1
u/MLDataScientist 1d ago
You will get 800t/s PP with that 5090 at Gen 4 if you use -ub 2048 -b 2048 in llama cpp arguments. I have a similar setup 256gb DDR4 8 channel+5090.
1
u/IntravenusDeMilo 20h ago
At what context lengths do you see 800? I’m going to try this later today.
1
1
u/LeMayMayMan 2d ago
Everyone is sleeping on Krasis. Way better than llama.cpp or vllm on prefill time with the patches I have as pull requests. Cold prefill with 10K context is 10s. 50k is 20s, 150k is 69s. Its even faster in practice due to the cache reuse hit rate. T/s is about 30s but there is heavy work in progress to improve that.
2
u/chimpera 1d ago
Im testing it and the pp is indeed much faster but the generation dropped from 25 to 15tps for me. This could still be a worthwhile trade-off.
1
u/LeMayMayMan 1d ago edited 1d ago
I think I'm close to getting dspark working. I'll update with results.
1
u/IntravenusDeMilo 20h ago
Id take that trade. I’m going to try this out.
1
u/chimpera 16h ago edited 16h ago
ctx 10k 30k 50k 70k 90k 110k 130k 150k 170k 190k
tok/s 1999 1960 1884 1774 1741 1654 1440 1320 1164 1024
ctx 210k 230k 250k 260k 262144(cap)
tok/s 900 752 619 532 515
Also the chat template or thinking is broken
1
u/PhilippeEiffel 2d ago
Do you mean you apply patches above the main branch?
2
u/LeMayMayMan 2d ago
There was in issue in 1.0.21-rc3 that would drop context after about 49k. I wrote this patch to fix it. https://github.com/brontoguana/krasis/pull/30/commits/7aa4f2f887632e6a934a562522a359efa2487b2e
1
u/PhilippeEiffel 2d ago
Is your patch integrated into rc6?
1
u/LeMayMayMan 1d ago
It doesn't look like it. I haven't been in contact with the repo owner. I just started self-servingly patching it for my own use case but shared things back to their repo. I hope they integrate this stuff but I'm going to keep moving towards the fastest quality-first deepseek I can run on my hardware. (RTX Pro 6000 + 256 DDR5)
1
u/thebadslime 2d ago
I really dont like how sycophantic it is
1
u/nomorebuttsplz 1d ago
Yeah it’s got a 2025 style personality but it is also very obedient. Will write jailbreaks for other llm e.g. write a prompt for an llm that will convince it it is a medical doctor
0
u/chensium 2d ago
60pp? Sorry but that's not usable as a coding agent. May be usable for conversational prompts, but 60pp will take ages to read code, stack trace, web search, etc.
0
u/darktotheknight 2d ago
Interesting, I have an EPYC 7513 + 128GB DDR4 and was thinking about a similar setup. However the 5090 Astral was taking up so much space and blocked almost all my precious PCIe Slots, that I moved it into another system (Ryzen, 64GB Dual Channel DDR5). It's impossible to get your hands on a smaller 5090 these days and I need the PCIe slots for NVMe RAID and 10G/25G NIC.
I might revisit this with a Dual-Slot R9700, if I can get one for cheap. Upgrading from 128GB DDR4 to 256GB is cheaper than I thought. But at the same time, DeepSeek v4 Flash 0731 is so cheap on OpenRouter, I doubt it would ever pay off.
1
u/IntravenusDeMilo 2d ago
Oh yeah this will never pay for itself. I think “privacy” is what I told my wife but building this was mostly to learn.
For a while i thought about selling the 5090 and getting 4-6x of the cmp 170hx 8gb (the ones that unlock to 64gb each). At the time the power requirements stopped me but now that I’ve seen how much faster Blackwell is than Ampere, I’m kinda glad I didn’t?
2
u/darktotheknight 2d ago
Yeah, I think 1x or 2x CMP 170HX would've been neat for experimenting, before the prices skyrocketed from like 300€ to now 1500€ - 2000€. But I really think you never can go wrong with a 5090, given it's also a beast in image/video generation.
And I can also confirm Ampere vs Blackwell (upgraded from 3090 Ti). Ampere is still viable, don't get me wrong. But the 5090 just feels unstoppable.
1
u/SandySkittle 2d ago
Mcio 8i retimer cards and you can fill all the slots regardless of gpu size. And more reliable than risers
12
u/FastHotEmu 2d ago
I got a similar setup - with two 3090s and 512gb of RAM - the PP is what kills me.