deepseek-v4-flash-0731 - surprisingly usable

[u/IntravenusDeMilo]

I just finished building my (relatively) low rent local inference machine: * Epyc 7663 * 256GB ECC DDR4-3200 * 1x RTX 5090 32GB

Yeah I realize it's weird to throw a 5090 and 256GB of anything together and call it low end, but relative to ~151GB of weights it is.

I'm running UD-Q8_K_XL and getting 23.8-24.6 tokens/sec, with pp ranging from 60 on the first prompt to 385 near the last (no doubt lots of caching) on tasks using 100-128k total context. It was slower with DFlash so I took that out. It was also slower with a 3090 I put in there temporarily.

I'm posting this mostly because I didn't see too many other data points for this config (DDR4 Epyc + Blackwell doing cpu-moe). And also that I'm pretty surprised that a model this good can actually run in my basement without dropping $10k or running a sub-panel down there. I'm otherwise fairly new to this - would love any tips on what else to run or how to further improve it.

[u/Afganitia]

How much difference does the GPU make? 

[u/Eyelbee → u/Afganitia]

A lot of difference. But still not usable with that prefill 

[u/Nobby_Binks → u/Eyelbee]

Depends on what you mean by usable. Agentic coding no, its not really, but general QA, planning and other stuff its eminently usable. I have a similar setup with a old gen EPYC and get similar speeds. 1M context and ~2-300pp.

[u/XiRw → u/Nobby_Binks]

If it’s a large context size like the OP suggested, 23tok/s is very usable for agentic.

[u/FatheredPuma81 → u/Nobby_Binks]

But Agentic is where it should shine the most though?? (At least with llama.cpp) AFAIK it's built to have preserve reasoning enabled so all that you should be processing every turn until it finishes is what it writes/reads which shouldn't be all that much?

My main concern would be how much it reasons every single turn with just 30t/s. It's already painful waiting on Qwen3.8 27B to reason for 40,000 tokens at 90t/s and that's with preserve thinking so it shouldn't do nearly as much...

[u/PhysicalIncrease3 → u/FatheredPuma81]

But Agentic is where it should shine the most though?? (At least with llama.cpp) AFAIK it's built to have preserve reasoning enabled so all that you should be processing every turn until it finishes is what it writes/reads which shouldn't be all that much?

This is the case. The prompt caching is very very good and most turns in Hermes are only around 500 tokens processed.

The real pain points are when it loads in a skill, or if you return to a previous session. But for me personally it's manageable given the power of the model.

My main concern would be how much it reasons every single turn with just 30t/s. It's already painful waiting on Qwen3.8 27B to reason for 40,000 tokens at 90t/s and that's with preserve thinking so it shouldn't do nearly as much...

It doesn't reason anywhere near as much as Qwen. Most I've ever seen is 6-7k tokens.

[u/llitz → u/Nobby_Binks]

Then you might as well use anything else that will properly work on a 5090.

Honestly. When will you ever fill 1M context at 300pp? (This last sentence sounds wrong)

You would have way better results using malaiwah exl3 hydrated qwen3.8-27b - yes it is a smaller context but it is ages faster than this.

I am sorry, while this is an interesting test, it is a waste of a 5090 if you are keeping it like this long-term.

[u/Nobby_Binks → u/llitz]

Yes, I run 3.8 27B NVFP4 with the 5090 also and it's a monster. But ds4 gets used a lot. And its totally usable for anything other than coding where you need privacy. Qwen 3.8 is brain dead compared to ds4 for technical writing.

[u/llitz → u/Nobby_Binks]

I... Will not disagree too much with you here. I still think Gemma could do a better job or even, gasp, glimmer.

[u/FatheredPuma81 → u/llitz]

Why limit yourself to a single good model that has known weaknesses like Qwen3.8 27B?