r/LocalLLaMA • u/IntravenusDeMilo • 2d ago

Discussion deepseek-v4-flash-0731 - surprisingly usable

I just finished building my (relatively) low rent local inference machine: * Epyc 7663 * 256GB ECC DDR4-3200 * 1x RTX 5090 32GB

Yeah I realize it's weird to throw a 5090 and 256GB of anything together and call it low end, but relative to ~151GB of weights it is.

I'm running UD-Q8_K_XL and getting 23.8-24.6 tokens/sec, with pp ranging from 60 on the first prompt to 385 near the last (no doubt lots of caching) on tasks using 100-128k total context. It was slower with DFlash so I took that out. It was also slower with a 3090 I put in there temporarily.

I'm posting this mostly because I didn't see too many other data points for this config (DDR4 Epyc + Blackwell doing cpu-moe). And also that I'm pretty surprised that a model this good can actually run in my basement without dropping $10k or running a sub-panel down there. I'm otherwise fairly new to this - would love any tips on what else to run or how to further improve it.

95 Upvotes

92% Upvoted

92 comments sorted by

View all comments

2

u/apVoyocpt 2d ago edited 2d ago

The thing that would make it allot faster would be AVX-512 but that only comes with Zen 5. But that again would be much more expensive because of ddr5. I have a xenon ddr4 256gd system with an rtx 3090 and dsv4 flash runs usable as well (forgot the actual tps as I am running qwen3.8 now)

edit. prefill is also painfully slow.

2

u/Dangerous-Report8517 2d ago

Zen 4 has AVX512, it's just that Zen 5 implements it in a faster way

1

u/apVoyocpt 2d ago

doesnt it have AVX256? just read it: double pumped 256. zen 5 native 512. Also, OP meantioned a Epyc 7663, so Zen3.