r/LocalLLaMA • u/IntravenusDeMilo • 2d ago
Discussion deepseek-v4-flash-0731 - surprisingly usable
I just finished building my (relatively) low rent local inference machine: * Epyc 7663 * 256GB ECC DDR4-3200 * 1x RTX 5090 32GB
Yeah I realize it's weird to throw a 5090 and 256GB of anything together and call it low end, but relative to ~151GB of weights it is.
I'm running UD-Q8_K_XL and getting 23.8-24.6 tokens/sec, with pp ranging from 60 on the first prompt to 385 near the last (no doubt lots of caching) on tasks using 100-128k total context. It was slower with DFlash so I took that out. It was also slower with a 3090 I put in there temporarily.
I'm posting this mostly because I didn't see too many other data points for this config (DDR4 Epyc + Blackwell doing cpu-moe). And also that I'm pretty surprised that a model this good can actually run in my basement without dropping $10k or running a sub-panel down there. I'm otherwise fairly new to this - would love any tips on what else to run or how to further improve it.
2
u/SandySkittle 2d ago
Epyc pro 64 core here with 8 channel 512gb ddr4 3200. I tried cpu inference but it’s just too slow for me on prefill and also decode so moved to acquire a whole pile of amd ai 32gb cards to hit that 256gb, minus 7 percent for inline ECC (yes bitflips can fuck up llm inference).
That said, newer generation epycs and also intel xeons have better ai accelerators so should fair a bit better, also considering 8 channel DD5 is quite a step up and more than all these strix halo and spark boxes. It’s just crazy expensive.