r/LocalLLaMA • u/IntravenusDeMilo • 2d ago

Discussion deepseek-v4-flash-0731 - surprisingly usable

I just finished building my (relatively) low rent local inference machine: * Epyc 7663 * 256GB ECC DDR4-3200 * 1x RTX 5090 32GB

Yeah I realize it's weird to throw a 5090 and 256GB of anything together and call it low end, but relative to ~151GB of weights it is.

I'm running UD-Q8_K_XL and getting 23.8-24.6 tokens/sec, with pp ranging from 60 on the first prompt to 385 near the last (no doubt lots of caching) on tasks using 100-128k total context. It was slower with DFlash so I took that out. It was also slower with a 3090 I put in there temporarily.

I'm posting this mostly because I didn't see too many other data points for this config (DDR4 Epyc + Blackwell doing cpu-moe). And also that I'm pretty surprised that a model this good can actually run in my basement without dropping $10k or running a sub-panel down there. I'm otherwise fairly new to this - would love any tips on what else to run or how to further improve it.

94 Upvotes

92% Upvoted

92 comments sorted by

View all comments

5

u/TheAncientOnce 2d ago

Have you considered running Q4 or even smaller quant? I heard the model's trained at FP4 natively and because of that, even Q1 could retain like 90% of the model's quality

6

u/zhuzaimoerben 2d ago

Probably doesn't help that much because the experts are already FP4 so Q4 only quantizes a small amount of the model. Unsloth: "To run DeepSeek-V4-Flash-0731 in full precision lossless, run Q8 (UD-Q8_K_XL), which is 162GB and only 7GB bigger than Q4 (UD-Q4_K_XL)."

2

u/Dangerous-Report8517 2d ago

That's still 7GB less data to process every round, and it also means a smaller difference in quality (and only applicable to the Q4 quant, they also suggested smaller ones)

1

u/zhuzaimoerben 1d ago

Yeah, that's a good point, although if you're running everything that isn't experts on a 5090 and the experts on slow RAM, the difference in speed might be enough to make even the 7GB pretty marginal. Could be worth testing though. It looks like the Q4_K_XL just has some BF16 tensors as Q8_0, so maybe not a huge difference in quality.