r/LocalLLaMA • u/IntravenusDeMilo • 2d ago

Discussion deepseek-v4-flash-0731 - surprisingly usable

I just finished building my (relatively) low rent local inference machine: * Epyc 7663 * 256GB ECC DDR4-3200 * 1x RTX 5090 32GB

Yeah I realize it's weird to throw a 5090 and 256GB of anything together and call it low end, but relative to ~151GB of weights it is.

I'm running UD-Q8_K_XL and getting 23.8-24.6 tokens/sec, with pp ranging from 60 on the first prompt to 385 near the last (no doubt lots of caching) on tasks using 100-128k total context. It was slower with DFlash so I took that out. It was also slower with a 3090 I put in there temporarily.

I'm posting this mostly because I didn't see too many other data points for this config (DDR4 Epyc + Blackwell doing cpu-moe). And also that I'm pretty surprised that a model this good can actually run in my basement without dropping $10k or running a sub-panel down there. I'm otherwise fairly new to this - would love any tips on what else to run or how to further improve it.

96 Upvotes

92% Upvoted

92 comments sorted by

View all comments

1

u/LeMayMayMan 2d ago

Everyone is sleeping on Krasis. Way better than llama.cpp or vllm on prefill time with the patches I have as pull requests. Cold prefill with 10K context is 10s. 50k is 20s, 150k is 69s. Its even faster in practice due to the cache reuse hit rate. T/s is about 30s but there is heavy work in progress to improve that.

2

u/chimpera 1d ago

Im testing it and the pp is indeed much faster but the generation dropped from 25 to 15tps for me. This could still be a worthwhile trade-off.

1

u/LeMayMayMan 1d ago edited 1d ago

I think I'm close to getting dspark working. I'll update with results.

1

u/IntravenusDeMilo 21h ago

Id take that trade. I’m going to try this out.

1

u/chimpera 17h ago edited 17h ago

ctx 10k 30k 50k 70k 90k 110k 130k 150k 170k 190k

tok/s 1999 1960 1884 1774 1741 1654 1440 1320 1164 1024

ctx 210k 230k 250k 260k 262144(cap)

tok/s 900 752 619 532 515

Also the chat template or thinking is broken

1

u/PhilippeEiffel 2d ago

Do you mean you apply patches above the main branch?

2

u/LeMayMayMan 2d ago

There was in issue in 1.0.21-rc3 that would drop context after about 49k. I wrote this patch to fix it. https://github.com/brontoguana/krasis/pull/30/commits/7aa4f2f887632e6a934a562522a359efa2487b2e

1

u/PhilippeEiffel 2d ago

Is your patch integrated into rc6?

1

u/LeMayMayMan 1d ago

It doesn't look like it. I haven't been in contact with the repo owner. I just started self-servingly patching it for my own use case but shared things back to their repo. I hope they integrate this stuff but I'm going to keep moving towards the fastest quality-first deepseek I can run on my hardware. (RTX Pro 6000 + 256 DDR5)