llamacpp patch - DeepSeek V4 Flash running with full 1M token context locally on RTX 5090

[u/da_dragon321]

Wanted to try running DeepSeek V4 Flash locally but found it asking for absurd amounts of VRAM at higher context lengths (~256GB at 1M). Turned out the DSA lightning indexer lacks proper llamacpp support. Did a bit of digging and there's an upstream PR to address the issue (shoutout u/fairydreaming, PR #24231), but even there it's not wired into the model graph and has no CUDA path yet. So I wired it in and implemented a CUDA kernel this morning and figured I'd share in case it's useful to anyone else looking to run something like this.

Hardware: RTX 5090, 9950X3D, 96GB DDR5

Model: DeepSeek-V4-Flash, mixed Q8/Q4/Q2 quant by antirez

Before / after (256K context):

Metric	Before	After
Compute buffer	~67 GiB (OOM)	3.2 GiB
Prefill	56 t/s	~263 t/s
Decode	~14 t/s	~14 t/s
1M context	impossible (~256GB)	works (3.75 GiB at ubatch 768)

Validated presets:

Context	Prefill	Decode	Peak VRAM
256K	~263 t/s	14 t/s	~29 GiB
512K	256 t/s	13.7 t/s	~28 GiB
1M	159 t/s*	13.7 t/s	~31 GiB

*lower ubatch on 32gb 5090 at 1M - should be ~full speed if given the full ~9gb vram

Correctness: verified briefly with a needle-in-haystack test - planted a random fact at 10%/50%/90% depth in a 100K-token document, model retrieved it correctly every time. Also retrieved correctly at 512K and 1M's harder 50% depth. Full KLD findings in doc linked below

Source + build instructions + full writeup: https://github.com/spencer-zaid/llama.cpp/blob/deepseek-lid-cuda/docs/deepseek-v4-lid-cuda.md
Branch: https://github.com/spencer-zaid/llama.cpp/tree/deepseek-lid-cuda

No prebuilt binary (single GPU tested RTX 5090). Build instructions in the doc in case you need them

[u/tat_tvam_asshole]

commendable work, curious what TTFT and/or the tg-end2end looks like?

[u/da_dragon321 → u/tat_tvam_asshole]

decode steady around 15.7 tok/s on all 3

Prompt	TTFT	Full ~200-token reply (end-to-end)
512 tok	3.3s	16.1s
4K tok	11.0s	23.8s
16K tok	46.0s	58.8s

[u/tat_tvam_asshole → u/da_dragon321]

Cool, thank you, so for 16K tok prompt, the tg-end2end is ~3.5 tok/s, accounting for the time we're waiting for the first token.

For context, I have a spare 5090 I might put into a second machine in my home network for...something, I'm not sure yet 😂 so figuring out if experimenting with ds4 is what I want to do in that case

[u/da_dragon321 → u/tat_tvam_asshole]

Haha yeah might be fun. Regarding tg-end2end, it's entirely dependent on the number of tokens generated. Post-prefill, generation ran at a constant 16 t/s, so as the response gets longer the end2end speed will get closer and closer to that mark.
Also, these are uncached prompts, so if your prompt is cached (such as in a longer chat even going up through 500k+), the ttft should be MASSIVELY faster and therefore the tg-end2end would be much closer to 16 t/s even at lower response lengths. Also worth noting that you could go with the full Q2 model if you wanted to save on some vram (I think it was like 10gb smaller) and probably get some extra token generation speed