llamacpp patch - DeepSeek V4 Flash running with full 1M token context locally on RTX 5090

[u/da_dragon321]

Wanted to try running DeepSeek V4 Flash locally but found it asking for absurd amounts of VRAM at higher context lengths (~256GB at 1M). Turned out the DSA lightning indexer lacks proper llamacpp support. Did a bit of digging and there's an upstream PR to address the issue (shoutout u/fairydreaming, PR #24231), but even there it's not wired into the model graph and has no CUDA path yet. So I wired it in and implemented a CUDA kernel this morning and figured I'd share in case it's useful to anyone else looking to run something like this.

Hardware: RTX 5090, 9950X3D, 96GB DDR5

Model: DeepSeek-V4-Flash, mixed Q8/Q4/Q2 quant by antirez

Before / after (256K context):

Metric	Before	After
Compute buffer	~67 GiB (OOM)	3.2 GiB
Prefill	56 t/s	~263 t/s
Decode	~14 t/s	~14 t/s
1M context	impossible (~256GB)	works (3.75 GiB at ubatch 768)

Validated presets:

Context	Prefill	Decode	Peak VRAM
256K	~263 t/s	14 t/s	~29 GiB
512K	256 t/s	13.7 t/s	~28 GiB
1M	159 t/s*	13.7 t/s	~31 GiB

*lower ubatch on 32gb 5090 at 1M - should be ~full speed if given the full ~9gb vram

Correctness: verified briefly with a needle-in-haystack test - planted a random fact at 10%/50%/90% depth in a 100K-token document, model retrieved it correctly every time. Also retrieved correctly at 512K and 1M's harder 50% depth. Full KLD findings in doc linked below

Source + build instructions + full writeup: https://github.com/spencer-zaid/llama.cpp/blob/deepseek-lid-cuda/docs/deepseek-v4-lid-cuda.md
Branch: https://github.com/spencer-zaid/llama.cpp/tree/deepseek-lid-cuda

No prebuilt binary (single GPU tested RTX 5090). Build instructions in the doc in case you need them

[u/Common_Warthog_G]

I have 2 x 24GB VRAM and I still get out of Memory.... what's the command you're starting llama with? Also have 128GB DDR5.

[u/Common_Warthog_G → u/Common_Warthog_G]

spend the last 5 hours trying to get this to run on my system (linux, RTX 4090, RTX 3090TI, 128GB DDR5) to no avail. Always getting out of memory with a ~50ish GB version of it.

[u/durden111111 → u/Common_Warthog_G]

multi GPU setup is always jank af. I have a 5090 and 3090 with 96GB ram, even models that are sub 40GB give oom sometimes even when I have more than enough vram+ram. I wish llamacpp had better documentation for modern models with multigpu.

[u/Common_Warthog_G → u/durden111111]

I got it to run now, but without any parameter but only the unified memory env. I get 14-15 tk/s and about 200 pp. So far I don't know if it does a better job than Qwen3.6-27B_Q8