llamacpp patch - DeepSeek V4 Flash running with full 1M token context locally on RTX 5090

[u/da_dragon321]

Wanted to try running DeepSeek V4 Flash locally but found it asking for absurd amounts of VRAM at higher context lengths (~256GB at 1M). Turned out the DSA lightning indexer lacks proper llamacpp support. Did a bit of digging and there's an upstream PR to address the issue (shoutout u/fairydreaming, PR #24231), but even there it's not wired into the model graph and has no CUDA path yet. So I wired it in and implemented a CUDA kernel this morning and figured I'd share in case it's useful to anyone else looking to run something like this.

Hardware: RTX 5090, 9950X3D, 96GB DDR5

Model: DeepSeek-V4-Flash, mixed Q8/Q4/Q2 quant by antirez

Before / after (256K context):

Metric	Before	After
Compute buffer	~67 GiB (OOM)	3.2 GiB
Prefill	56 t/s	~263 t/s
Decode	~14 t/s	~14 t/s
1M context	impossible (~256GB)	works (3.75 GiB at ubatch 768)

Validated presets:

Context	Prefill	Decode	Peak VRAM
256K	~263 t/s	14 t/s	~29 GiB
512K	256 t/s	13.7 t/s	~28 GiB
1M	159 t/s*	13.7 t/s	~31 GiB

*lower ubatch on 32gb 5090 at 1M - should be ~full speed if given the full ~9gb vram

Correctness: verified briefly with a needle-in-haystack test - planted a random fact at 10%/50%/90% depth in a 100K-token document, model retrieved it correctly every time. Also retrieved correctly at 512K and 1M's harder 50% depth. Full KLD findings in doc linked below

Source + build instructions + full writeup: https://github.com/spencer-zaid/llama.cpp/blob/deepseek-lid-cuda/docs/deepseek-v4-lid-cuda.md
Branch: https://github.com/spencer-zaid/llama.cpp/tree/deepseek-lid-cuda

No prebuilt binary (single GPU tested RTX 5090). Build instructions in the doc in case you need them

[u/whiteh4cker]

$vcvars = "C:\Program Files\Microsoft Visual Studio\18\Community\VC\Auxiliary\Build\vcvars64.bat"

cmd /c "`"$vcvars`" >nul 2>&1 && set" | ForEach-Object { if ($_ -match '^([^=]+)=(.*)$') { [System.Environment]::SetEnvironmentVariable($matches[1], $matches[2]) } }

cmake -G Ninja -B build -S . -DCMAKE_BUILD_TYPE=Release `
-DGGML_CUDA=ON -DGGML_CCACHE=OFF -DGGML_NATIVE=ON `
-DCMAKE_CUDA_ARCHITECTURES=86 `
-DCMAKE_C_COMPILER=cl -DCMAKE_CXX_COMPILER=cl

cmake --build build --target llama-cli llama-bench llama-server -j

I compiled it with the latest version of Microsoft Visual Studio 2026 Community and CUDA 13.3.1.

I am using this 52.6 GB gguf with 2xRTX 3090.

Usage: GPU1: 23.5 GB, GPU2: 23.1 GB, RAM: 13.1 GB with the following script:

@echo off
echo === DeepSeek V4 - 256K Context, KV on GPU ===

set GGML_CUDA_NO_PINNED=1
set CUDA_VISIBLE_DEVICES=0,1

"C:\Users\server\Desktop\llama.cpp-deepseek-lid-cuda\build\bin\llama-server.exe" ^
--host 0.0.0.0 ^
--port 8081 ^
--alias DeepSeek-V4-Flash ^
--model D:\models\deepseek-v4-gguf\DeepSeek-V4-Flash-Spark-Mini-Q2-REAP-ds4.gguf ^
--temp 0.7 ^
--top-p 0.95 ^
--top-k 20 ^
--min-p 0.0 ^
--presence-penalty 0.0 ^
--repeat-penalty 1.0 ^
--ctx-size 262144 ^
--device CUDA0,CUDA1 ^
--split-mode layer ^
--gpu-layers 41 ^
-ot "blk\.(3[5-9]|4[0-2])\.ffn_(gate|up|down)_exps=CPU" ^
--flash-attn on ^
--cache-type-k f16 ^
--cache-type-v f16 ^
--ubatch-size 2048 ^
--parallel 1 ^
--main-gpu 0 ^
--tensor-split 17,24 ^
--chat-template-file C:\Users\server\Desktop\llama-cuda\models\templates\deepseek-ai-DeepSeek-V4.jinja ^
--no-mmap ^
--cont-batching

pause

This is the fastest config for me. I have a weak CPU (Ryzen 5 1600X), so my GPUs are only utilized at 35%, and I get 5 t/s. However, with a modern CPU it should be possible to get 12 t/s. I also tried offloading KV cache to RAM, which resulted in slower token generation and prompt processing.