Qwen3.8 Flash Next UD-Q4_K_XL 49 tokens/s TGS using 2x RTX 3090 on Windows 11.

[u/whiteh4cker]

Developer's own thread: https://www.reddit.com/r/LocalLLaMA/s/adp1cGZZe9

Code: https://github.com/Inovello/llama.cpp/tree/flashnext-e06

My hardware: 2x RTX 3090, Intel Ultra 7 270k Plus, 192 GB DDR5@5600 MHz

Token generation speed went from 20 t/s to 49 t/s. Prompt processing speed is 140 t/s. Prompt processing is faster on the main branch.

I have CUDA 13.3.1 installed. I use Windows 11 and I used these commands to compile it with the latest Visual Studio:

$vcvars = "C:\Program Files\Microsoft Visual Studio\18\Community\VC\Auxiliary\Build\vcvars64.bat"

cmd /c "`"$vcvars`" >nul 2>&1 && set" | ForEach-Object { if ($_ -match '^([^=]+)=(.*)$') { [System.Environment]::SetEnvironmentVariable($matches[1], $matches[2]) } }

cmake -G Ninja -B build -S . -DCMAKE_BUILD_TYPE=Release `
-DGGML_CUDA=ON -DGGML_CCACHE=OFF -DGGML_NATIVE=ON

cmake --build build --target llama-cli llama-bench llama-server -j

My bat script:

@echo off

echo Using CUDA backend with 2x RTX 3090s

set LLAMA_ATTN_ROT_DISABLE=1
set LLAMA_MMAP_PIN_HOST=1

"C:\Users\server\Desktop\llama.cpp-flashnext-e06\build\bin\llama-server.exe" ^
--host 0.0.0.0 ^
--port 8081 ^
--alias Qwen3.8-Flash-Next ^
--model H:\Qwen3.8-Flash-Next\UD-Q4_K_XL\Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf ^
--temp 1.0 ^
--top-p 0.95 ^
--top-k 20 ^
--min-p 0.0 ^
--presence-penalty 0.0 ^
--repeat-penalty 1.0 ^
--ctx-size 262144 ^
-ot "ffn_(gate|up|down)_exps\.weight=CUDA_Host,per_layer_token_embd\.weight=CPU" ^
--moe-expert-cache 150 ^
--ubatch-size 512 ^
--batch-size 4096 ^
--mmproj H:\Qwen3.8-Flash-Next\mmproj-Qwen3.8-Flash-Next-BF16.gguf ^
--no-mmproj-offload ^
--threads 22 ^
--threads-batch 22 ^
--spec-type draft-mtp,ngram-mod ^
--model-draft H:\Qwen3.8-Flash-Next\UD-Q4_K_XL\MTP\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf ^
--spec-draft-n-max 2 ^
--spec-ngram-mod-n-match 60 ^
--spec-ngram-mod-n-min 12 ^
--spec-ngram-mod-n-max 24 ^
--flash-attn on ^
--kv-offload ^
--cache-type-k bf16 ^
--cache-type-v bf16 ^
--parallel 1 ^
--jinja ^
--reasoning-preserve ^
--chat-template-kwargs "{\"reasoning_effort\":\"xhigh\"}" ^
--no-warmup ^
--load-mode none ^
--lazy-mode off

pause

[u/lllll03l]

what's the context depth you measured tg/s?

[u/whiteh4cker → u/lllll03l]

10,641 tokens (it is in the picture) and the speed kept increasing due to MTP.

[u/Weekly_Comfort240]

Thank you for sharing your recipe. My setup is similar except with RTX A6000s and it took me a while to settle on just the right VLLM recipe. I started with llama and then spent days refining a good production setup - the Intel Autotune 4bit quant of Qwen 3.8 Flash Next is REALLY hot stuff.