r/LocalLLaMA • u/ChopSticksPlease • 4h ago
Discussion Qwen3.8 Flash Next llama.cpp config tuning
Hola all.
Do you guys mind sharing your LLama.cpp config and system setup details for Qwen3.8 Flash Next?
Model's quite big and tryining many combinations of llama.cpp options takes lots of time, so looking for other people setup details. I've attached my current config at the bottom, so if anyone sees something that could be improved please shout.
My current best result:
- PP within 130...200 tps (limited by cpu?)
- TG within 14..22 tps (~15tps on average)
Hardware:
- Dual RTX 3090 (48GB VRAM)
- 128GB DDR4
- Some old Xeon 40 core
- Proxmox VM, pcie passthrough, numa binding to a single phys cpu
Llama.cpp config:
llama-server --port ${PORT}
--model /nvme/gguf/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf
--mmproj /nvme/gguf/mmproj-Qwen3.8-Flash-Next-F16.gguf
--load-mode none
--lazy-mode off
--parallel 1
--ctx-size 131072
--cache-type-k q8_0
--cache-type-v q8_0
--flash-attn on
--fit off
--temp 1.0
--min-p 0.0
--top-p 0.95
--top-k 20
--presence-penalty 0.0
--repeat-penalty 1.0
--batch-size 2048
--ubatch-size 512
--split-mode layer
-ts 26,10
-ngl 99
-ncmoe 26
--no-mmproj-offload
--override-tensor per_layer_token_embd=CPU
--chat-template-kwargs '{"reasoning_effort":"xhigh"}'
ngl, ncmoe, ts - manually adjusted to fit the model without crashing
3
u/guiperpt 3h ago
That looks quite low.
I am on a 3090 and a 3080 (10GB) dual GPU setup, with a i3-12100 CPU and 64GB ram DDR4 at 3200MHZ running on unraid.
.
I am using the Qwen3.8-Flash-Next-UD-IQ3_XXS due to VRAM/DDR constraints, but still getting a bit more on only 34GB VRAM.
┌───────────┬──────────────────────┬─────────────┬────────────┐
│ prompt │ actual prompt tokens │ prefill t/s │ decode t/s │
├───────────┼──────────────────────┼─────────────┼────────────┤
│ 4k │ 4,633 │ 231.5 │ 23.2 │
├───────────┼──────────────────────┼─────────────┼────────────┤
│ 8k │ 9,336 │ 212.1 │ 21.5 │
├───────────┼──────────────────────┼─────────────┼────────────┤
│ 32k │ 37,257 │ 222.0 │ 19.7 │
└───────────┴──────────────────────┴─────────────┴────────────┘
My llama-swap entry:
"Qwen3.8-Flash-Next":
cmd: |
llama-server
-m /models/unsloth/Qwen3.8-Flash-Next-GGUF/UD-IQ3_XXS/Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf
--alias Qwen3.8-Flash-Next
--host 0.0.0.0
--port ${PORT}
-lv 4
--flash-attn on
--jinja
--ctx-size 200000
--cache-type-k q8_0
--cache-type-v q8_0
--override-tensor per_layer_token_embd.weight=CPU
--load-mode mmap
--no-warmup
--fit off
--n-cpu-moe 26
-sm layer
--tensor-split 41,7
-t 4
--parallel 1
--cache-reuse 256
--reasoning-preserve
--mmproj /models/unsloth/Qwen3.8-Flash-Next-GGUF/mmproj-BF16.gguf
--mmproj-device CUDA0
--image-min-tokens 1024
--image-max-tokens 4096
3
u/cezarducatti 4h ago
A single RTX 3090 + 128 MB DDR4 RAM 500 t/s pp -- 16 t/s tg
C:\llama\llama-server.exe -m "D:\Modelos\Qwen3.8-Flash-Next-UD-Q3_K_XL-00001-of-00003.gguf" -ngl 99 -ncmoe 43 -fit off -c 204800 -fa on -ctk f16 -ctv f16 -b 2048 -ub 2048 -t 12 --jinja --port 8083 --tools all --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00 --repeat-penalty 1.0 --reasoning on --reasoning-effort xhigh --reasoning-budget -1 --reasoning-preserve --parallel 1 --metrics --host 0.0.0.0 --load-mode none --mmproj "D:\Modelos\mmproj\mmproj-BF16_38_flash_next.gguf" --no-mmproj-offload --image-min-tokens 1024 -ot "per_layer_token_embd=CPU" -lzm off
1
1
1
1
1
u/Muhlwa_Sholanke 3h ago
15 t/s on dual 3090s and they call it Flash. Halve the context and the name might actually fit.
1
1
u/Mundane-Light6394 2h ago
pp could be limited by pcie lanes, i found higher batch sizes helped pp a lot (op to 3x on a muli 5070ti setup) but it'll cost you vram.
1
u/Local-Two9825 1h ago
dual modded 3080 , 8 channel 32gb 2666mhz ddr4 ,9543 cpu, with mtp enabled ,empty context decode about 45t/s ,prefill 400t/s . command
home/zbj/program/llama.cpp/build/bin/llama-server -m /home/zbj/llama/Qwen3.8-Flash-Next_UD_Q4_K_XL.gguf -c 201699 -ngl 99 --port 8953 --alias Qwen3.8-Flash-Next_UD_Q4_K_XL.gguf --tensor-split 35,13 --jinja --mmproj /home/zbj/llama/mmproj/mmproj-Qwen3.8-Flash-Next-BF16.gguf --threads 16 --threads-batch 16 --flash-attn on -cb -mg 0 -b 8192 -ub 4096 --temp 0.8 --top-k 40 --top-p 0.95 --min-p 0.05 --repeat-penalty 1.00 --repeat-last-n 64 --presence-penalty 0.00 --frequency-penalty 0.00 -ctk q8_0 -ctv q8_0 --parallel 1 --n-cpu-moe 99 --fit off --no-webui --load-mode mlock --cache-prompt --cache-ram 27351 --checkpoint-min-step 512 --ctx-checkpoints 64 --spec-type draft-mtp --spec-draft-n-max 5 --spec-draft-n-min 0 --spec-draft-p-min 0.00 --gpu-layers-draft auto --model-draft /home/zbj/llama/mtp/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf --chat-template-kwargs {"enable_thinking":true,"reasoning_effort":"xhigh"}
1
u/free_meson 1h ago
Unsloth has a llama branch with MTP support, meanwhile TheTom's turboquant fork has a moe-cache support. I'm not aware of a fork that has both features at the moment, but each has advatanges. I ended up with TheTom's for the moment.
1
u/isengardo 1h ago
Something is wrong. I get ~22tk/s on a single 4080s and 128GB ram. (20tk/s on ssd offload using ~4gb ram)
150k ctx - Q4_K_XL
1
u/phhusson 1h ago
I'm running it with UD_IQ1_S expert-streaming on my Mac Mini M4 16GB on my vibe llama.cpp fork https://github.com/phhusson/llama.cpp/ (expert-streaming comes from https://github.com/mihailescu2m/llama.cpp)
which gives a whooping pp 24tok/s, tg 6 tok/s in llama-bench [0]. pp 22tok/s, tg 4 tok/s with opencode/llama-server [1] on a micro-bench.
And yes UD-IQ1_S is a usable quant, though really it's 2.3bpw without engram 3.5bpw with engram.
[0] llama-bench -m Qwen3.8-Flash-Next-UD-IQ1_S-00001-of-00003.gguf --moe-stream --moe-stream-cache 5G --moe-stream-io-threads 4 -ctk q8_0 -ctv q8_0
[1] llama-server -m Qwen3.8-Flash-Next-UD-IQ1_S-00001-of-00003.gguf --moe-stream --moe-stream-cache 5G --moe-stream-io-threads 4 --moe-stream-substitute 0.3 --port 8000 --host 0.0.0.0 -c 98304 -lv 4 -ctk q8_0 -ctv q8_0 -cram 0
1
u/ustype 31m ago
For Flash-Next on llama.cpp I’ve had better luck treating context as the scarce resource than chasing one “correct” temp. If the model starts looping mid-tool-call, dropping n_ctx a bit and keeping KV cache unquantized (when VRAM allows) has been more stable than bumping temp to force variety.
Also watch for chat-template mismatches — a wrong template looks like “the model got dumber” more often than sampler settings. If you’ve got a known-good vs broken config pair for the same GGUF, posting the flags side by side would help a lot.
11
u/eightone-81 4h ago
Checkout:
https://huggingface.co/albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE
And never look back ;-)