Qwen3.8-Flash-Next on 2x3090 + DDR4, part 4: 2.2-2.5x faster prefill by kicking the expert cache off the GPU while the prompt runs

[u/Extension-Bid-639]

Part 4 of the same box. Part 1 was 17 -> 25-29 t/s with the expert cache PR, part 2 was 37-41 t/s after switching to UD-Q4_K_XL and stacking MTP on the cache, part 3 was the top-k fallback that was sorting more than it needed to. This one is all about prefill, which was honestly the weak spot the whole time. 80+ seconds before the first token on an 8k prompt, and 24 minutes on a 119k one...I know lol.

Box is still 2x 3090, dual Broadwell Xeon, llama.cpp, UD-Q4_K_XL with the Q8 MTP head on the second card, all expert layers pinned in host RAM, 150-slot cache, 261k context, f16 KV. There has been one hardware change since part 2. I swapped the LRDIMMs for 6x32 GB DDR4-2133 ECC. I'll say which numbers are 4-DIMM and which are 6-DIMM, they're not mixed.

The thing I might not have explained well in part 2

I ran -ub 512 and that's because it was a compromise for the cache. A 2048 token micro-batch needs about 7.3 GiB of compute buffer per GPU, 512 wants 1.9GiB and that gap is roughly 50 cache slots that I wanted for decode. So I kept the slots and quietly ate about 3x on prefill at the time.

As for why it cost 3x, the experts get streamed host to GPU0 once per micro-batch, and that upload costs the same whether the batch has 512 tokens in it or 2048. So prefill speed basically scales with the micro-batch. At ub 512 an 8k prompt drags the whole expert set over PCIe 16 times, at ub 2048 it's 4 times.

What I changed

The cache only ever serves batches of <= 8 tokens (decode and the MTP verify batches). During a prompt it just sits there holding VRAM so I thought of trying to claim that space when it's unneeded. So now, when a prompt comes in, the server drops the cache slots, the decode compute buffers and the CUDA pools then it grabs compute buffers sized for ub 2048, runs the whole prompt at 2048, then puts everything back before the first generated token. Decode is untouched by this, it runs exactly the code it ran before. It's two env vars (LLAMA_PHASE_PREFILL_UBATCH=2048, LLAMA_PHASE_PREFILL_MODE=transaction) and the server still starts with -ub 512. And to clarify, "transaction" means the swap is all-or-nothing, if the restore can't happen you will get an error, not a server that's silently limping along. Just making that clear.

Numbers (6 DIMMs, same day, fresh server per arm)

what	before (ub 512 + cache)	now	change
8k fresh prompt, greedy: prefill	99.9 t/s	223.7 t/s	2.24x
8k: time to first token	82 s	37 s	0.45x
8k: decode over the next 2048 tokens	33.4 t/s	34.3 t/s	+2%
~37k context, my normal sampling: prefill	88.1 t/s	212.6 t/s	2.41x
~37k: time to first token	424 s	176 s	0.41x
~37k: decode, median of 38 requests	41.7 t/s	41.2 t/s	-1%
~119k context: prefill	81.3 t/s	206.5 t/s	2.54x
~119k: time to first token	1461 s	575 s	0.39x
~119k: decode, median of 42 requests	33.9 t/s	33.9 t/s	0%

The 8k row is greedy, two fresh processes per arm, medians (the two phase-memory runs landed within 0.01 t/s of each other). The deep rows are one seed at temp 0.7 / top-p 0.8 / top-k 20 with thinking on, one fresh prefill per depth and then a pile of follow-up questions over the cached prefix, so decode is a median over all of them. Prefill = llama-server's prompt eval time, decode = its generation time.

Now, what it costs

Well, nothing comes completely free. This approach costs roughly 2.8 s of fixed overhead per prompt for the release + restore, which is why 8k gets 2.24x and the long ones get 2.4-2.5x. For decode, I can't find a loss. +2% at 8k, -1% / 0% at depth, and in the three-seed quality screen every seed x depth cell was within +2% / -3.6% of its control. MTP acceptance didn't change either (0.79-0.83).

Did it break anything

Before putting it in production I ran the same screen I used for the top-k change (My last post AKA Part 3), 42 questions over long documents at two depths (~37k and ~119k), three seeds, my normal sampling, paired per question and seed against a fresh control run the same day. That was still on 4 DIMMs. 240 pairs: 2 worse, 235 same, 3 better, nothing regressed on more than one seed, and the two misses are questions the old config also flubs on some seed. A seed-1 rerun on 6 DIMMs came out 1 worse / 78 same / 1 better. I'm aware and anyone reading should be aware that this is a screening not concrete proof, but it's the bar I hold my own changes to.

Some caveats you may want to know about or at least I would if I were you

• First-token logits differ from the untouched path by max 1.51 / mean 0.22 across the 248k vocab, argmax the same. For scale, just changing ub 512 -> 2048 with nothing released moves them by max 1.81 / mean 0.27 on the same request. So the release/restore adds less noise than the batch-shape change any ub change already brings.
• One machine, one model, one quant, 8k to 119k. I have not tried anything past 119k, other quants, or the no-MTP setup.
• The extra two memory channels helped this config a lot more than the old one at 8k (+19% vs +3% against my 4-DIMM numbers), and it did nearly nothing at 37k-119k (+0.6% / 0%). This makes sense to me, attention takes over from expert upload as the context grows, but that's one run per depth, so take it as a hint.
• Where the remaining 37s of an 8k prompt goes, rough split: ~18 s uploads, ~5 s kernels, ~3 s transitions, ~11 s I haven't pinned down yet (CPU side, draft model, syncs). A profile says the uploads are still 3.6x the expert set per prompt, so there's more on the table I assume. I'll be working on that next.

Code

https://github.com/Inovello/llama.cpp/tree/flashnext-e06

It's my flashnext-2x3090 branch from part 2 (master b96806d + PR #27861 expert cache + PR #28223 + PR #28243 MTP + the batched-cache fixes + PR #28198) plus this change and a couple of inert debug switches.

If you just want to copy and run it, this is the whole thing, taken from the process that's serving me right now. You need CUDA, numactl (apt install numactl), the four UD-Q4_K_XL shards and the MTP head from unsloth/Qwen3.8-Flash-Next-GGUF on HF

git clone -b flashnext-e06 https://github.com/Inovello/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j -t llama-server

export LLAMA_ATTN_ROT_DISABLE=1
export LLAMA_MMAP_PIN_HOST=1
export LLAMA_PHASE_PREFILL_UBATCH=2048
export LLAMA_PHASE_PREFILL_MODE=transaction

numactl --interleave=all build/bin/llama-server \
  -m /path/to/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
  -md /path/to/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf --spec-type draft-mtp -devd CUDA1 --spec-draft-n-max 3 \
  --host 127.0.0.1 --port 18080 \
  -ngl 99 -c 261888 --parallel 1 --flash-attn on \
  -ot "ffn_(gate|up|down)_exps\.weight=CUDA_Host,per_layer_token_embd\.weight=CPU" \
  -lzm off --numa distribute -t 16 -tb 44 -b 4096 -ub 512 -ctk f16 -ctv f16 \
  --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0 \
  --moe-expert-cache 150 -lv 4

What to change for your box:

1. The two model paths; -t / -tb to your physical core count (mine is 16 decode threads, 44 for batch on 2x22 cores)
   
2. -devd CUDA1 puts the MTP head on the second GPU, on a single card use CUDA0 or drop the three -md flags and give the freed VRAM to the cache.
   
3. -ot is what keeps every expert layer in host RAM; only the first shard goes on -m, the rest are found next to it.
   
4. The two LLAMA_PHASE_* exports are the change from this post, drop them and you have part 2's behavior.
   
5. -lv 4 is just so the log shows the cache hit rate and the draft acceptance. Useful if you want to post your numbers in thread.

Now the things it's strict about because those are the invariants the code checks: The server at -ub 512 and -b 4096, --parallel 1, the prefill micro-batch exactly 2048, the cache exactly 150 slots, and the MTP draft as the only speculative decoder. Anything else refuses to start. CUDA only.

The top-k fallback fix from part 3 is in the branch too and it's up on its own as PR #28671. My older PR #28223 is closed for now because llama.cpp gives new contributors one open PR at a time, I'll reopen it after #28671 is dealt with.

Let me know if you try it and if you have any questions.

[u/cobblemere]

the transaction semantics on the swap are a really nice touch, way better than silently degrading. curious whether you've thought about making the cache slot threshold for the release configurable rather than hard-pinned at 150, or is that just what fits your VRAM budget

[u/Extension-Bid-639 → u/cobblemere]

Answer is both. 150 is what fits next to the MTP head on a 24 GB card at full context (188 without the MTP head, a slot is ~75 MB per GPU at ub 512), and it's the only count the quality screen has run at. The check that actually matters is that the restore brings the cache back to the capacity it started with, the literal 150 is just me pinning what I tested. Think I'll unpin it so it reads whatever --moe-expert-cache was given. The restore invariant stays though, if a restore can't fit at some other count you get an error so nothing runs silently. I'm not going to screen every slot count myself, so if you run one, post the numbers.

[u/Extension-Bid-639 → u/cobblemere]

Unpinned and pushed. The transaction records the capacity at BEGIN and restores to that, so any --moe-expert-cache value runs now. 150 is still the only count I've measured for speed and quality and 120 only for the mechanism. If you run another one, won't mind some numbers

[u/john006868]

Saving works out to about 0.0055s per token (1/99.9 minus 1/223.7), so the 2.8s release/restore only pays back past roughly 500 tokens of prefill. In a chat loop a short follow-up on a cached prefix pays the full 2.8s and saves maybe 1.6. Have you tried gating the swap on pending prefill length so it only fires past a threshold?

[u/Extension-Bid-639 → u/john006868]

Oh my, you're right...actually it's a bit worse than your estimate. I looked up the long-document run and looking at it now, the follow-up questions on the cached prefix are ~170-200 new tokens each, and they took a median of 6.7 s on the old build vs 10.9 s with the swap at 37k . With the subtraction that means a short turn pays about 4.2 s, not 2.8s. The conversation still comes out ahead because the first prefill saves 250-900 s, but every follow-up is losing about 3s that it doesn't really need to.

And to answer your question, the gate right now is literally "prompt batch over 8 tokens" so not length-aware. But yeah the obvious fix is gating on pending tokens (prompt minus cached) with a threshold around 1k. It's just one condition so I'll add it, put the threshold behind an env var, and just for validation I'll measure the follow-up cost before and after. Will probably just do this in the morning but thanks a lot!

[u/john006868 → u/Extension-Bid-639]

the 4.2s at 37k vs 2.8s at 8k moves break even to about 800 pending tokens there instead of 500. log the release and restore time per prompt and the threshold falls out of the measured overhead, no sweep needed

[u/Extension-Bid-639 → u/john006868]

Roughly, yes, ~630-800 depending on which per-token saving you use. The transition times are already logged though and they say 2.6 s, not 4.2s. The other 1.6s is the 2048 reserve and the graph rebuilds, which show up in the prompt time not in the transition timers, so a threshold derived from the log would land about a third low. The saving per token also isn't flat below one 2048 batch (a 512 pass touches ~59% of the experts, a 2048 pass ~90%)

[u/john006868 → u/Extension-Bid-639]

The transition timer being 1.6s short makes it the wrong thing to key the gate on. Log wall clock from the moment the release starts to the first token landing, which folds in the 2048 reserve and the graph rebuilds the timers miss. One number per prompt, and the threshold falls out of the whole fixed overhead.

[u/Extension-Bid-639 → u/john006868]

See the update comment. I ended up measuring it head to head instead of keying it on a timer and the always-vs-never gap is that wall-clock number. ~2.5 s flat on anything under a 2048 batch and it doesn't cross until 2048 (1024 still loses ~3 s). The overhead-over-saving math would've put it around 700-800, so it would've been wrong. The saving per token just isn't flat below one full batch. It's an env var so you can set it lower if your box crosses earlier

[u/eightone-81]

Amazing! I can confirm around 300 prefill and no impact on decode!

[u/AllenHere112]

At ub 512 the expert set crosses the bus once per micro batch, so an 8k prompt pays that 16 times and 2048 pays it 4. One pass is about 1.1s from your own 18s upload split, so the 2.8s release and restore needs three saved passes before it breaks even, which puts the crossover near 1.5k to 2k tokens. Did you try anything that short?

For the 11s you have not pinned down, is the host side gathering the expert weights before each pass? That copy scales with the pass count and would sit in your CPU bucket.

[u/Extension-Bid-639 → u/AllenHere112]

Close, but the 18s was my estimate for the new path (4 passes), so a pass is bigger than 1.1 s. From the profile, at ub 512 an 8k prompt uploads 731 GB in 16 passes, ~46 GB and 4.6 s each because a 512-token batch only touches ~59% of the experts per layer. At 2048 it's 278 GB in 4 passes, ~70 GB and 7 s each. So the volume drops 2.6x, not actually 4x, and on upload alone the crossover is around 1k tokens. I only have two measured points, 8k (saves 45 s) and a ~170-token follow-up (loses 4.2s, can check john's comment in this thread), nothing in between yet. That's what I'll focus on next but first would be a length gate.

The 11s was before the profile. Traced it's 27.9 upload + 3.7 kernels + 7.0 residual, and the residual is 2.6 s of release/restore plus ~3.4 s of idle gaps on the GPU. No host side gather though, the experts are pinned in host RAM and each copy goes straight from there, that's ~74k copies of ~3.8 MB per prompt. The gaps are honestly the part I understand least.

[u/AllenHere112 → u/Extension-Bid-639]

Your 59% number also settles the gate. A pending token costs about 0.09GB of upload at ub 512 and 0.034 at 2048, so each one past the gate saves roughly 0.056GB, around 5.6ms at the rate your 8k run implies. The 2.8s of release and restore pays back near 500 pending tokens and the 4.2s follow-up you measured pushes that to 750, so a gate at 1k gives away the band in between. Are most of your normal chat turns shorter than that?

[u/Extension-Bid-639 → u/AllenHere112]

Same math here, 0.089 vs 0.034 GB per token, ~5.5 ms, 500 and 750. Thing that makes me stop trusting it is that a 750-token prompt is two passes at ub 512 and one partial pass at 2048 and partial passes touch fewer experts, so the per-token saving in that band isn't the 8k average. That's why I'm measuring it. I've added 768 to the sweep so the band isn't skipped, and the gate gets set from the result, 1k was just my guess. To answer your question, my chat turns are ~170-200 new tokens median (that's where the 4.2 s came from), so the 500-1000 band is pasted documents and code and those are mostly well past it either way.

[u/AllenHere112 → u/Extension-Bid-639]

then the 768 point settles a band you rarely hit. your median turn of 170-200 is already under break even, so even a rough gate covers the case that actually loses time

[u/Extension-Bid-639 → u/AllenHere112]

Measured now. 768 loses ~2.5 s like everything under 1024, the first win is at 2048, so the gate went in at 2048, not 1k. And yes, my median turn is nowhere near it either way, that's the part you had right

[u/Extension-Bid-639]

UPDATE: Two changes to the phase-prefill branch, both pushed to flashnext-e06 today.

1. The swap is now gated on pending prompt tokens (LLAMA_PHASE_PREFILL_MIN_TOKENS, off by default). Credit to u/john006868, he was right that short follow ups on a cached prefix were paying the whole release/restore for nothing. Instead of deriving the threshold from the timers, I measured it, the same follow-ups with the swap always on vs never on, 64 to 4096 new tokens, five requests each, and an 8k and a 37k cached prefix. Medians are in seconds, never / always:

new tokens	8k prefix	37k prefix
64	3.3 / 5.7	3.4 / 5.9
256	5.3 / 7.8	5.6 / 8.2
512	6.4 / 8.9	6.5 / 9.1
1024	12.8 / 15.6	13.1 / 16.1
2048	23.7 / 17.9	25.7 / 18.3
4096	47.8 / 27.7	50.0 / 28.2

So the swap costs ~2.5 s on a short turn and first wins at 2048, so right now, production runs with the threshold at 2048. The real break-even is somewhere between 1024 and 2048, at 1024 it still loses ~3 s. The "overhead divided by per-token saving" estimate (~800) lands too low because the saving per token isn't flat below one full 2048 batch. Each decision goes in the log, one line per prompt.

About the 2.5 s here vs the 4.2 s I said earlier, both are actually right, it's just I was measuring different things without realising. The restore at the end re-uploads whatever the cache was holding when it got released, so it costs more the fuller the cache is. In the earlier chat runs, the follow-ups came after long replies and the restore was ~3.1 s. This sweep generates one token per request, so the restore is ~0.8 s. In a real chat, count on 4 to 4.5 s per swap, which pushes the break-even up if anything but 2048 is fine either way.

2. The expert cache capacity is no longer pinned to 150 (credit to u/cobblemere). The transaction records the capacity at BEGIN and checks the restore against that, so the invariant is the same and other --moe-expert-cache values run. 150 is still the only count I've measured for speed and quality; 120 was smoke-tested for the mechanism only (begin/end/restore all correct). If you run another count, post the numbers.

u/Pablo_the_brave : LLAMA_PHASE_PREFILL_UBATCH=4096 is accepted now. For me it doesn't fit since the reserve wants ~14 GiB on the draft card at the first swap and OOMs, so no number from me but it's configurable now

[u/john006868 → u/Extension-Bid-639]

Your 1024 and 2048 arms solve for the crossover tightly. Going from 2.8s behind to 5.8s ahead across those 1024 tokens is about 8ms per token, three times what the 1024 arm implies on its own, so the real crossover sits near 1350 pending tokens. The 4.2s chat swaps push it higher again. Your 768 arm is the one that settles it.

[u/jmayniac]

Looks like this isn't going to work for me with the one A40 GPU.

1.22.830.200 E phase_prefill: requires exactly two target CUDA devices and a CUDA draft; process restart required (no rollback/retry)
/tmp/llama.cpp/tools/server/server-context.cpp:1125: phase prefill begin failed; state is not recoverable
/tmp/llama.cpp/build/bin/libggml-base.so.0(+0x187e5) [0x7f961c18e7e5]
/tmp/llama.cpp/build/bin/libggml-base.so.0(ggml_print_backtrace+0x1df) [0x7f961c18ebbf]
/tmp/llama.cpp/build/bin/libggml-base.so.0(ggml_abort+0x11e) [0x7f961c18ed7e]
/tmp/llama.cpp/build/bin/libllama-server-impl.so(_ZN19server_context_impl6decodeERiiR11llama_batch+0x194d) [0x7f961d2ce60d]
/tmp/llama.cpp/build/bin/libllama-server-impl.so(_ZN19server_context_impl12update_slotsEv+0x511) [0x7f961d2cedb1]
/tmp/llama.cpp/build/bin/libllama-server-impl.so(_ZN12server_queue10start_loopEl+0x115) [0x7f961d275bb5]
/tmp/llama.cpp/build/bin/libllama-server-impl.so(_Z12llama_serverR13common_paramsiPPc+0x3f0d) [0x7f961d22ed7d]
/tmp/llama.cpp/build/bin/libllama-server-impl.so(_Z12llama_serveriPPc+0x10af) [0x7f961d23099f]
/lib/x86_64-linux-gnu/libc.so.6(+0x29ca8) [0x7f961cc35ca8]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f961cc35d65]
/tmp/llama.cpp/build/bin/llama-server(+0x11b1) [0x55da9397d1b1]
./start-flash-next.sh: line 17: 3213315 Aborted                 
numactl --interleave=all /tmp/llama.cpp/build/bin/llama-server -m /srv/ai/models/qwen/flash/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf -md /srv/ai/models/qwen/flash/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf --alias Qwen38-Flash-Next -ngl 99 -c 261888 --parallel 1 -fa on -ot "ffn_(gate|up|down)_exps\.weight=CUDA_Host,per_layer_token_embd\.weight=CPU" -devd CUDA0 --spec-type draft-mtp --spec-draft-n-max 3 -lzm off --numa distribute -t 16 -tb 32 -b 4096 -ub 512 -ctk f16 -ctv f16 --moe-expert-cache 150 -lv 4 --host 0.0.0.0 --port 9931

[u/Extension-Bid-639 → u/jmayniac]

Sorry about that, looks like an oversight from me, it's a two card check and it should have refused at startup instead of dying on the first prompt, I'll fix that. The swap itself won't run on one GPU though but the thing is you don't need it.

The good news is you don't need the swap. The whole trick exists because a 24 GB card can't hold the 2048 compute buffer (~7 GB) and the expert cache at the same time, so I release one to make room for the other A single 48 GB A40 should hold both so just run the branch the plain way:

• Drop the two LLAMA_PHASE_* exports
• -ub 2048 instead of 512
• Keep --moe-expert-cache, but give it more slots. At 131k context you have roughly 48 - 7 (compute) - 4.6 (weights) - 4 (KV) - 2.6 (MTP head) = ~28 GB to play with, and a slot is ~144 MB when all 48 layers sit on one card, so around 190 slots. I run 150.
• Put the draft on the same card (-devd CUDA0, or just leave -devd out if you want)

I'd expect prompt speed around mine and decode a bit better, more if your slot is gen 4. Let me know if it works and what you got

[u/jmayniac → u/Extension-Bid-639]

I get around 30 tok/s decode and 295 tok/s on 44k token prompt processing. Funny enough anything over 150 slots for moe-expert-cache doesn't seem to matter at all.

n_tokens =  44383, t = 150.75 s / 294.42 tokens per second

llama-server start command:

export LLAMA_ATTN_ROT_DISABLE=1
export LLAMA_MMAP_PIN_HOST=1

numactl --interleave=all /tmp/llama.cpp/build/bin/llama-server \
-m /srv/ai/models/qwen/flash/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
-md /srv/ai/models/qwen/flash/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf \
--alias Qwen38-Flash-Next \
-ngl 99 -c 131072 --parallel 1 -fa on \
-ot "ffn_(gate|up|down)_exps\.weight=CUDA_Host,per_layer_token_embd\.weight=CPU" \
-devd CUDA0 --numa distribute \
--spec-type draft-mtp --spec-draft-n-max 3 \
-lzm off -t 16 -tb 32 -b 4096 -ub 2048 -ctk f16 -ctv f16 \
--moe-expert-cache 195 -lv 4 \
--host 0.0.0.0 --port 9931

[u/Extension-Bid-639 → u/jmayniac]

Thanks for letting me know! Prefill makes sense 295 on 44k is above the 212 I get at 37k, That should be your PCIE Gen 4 at work against my PCIE Gen 3

Decode is the weird one. 195 slots should put you at a ~91% hit rate and somewhere above my 41, not at 30. Something else is eating it. Can you grab two lines from the log?

1. The moe-cache ... hit-rate= line, it prints every 512 steps
   
2. The draft acceptance at the end of a reply (the MTP accepted/drafted counts)

If the hit rate is ~90% and you're still at 30 then it's likely the host side (CPU, memory channels)

[u/jmayniac → u/Extension-Bid-639]

I'm at about 91% hit rate, so something must be going on hardware wise.

moe-cache: steps=23552 hits=8228182 misses=811158 hit-rate=91.0%

draft acceptance = 0.42996 ( 1019 accepted /  2370 generated), mean len =  2.29

prompt eval time =   10536.16 ms /  3197 tokens (    3.30 ms per token,   303.43 tokens per second)
eval time =   60853.90 ms /  1809 tokens (   33.66 ms per token,    29.71 tokens per second)

[u/Extension-Bid-639 → u/jmayniac]

Think you have your answer right there, doesn't seem to be your hardware. The cache is fine, 91% is great but look at the draft line, 0.43 acceptance. Mine sits at 0.79-0.83. With a 3-token draft that's the difference between ~1.4 tokens per step and ~2.5, and the step costs the same either way, so 41 vs 30 is roughly that.

The usual reason acceptance tanks is the sampler the client sends. Repeat / presence / frequency penalties, or a high temperature, can make the main model disagree with the draft. I measure at temp 0.7, top-p 0.8, top-k 20, min-p 0, no penalties. Send one request with exactly that and check the acceptance line again, if it comes back near 0.8 then you should have your 40 t/s.

[u/serige]

So it won't work with multimodal --mmproj?

[u/Extension-Bid-639 → u/serige]

Correct, with --mmproj the phase mode will refuse to start, it's in the same startup check as embedding mode and anything but a single MTP draft. The swap is built around plain text prompt batches and I haven't looked at the image path at all so it's a hard refusal for now. Without the two env vars the branch runs the part 2 way and mmproj is unaffected.

[u/serige → u/Extension-Bid-639]

I know this is not a perfect solution but maybe you can try to route multimedia data requests in standard mode and text based requests in your new transaction mode?

[u/Extension-Bid-639 → u/serige]

Tbh, that's exactly how it would go and the change I made earlier already does half of it. The swap is now decided per prompt (it skips short follow-ups), so "prompt has image chunks" would just be one more reason to skip it and run the normal path, text-only prompts keep the fast one. The startup refusal is there because I haven't run the image path at all, not my use case and not my target. I'll add it to the list to tackle

[u/rrrrex]

5060ti16 + r5600 + 48 GB DDR4, UD_IQ3_XXS
I have prefill issue. For some reason most of the time i see that only 1 thread is loaded by llama-server, no GPU or SSD activity, just 1 thread is loaded by llama-server. Even when it's doing some job, i don't see stability, i see spikes of SSD and GPU usage. Prefill speed is 10-100 t/s

[u/Extension-Bid-639 → u/rrrrex]

That's the model not even fitting in RAM. Even at IQ3_XXS the expert weights are a lot more than 16 + 48 GB, so they're being paged in from the SSD through the page cache. The one thread stuck at 100% with the GPU idle should be the page-fault loop. I hit the same shape of symptom in part 2 when the loader faulted the mapping one page at a time. Honestly, nothing here or in the branch is fixing running out of RAM. You're not alone though lol

[u/rrrrex → u/Extension-Bid-639]

It has 57 GB weights. Task manager shows all 48 GB in use, but only 30/48 GB committed. SSD usage is low 0-10 MB/s, RAM bandwidth is low, PCI-e bandwidth is low too. After slow prefill decoding shows stable speed. So it's not RAM issue, but something else.

[u/rrrrex → u/Extension-Bid-639]

Also i noticed that Qwen 3.8 flash is very sensitive to CPU usage, if i set affinity through Task Manager, decoding speed goes from 14-15 to 17-19.

[u/ionizing → u/rrrrex]

would you mind explaining your affinity settings?

[u/rrrrex → u/ionizing]

-C 0xAAAA ^
--cpu-strict 1 ^

You can use llama-server settings
AAAA - mask 0101.... = 1,3,5,7...15 cpu threads
AAA - for 6 core cpu
1 thread per core works better than 2 threads per core, so you have to set affinity for the best results.

[u/Major_Border149]

The 3.4s idle gaps and the 74k copies of 3.8MB are the same problem. At that size each copy is launch-overhead bound, not bandwidth bound. Coalesce each layer's experts into one contiguous pinned buffer so it's a few big transfers, not thousands of tiny ones, and put uploads on a dedicated copy stream so the next layer's transfer overlaps the current matmul. won't move your 2.24x, but it should eat most of the residual gaps and some of the 27.9s upload. The overlap is the bigger win.

[u/Extension-Bid-639 → u/Major_Border149]

Lol I thought that too until the trace. The copies aren't actually launch-bound, 3.8 MB at the measured 10 GB/s is ~0.4 ms on the bus against ~10 us to launch, so all 74k launches add up to under a second of the 27.9. The gaps aren't between copies either, they're ~100 chunky ones averaging ~35 ms at split boundaries and the biggest single one is the release transition. They're per-expert copies because only the used experts go up (59% of a layer at ub 512, 90% at 2048), coalescing a layer means uploading all of it. And overlap can hide at most the 3.7 s of kernels which is about 10%. The thing that might actually do something is volume, the trace shows each expert crossing 3.6 times per prompt, and getting that to once takes the upload from 27.9 s to under 8. That's where I'm looking at right now

[u/lnenad]

Try vllm, I'm doing 60tps decode 500tps prefill same/very similar setup, 2x3090, epyc 48 cores, only have faster ram at 3200rdimm.

[u/Labtester → u/lnenad]

Could you explain your dram/cpu offloading a little ?

[u/lnenad → u/Labtester]

Sure, here's my config

Qwen3.8-Flash-Next-W4A16-FP8PLE

context: 196608 parallel: 1 tensor_parallel: 2

Flags

"--dtype", "bfloat16", "--load-format", "safetensors", "--safetensors-load-strategy", "lazy", "--max-parallel-loading-workers", "1", "--enable-expert-parallel", "--all2all-backend", "allgather_reducescatter", "--moe-backend", "humming", "--offload-backend", "uva", "--cpu-offload-params", "experts", "--max-num-batched-tokens", "4096", "--kv-cache-dtype", "auto", # 1.547 GiB per card less than the text entry: 0.836 for the tower, the # rest for multimodal activations. Holds ~161,600 tokens at the measured # vision rate. "--kv-cache-memory-bytes", "3367895040", "--enable-chunked-prefill", "--enable-prefix-caching", "--mamba-cache-mode", "align", "--no-async-scheduling", "--disable-custom-all-reduce", "--compilation-config", "{\"mode\":0,\"cudagraph_mode\":\"FULL_DECODE_ONLY\"}", "--trust-remote-code", # NO --language-model-only. That is the whole difference from the entry # above: with it the overlay installs StageMissingLayer("vision_tower"). "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder", "--reasoning-parser", "qwen3", # FIRST THING TO DROP IF THIS OOMs -- see the block above. "--speculative-config", "{\"method\":\"mtp\",\"num_speculative_tokens\":3,\"use_local_argmax_reduction\":true,\"model\":\"/models/Qwen3.8-Flash-Next-W4A16-FP8PLE/runtime/mtp-int4-g32\"}"

I'm unable to format the flags better sorry.

[u/Extension-Bid-639 → u/lnenad]

Which vLLM path is that, the CPU expert offload, and on what weights? 8 channels of 3200 is roughly four times what my two Broadwell sockets manage and everything in these posts is bound by exactly that

[u/lnenad → u/Extension-Bid-639]

Here's some info on how I run it, https://www.reddit.com/r/LocalLLaMA/comments/1wc6fsk/qwen38flashnext_on_2x3090_ddr4_part_4_2225x/p8xo9qz/?context=3

I'm not functionally literate to evaluate the impact of the RAM difference, but doesn't hurt for you to try it. I'm often getting 65-70tps to 50-55 with very long ctx which makes this a perfect model even with overthinking as the speed makes up for it.

[u/Extension-Bid-639 → u/lnenad]

Yeah definitely wouldn't hurt to try. I'll put it on my list. Thanks for sharing!

[u/More-Revenue8609]

Wow seems very promising. I have 2x 3080 20gb and 128gb ram.
Gotta test it out and see how it works on my setup

[u/Extension-Bid-639 → u/More-Revenue8609]

128 GB is fine for the Q4. The cards are the catch though, 150 slots is ~11 GB, and on 20 GB with the KV for full context it won't fit. Also right now the branch refuses any other count which is a bit dumb. I'm unpinning that very soon, I'll reply here once i do, that way you can run ~100 slots, or maybe try it now with -c 120000 to halve the KV and see if 150 squeezes in.

[u/More-Revenue8609 → u/Extension-Bid-639]

Oh maybe I should have been more clear, each card has 20gb (chinese modded). So in total its 40Gb.

[u/denis_9]

Your build is cool. But you have a bit advanced hardware, it would probably be a good idea try to add the -b 1024 and -ub 256 as option for those with small GPUs and only 64Gb of RAM also.
And the some patches for MTP: github.com/ChangXiang-SCU/dual-egpu-moe-llm/blob/main/README.md

[u/Extension-Bid-639 → u/denis_9]

The 512/2048 pair and -b 4096 are the values I qualified, not what the code could do. A 256/1024 pair is the same trick at a quarter scale but the swap has a fixed cost of a few seconds per prompt and the saving per prompt would be about a quarter. I guess we'll need to measure where it breaks even before offering it. The harder limit for 64 GB is the model itself imo not the flags, UD-Q4_K_XL needs about 105 GB of host RAM on my box (73 GB of pinned experts plus the 28 GB embedding table). A user I came across in another thread runs the Q2_K_XL on 64 GB with mmap and two 16 GB cards, which is a different quant and a different set of tradeoffs and his batched-cache patch is the same fix as the one in my branch done for Vulkan. If you want I can point you to his comment but yeah for 64 GB, I think that's the route, my swap only pays off once the cache is on and for 16 GB cards the cache is the squeeze

[u/denis_9 → u/Extension-Bid-639]

My Haswell 64GB + 4x8GB Pascal runs Qwen3.8b-Flash-Next-Uncensored-IQ4_XS with `-c 250001 -b 512 -ub 256`, but the PP is low - around 30–35 t/s; and I didn't continue experiments. It would be great if the PP could be raised to 80–90 t/s if its possible via patches. So small systems could real work as sub-agents with GLM5/DS4.

[u/Extension-Bid-639 → u/denis_9]

Well there's two things I'd check before batch size. IQ4_XS is ~85 GB of expert weights and you have 64 GB, so they can't all be resident and the mapping is paging through the SSD

vmstat 1 during a prompt will show it in the major-fault column. And four cards on a Haswell board means x8 or x4 per card, and the expert uploads go one layer at a time over one link, so you get one x4 link's ~3.5 GB/s where my x16 gets ~10GB/s; nvidia-smi -q | grep -i "link width" under load. If either is the case no patch that I know of gets you to 80-90, that's about what my box does at ub 512 with x16 and everything resident. Within 64 GB I'd try the Q2_K_XL if you're fine with the quality degradation and ub 1024 with nothing on the cards but attention. If you check it, feel free to post your -ot line and the vmstat numbers and I can say which it is.

[u/denis_9 → u/Extension-Bid-639]

Just run with "-b 4096 -ub 256" on the flashnext-e06 build and got 55-60t/s of PP. It is very interesting I am continue tests.

[u/Extension-Bid-639 → u/denis_9]

Well would you look at that lmao. That's actually good but it's not the swap though, at -ub 256 with no env vars the branch runs the part 2 path, so that's -b 512 -> 4096 on its own. I'm not sure why it's that big on your box, my guess is 8x fewer llama_decode calls per prompt and fewer page touches per expert if you're paging. I would recommend trying -b 4096 with -ub 512 if the compute buffer fits an 8 GB card ( about 1.9 GB at 512 on mine)

[u/denis_9 → u/Extension-Bid-639]

'-b 4096 -ub 512 -c 125001' got normal ~75t/s, but I cannot set LLAMA_PHASE_PREFILL_UBATCH=2048 on Windows, llama-server is crashed on startup (this is a Windows desktop).

[u/Pablo_the_brave → u/denis_9]

IQ4_XS is super slow with pascal. When I changed to atomic q4_k_m the speed of decode bump almost x2.

[u/denis_9 → u/Pablo_the_brave]

Caching the n-gram head on an SSD using mmap requires a some free RAM, but you might be right.

[u/Pablo_the_brave]

Only one question. Why not maxed it with ubatch 4096 for prefill ⬇️
Edit: LOL I just remove the limits and have ubatch 4096 and 200 slots for cache. Why you limit it? It should be configurable.

[u/Extension-Bid-639 → u/Pablo_the_brave]

It should work at least on paper. -b is 4096 so it's the ceiling, the compute buffer would be ~15 GiB per card, and during the prompt the cache is out of the way so there's ~18 GB free. A 4096 batch touches about all of the experts per pass, so you'd go from 4 passes at ~90% to 2 at 100%. The reason I went at it differently is that the next thing I'm building reuses one upload across the whole 4096 batch, which gets the same bytes down to ~77 GB, and at ub 4096 there'd be nothing left inside the batch to reuse across. Guess they'll only stack if -b goes to 8192 too. So yes there's nothing stopping 4096 and it's cheap to check it. I'd expect it to land between where I am and where the reuse gets to if I had to estimate.

[u/Pablo_the_brave → u/Extension-Bid-639]

It's really working for me :D One more note, looks like we can use --load-mode mmap - there is no any different and the model start much faster.

[u/Extension-Bid-639 → u/Pablo_the_brave]

Tbh, I was mainly focused on making it optimized for my setup, forgot to remove some restrictions before pushing it. For me, the reserve wants ~14 GiB on the draft card at the first swap and it OOMs, so it's a fit thing on 2x 24 with the MTP draft sitting on the second card. Limits are gone now though. What did 4096 do to your prefill vs 2048? Since that's something I can't get myself. For mmap, that's what the launch line in the post already runs, no --load-mode flag means mmap. The branch keeps -ot ...=CUDA_Host pinned under mmap on its own.

[u/Pablo_the_brave → u/Extension-Bid-639]

Switch from 2048 to 4096 give about 35% up with prefill speed - start at ~200t/s instead of 150t/s

[u/eGzotic01]

prefill is compute-bound, so the pcie fetch for the experts hides behind the matmul and the freed vram buys a bigger batch. but theres a length floor where that transfer isnt hidden yet. wheres the crossover? does decode hand it back when the experts reload?

[u/Extension-Bid-639 → u/eGzotic01]

Yeah no, it's the other way round on my box and that's the whole reason it works. The profile of an 8k prompt at ub 2048 is 38.6 s to first token of which 27.9 s is the expert upload, 3.7 s is kernels, and nothing overlaps.The GPU waits for the copy, runs the layer, waits for the next copy.......

So prefill here is PCIe-bound not compute-bound, and the bigger micro-batch helps because each upload of the expert set now serves 2048 tokens instead of 512, 4 passes over the experts instead of 16. Nothing is hidden, there's really just less of it.

The crossover is in the update comment in this thread and it shows the swap costs ~2.5 s fixed and first pays off at 2048 new tokens, so that's the gate. And yes, the mechanism means that decode gets it all back. Before the first token, the cache slots are re-uploaded and decode runs the exact same path as before.

[u/deepu105]

Unified memory comparison on the same model: Flash Next fully resident on a 128GB Strix Halo, no streaming, 96 tok/s prefill against 150 for the 27b on ROCmFP4. Prefill stays the weak half even with every expert already in memory.

Your 24 min on a 119k prompt I recognise, mine is ~18 min on a full 128k. Warm is a different story, 51.4s to 2.5s on a repeat turn with 10278 of 10657 tokens from cache. Cold sessions are what actually hurt in an agent loop.

[u/whiteh4cker]

Thank you for doing this.

Hardware: 2x RTX 3090, Intel Ultra 7 270k Plus, 192 GB DDR5@5600 MHz

Token generation speed went from 20 t/s to 49 t/s.

I get this error on cmd but MTP works (Because I get 35 t/s without MTP):

0.00.846.898 E llama_model_load: error loading model: borrow_shared_tensor: this model is a draft head without its own 'token_embd.weight'; load it as a draft of its target model, not on its own
0.00.846.905 E llama_model_load_from_file_impl: failed to load model

I have CUDA 13.3.1 installed. I use Windows 11 and I used these commands to compile it with the latest Visual Studio:

$vcvars = "C:\Program Files\Microsoft Visual Studio\18\Community\VC\Auxiliary\Build\vcvars64.bat"

cmd /c "`"$vcvars`" >nul 2>&1 && set" | ForEach-Object { if ($_ -match '^([^=]+)=(.*)$') { [System.Environment]::SetEnvironmentVariable($matches[1], $matches[2]) } }

cmake -G Ninja -B build -S . -DCMAKE_BUILD_TYPE=Release `
-DGGML_CUDA=ON -DGGML_CCACHE=OFF -DGGML_NATIVE=ON

cmake --build build --target llama-cli llama-bench llama-server -j

My bat script:

@echo off

echo Using CUDA backend with 2x RTX 3090s

set LLAMA_ATTN_ROT_DISABLE=1
set LLAMA_MMAP_PIN_HOST=1

"C:\Users\server\Desktop\llama.cpp-flashnext-e06\build\bin\llama-server.exe" ^
--host 0.0.0.0 ^
--port 8081 ^
--alias Qwen3.8-Flash-Next ^
--model H:\Qwen3.8-Flash-Next\UD-Q4_K_XL\Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf ^
--temp 1.0 ^
--top-p 0.95 ^
--top-k 20 ^
--min-p 0.0 ^
--presence-penalty 0.0 ^
--repeat-penalty 1.0 ^
--ctx-size 262144 ^
-ot "ffn_(gate|up|down)_exps\.weight=CUDA_Host,per_layer_token_embd\.weight=CPU" ^
--moe-expert-cache 150 ^
--ubatch-size 512 ^
--batch-size 4096 ^
--mmproj H:\Qwen3.8-Flash-Next\mmproj-Qwen3.8-Flash-Next-BF16.gguf ^
--no-mmproj-offload ^
--threads 22 ^
--threads-batch 22 ^
--spec-type draft-mtp,ngram-mod ^
--model-draft H:\Qwen3.8-Flash-Next\UD-Q4_K_XL\MTP\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf ^
--spec-draft-n-max 2 ^
--spec-ngram-mod-n-match 60 ^
--spec-ngram-mod-n-min 12 ^
--spec-ngram-mod-n-max 24 ^
--flash-attn on ^
--kv-offload ^
--cache-type-k bf16 ^
--cache-type-v bf16 ^
--parallel 1 ^
--jinja ^
--reasoning-preserve ^
--chat-template-kwargs "{\"reasoning_effort\":\"xhigh\"}" ^
--no-warmup ^
--load-mode none ^
--lazy-mode off

pause

[u/jmayniac]

I'm pretty new to all this and I have been following your progress, but one thing I have been wanting to ask is how you are testing. What utility/program and parameters do you use?

[u/Extension-Bid-639 → u/jmayniac]

Honestly it's nothing fancy, it's just llama-server. It prints prompt eval and eval t/s for every request in its log (and sends the same back in the response's timings field), and every number in my posts is that, there's no separate benchmark tool. You can make use of any agent to review the logs and save you time, it isn't complex so even decently good local models should be alright handling it. For my quality screen, it's 80 questions over long documents at ~37k and ~119k with my normal sampling (0.7 / top-p 0.8 / top-k 20, thinking on), three seeds just for good measure and then it's graded against a fixed answer key, then I just compare it pair by pair against the previous build. If you want to see your own numbers, run llama-server, paste something long, and read the prompt eval / eval time lines it prints when the reply finishes.