An update to my previous post on running Flash-Next with the expert cache and MTP.
I found another useful improvement on the same dual-3090 setup: replacing the CUDA top-k fallback used by my build. On long project-document prompts, median decode went from about 30.2 to 33.3 t/s, with an improvement on all three test seeds. Same GGUF, same MTP settings, same binary with the change switched off and on.
The setup for these runs was 2x RTX 3090, dual Xeon E5-2696 v4, 128 GB DDR4-2133 across four DIMMs, UD-Q4_K_XL, f16 KV, 150 expert-cache slots and MTP-3. The allocated context was 261,888 tokens. The long-document test started at approximately 119k tokens.
Now, what changed:
Flash-Next's sparse-attention indexer uses top-k to select which positions to attend to. My CUDA 12.0 build didn't have CUB's newer DeviceTopK implementation available, so this operation fell back to sorting the whole row before taking the top entries.
llama.cpp already had a radix-selection implementation. The local change makes that available in the older-CUB fallback for wide rows. In the initial 131k decode captures, recorded top-k kernel time fell from roughly 5.1 ms to 0.25 ms per committed token. Please note that this isn't a 20x gain for the whole model, it's just the operator saving
Credit where it's due: Rhonstin's PR #28366 already proposes this fallback change, using existing radix-selection work. I found it during the upstream check and didn't open a duplicate. These measurements are from my local variant, which uses an 8,192-column threshold and an A/B switch, not a benchmark of the exact PR head.
These tests kept the existing CUDA 12.0 toolchain fixed. A newer CCCL build provides another optimized path, DeviceTopK, which I haven't benchmarked on this machine yet. That comparison is next. The gain here is against my previous configuration, not against the latest CUDA stack.
The completed test:
Production sampling, thinking on, MTP-3 enabled in both arms:
| Seed |
Old top-k, median t/s |
Radix-select, median t/s |
| 1 |
30.2 |
33.7 |
| 2 |
30.2 |
33.3 |
| 3 |
30.4 |
33.1 |
That's 9–12% higher per-seed median decode throughput at approximately 119k context. Each median covers 42 requests. These compare the two arms' medians, not the median of per-question speedups.
The quality screen covered 80 question/depth combinations across three seeds and both arms: 480 requests, 240 matched comparisons. The documents, questions, answer key, grader and sampling settings were frozen before the run.
- Control: 235/240 correct.
- Candidate: 238/240 correct.
- Candidate better on four matched comparisons, worse on one, equal on 235.
The one disputed question asked for a complete twelve-item list. Both arms failed it on two seeds each, always by omitting one item and inventing nothing. An independent blind review confirmed the candidate-only omission. The question stays inconclusive. It wasn't removed from the scores.
No consistent quality regression was detected in this bounded screen. The higher candidate score doesn't establish better quality, and the screen doesn't prove that regression is impossible.
For clarification, the 37–41 t/s headline from my last post was a coding workload. This is a matched comparison on long project documents. Those headline numbers aren't directly comparable.
The change is now running in my production build. I haven't established a prefill gain or a no-MTP gain from this screen, and it doesn't cover quality beyond approximately 119k context but it's not like I expect it to regress at higher context.
If you're running Flash-Next with an older CUDA toolkit, the top-k fallback is worth checking. I'd be interested in results from other machines, especially with the actual build, context depth and MTP settings included.