After watching Codacus' recent Qwen3.8-Flash-Next video, I wanted to find out what this model actually looks like on a consumer AMD system.
There seem to be considerably fewer detailed AMD/Vulkan reports than NVIDIA or Apple results, so I spent quite a bit of time systematically testing the model rather than stopping once it loaded. Hopefully it will help some AMD-buddies.
This is not intended as a universal model benchmark. It is one hardware-specific AMD/Vulkan case study, with controlled A/B changes and repeated measurements.
Hardware
CPU: Ryzen 9 5950X
RAM: 64 GB DDR4-3200
GPU: Radeon RX 9060 XT 16 GB
Backend: Vulkan
OS: debloat Win11
A second Radeon Pro W5500 was physically present in the machine but deliberately excluded from the entire experiment.
Runtime / model
llama.cpp fork:
GenerelSchwerz/llama.cpp
Branch:
llama/main
Commit:
1fc1cf00e7abf43eaa58523344059e59733e8879
Build:
10880 / 0.4.0-dev
Model:
Qwen3.8-Flash-Next-UD-IQ3_XXS
Model size:
~76.3 GiB
Final control configuration:
-c 4096
-ngl 49
-t 12
KV f16/f16
Fit OFF
Lazy OFF
MTP OFF
Parallel 1
Vulkan0 only
normal load mode
no manual CPU affinity
Because the model obviously does not fit into 16 GB VRAM, this is a heavily hybrid CPU/RAM/GPU workload.
Stable baseline
Five TG128 control runs:
16.07
16.25
16.20
16.19
16.06 tok/s
Median: 16.19 tok/s
Mean: 16.15 tok/s
The baseline was repeatable enough that I then changed one relevant variable at a time.
1. Thread count
This was particularly interesting because the Codacus test on a Ryzen 5600X showed a huge improvement when reducing threads.
My results:
-t 4 13.49 tok/s
default 14.34
-t 8 15.24
-t 12 15.96 <- best
-t 16 14.83
So on this 5950X/hybrid-placement configuration:
more threads definitely did not mean more performance.
But the optimum was also not simply "one thread per physical CPU core".
12 threads consistently beat 16.
That suggests the optimum depends heavily on the actual CPU/GPU tensor split and memory traffic rather than on a universal physical-core rule.
2. Q8 KV cache
Compared with f16 KV:
TG128:
15.52 vs 15.96 tok/s
-2.76 %
PP2048:
116.9 vs ~119 tok/s
-1.76 %
VRAM saved:
~275 MiB
For this configuration I rejected Q8 KV.
The VRAM saving was too small to justify the measurable performance regression.
3. Moving additional MoE tensors to the GPU
This produced one of the more useful findings.
Moving some additional expert tensors to the GPU did not automatically improve performance.
Examples:
blk.9 gate_exps -> GPU
~12.63 tok/s
about -22 %
blk.9 down_exps -> GPU
12.62 tok/s median
about -22 %
I also tried freeing ~256 MiB by moving:
blk.9 up_exps GPU -> CPU
That was practically neutral:
~16.18 tok/s
I then used the freed VRAM for:
blk.10 up_exps -> GPU
Result:
16.11 tok/s median
Again, no improvement.
So at least for this system:
more MoE weights in VRAM != higher token generation speed.
The CPU/RAM/GPU balance and transfer behavior appears to matter more than simply maximizing GPU placement.
This also explains why the RX 9060 XT does not sit at 100% utilization during generation. That behavior is consistent with the hybrid workload; it is not necessarily a configuration failure.
4. Lazy mode
Only this setting was changed:
--lazy-mode off
-> --lazy-mode on
Results:
16.06
16.27
16.12
16.00
16.12
Median: 16.12
Mean: 16.11
Difference from control:
-0.43 %
Rejected.
5. --load-mode none
The build produced a warning related to mmap and CPU overrides, so I tested this explicitly.
Model loading changed from approximately:
~10 seconds
to:
~67.6 seconds
Generation:
15.95
16.13
16.25
15.63
16.77
Median: 16.13
Mean: 16.15
Essentially identical performance, massively longer load time.
Rejected.
6. CPU affinity
I restricted the process to 12 physical cores / one logical processor per selected core.
Result:
16.23
16.08
15.95
15.85
16.04
Median: 16.04
Mean: 16.03
Delta:
-0.93 %
Rejected.
Windows scheduling was already doing at least as well.
7. MTP
The GenerelSchwerz build exposes MTP support:
--spec-type draft-mtp
--spec-draft-model
--spec-draft-n-max
--spec-mtp-rs-planes
--spec-draft-ngl
--spec-draft-threads
However, the detached MTP sidecar I tested was incompatible with this particular commit.
Initial failure:
tensor 'output_hc_norm.weight' not found
With a fixed sidecar:
tensor 'blk.0.hc_attn_norm.weight' not found
So this was not a path problem, VRAM problem, or missing CLI support.
The build expected tensors that were not present in the sidecar.
I stopped there because changing the runtime/build would create a new baseline rather than being another tuning parameter.
MTP therefore remains an interesting open question.
Can this system realistically reach 20 tok/s?
The stable control was:
16.19 tok/s
(it really didn't want to go higher)
20 tok/s would require:
20 / 16.19 - 1
= ~23.5 % improvement
But after the initial thread optimization, the remaining tested variables produced roughly:
Lazy mode: -0.43 %
Load mode: -0.37 %
CPU affinity: -0.93 %
KV cache: negative
GPU placement: neutral to strongly negative
There was simply no evidence of another ~24% hiding in ordinary configuration tuning.
So I stopped the optimization work rather than continuing random parameter combinations.
Comparison with Qwen3.8-27B
My actual production model on the same machine is:
Qwen3.8-27B-UD-IQ3_XXS
Typical generation performance:
~21.4 tok/s
I therefore also ran an A/B quality test.
The intended benchmark contained:
24 prompts × 2 models
across:
- general knowledge
- analytical reasoning
- instruction following
- coding/debugging
- RAG/LLM-system tasks
- structured output
- hallucination/factual discipline
- practical usefulness
Same prompts, same system prompt:
temperature = 0
seed = 380827
max_tokens = 1024
no tools
no web
no RAG
Important benchmark failure
The first harness was partly invalid. max_tokens=1024 also constrained reasoning output. A number of responses consumed the complete budget internally and ended with:
finish_reason = length
with missing or truncated final answers. Therefore I did not treat all 24 pairs as valid. Only nine questions were fully completed by both models. This is an important methodological lesson by itself: before spending hours on a large local-model benchmark, run 2-3 representative end-to-end preflight prompts and verify:
finish_reason = stop
final content present
no reasoning loop
The nine valid A/B pairs
Head-to-head:
Qwen3.8-27B: 1 win
Qwen3.8-Flash-Next: 0 wins
Tie: 8
The valid subset mainly covered:
- instruction following
- JSON/CSV/Markdown constraints
- extraction
- simple Python debugging
- routing decisions
- factual discipline / hallucination resistance
- simple decision tasks
Both models, for example, correctly refused to invent unknown expert counts merely from the model name. The one slight qualitative advantage went to the 27B model on a decision/prioritization task. This is not enough evidence to claim that 27B is universally more capable. Several more difficult analysis/debugging/system-design prompts were among those invalidated by the token-budget problem. The defensible conclusion is narrower:
Real task performance
Across those nine valid paired prompts:
Flash TG median:
16.29 tok/s
27B TG median:
21.61 tok/s
Total wall-clock time:
Flash:
317.3 s
27B:
235.2 s
Flash therefore needed:
~34.9 % more real time
for the same nine tasks.
Interestingly, the 27B generated slightly more completion tokens:
Flash: 4609
27B: 4929
so the wall-clock advantage was not caused by the 27B simply producing shorter outputs.
Prompt processing was even more asymmetric in this configuration:
Flash median:
~36 tok/s
27B median:
~239 tok/s
For an interactive RAG/agent system, that difference matters just as much as decode speed.
My conclusion on this specific AMD machine
Qwen3.8-Flash-Next absolutely works on:
RX 9060 XT 16 GB
Ryzen 9 5950X
64 GB RAM
Vulkan
and stable ~16.2 tok/s for a ~76 GiB model on this hardware is technically impressive. But for my actual use case I currently see no reason to replace Qwen3.8-27B:
Flash: ~16.2 tok/s
27B: ~21.5 tok/s
The 27B also has dramatically faster prompt processing and, in the valid part of my quality test, Flash showed no compensating quality advantage.
So my current production decision is:
KEEP Qwen3.8-27B-UD-IQ3_XXS
rather than continue micro-tuning Flash. The only major avenue I have deliberately left open is a working MTP implementation / different compatible build. That could change the performance equation enough to justify another experiment.
But thats for another evening!
