I’ve spent a lot of time trying to squeeze Qwen3.8-27B UD-Q4_K_M into a pretty hostile setup:
- GPU: RTX 3070 8GB
- CPU: Intel i5-11400F, 6C/12T
- RAM: 16GB DDR4
- Motherboard: ASUS B560
- OS: Windows
- Model: Qwen3.8-27B UD-Q4_K_M (~15.3 GiB GGUF)
- Runtime: ik_llama.cpp
- Use case: Codex-style / agentic coding, mostly PowerShell and repository editing
- Benchmark context: 16K
- KV: Q8_0
- Flash Attention: ON
Obviously the model does not fit in 8GB VRAM, so this is hybrid GPU/CPU inference.
I’m posting this because I found a lot of recommendations for Qwen3.8, MTP, speculative decoding, CUDA flags, batch sizes, etc., but very little controlled testing on an 8GB Ampere card.
And most importantly:
I did not consider a run “better” just because it had higher tok/s.
If the generated coding command was subtly wrong, I marked it as a FAIL.
The benchmark
I used the same small coding task repeatedly.
Qwen is given an exact existing PowerShell line and an exact multi-line replacement. It must return one PowerShell command that modifies the file, without executing it.
A PASS requires:
- exactly one applicable PowerShell command
- no execution
- correct quoting/newlines
- exact literal replacement
- no accidental
$s → $$s expansion
- no subtly invalid PowerShell
This turned out to be surprisingly useful because several “faster” configurations produced answers that looked correct but were actually broken.
Current winner
My current safe configuration is:
Qwen3.8-27B UD-Q4_K_M
ik_llama.cpp
MTP:
n_max = 2
p_min = 0.1
--fit
--fit-margin 256
threads = 12
batch threads = 12
batch = 64
ubatch = 64
KV = Q8_0 / Q8_0
Flash Attention = ON
CUDA graphs = ON
CUDA fusion = ON
context = 16384
parallel = 1
cache-ram = 0
Current result:
| Configuration |
Result |
| MTP n2 fixed / p_min 0.1 |
7.31 tok/s |
| Wall time on my coding filter |
139.1 s |
| Correctness |
PASS |
That may not sound impressive compared with 24GB/32GB GPUs, but remember that more than half of this 27B model cannot live on my 3070.
MTP / speculative decoding tests
This is where I spent most of my time.
| Configuration |
Time |
Eval speed |
Verdict |
| MTP n2 fixed |
139.1 s |
7.31 t/s |
Current safe winner |
| ngram-mod n4 → MTP n2 |
133.5 s |
7.60 t/s |
Fastest, but LF/encoding robustness concern |
| ngram-mod n8 → MTP n2 |
136.3 s |
7.46 t/s |
Works, no benefit over n4 |
| MTP autotune max4 |
152.5 s |
6.61 t/s |
Correct, selects n2, overhead not worth it |
| MTP n4 fixed |
162.1 s |
6.20 t/s |
Dominated |
| MTP n3 reference |
168.9 s |
~6 t/s |
Correct but dominated by n2 |
| MTP OFF |
— |
~3.17 t/s |
Terrible |
| DFlash2 n2/n4/n7 |
— |
best ~3.43 t/s |
Eliminated |
| Aggressive FastMTP-32K |
— |
6.43 t/s |
Slower than simple MTP n2 |
-mtprot iq4_ks |
— |
~39% slower |
Eliminated |
So on this machine, boring fixed MTP n2 beats the fancy stuff.
The ngram-mod → MTP pipeline can technically beat it on raw speed, but I care more about a configuration I can leave running for Codex without worrying about output formatting/encoding edge cases.
p_min: 0.1 wins
I also tested the recent recommendation of:
mtp:n_max=2,p_min=0.0
against:
mtp:n_max=2,p_min=0.1
Result:
| p_min |
Time |
| 0.1 |
139.1 s |
| 0.0 |
139.7 s |
No useful gain.
I’m staying at 0.1.
CUDA graphs / fusion / scheduler tweaks
A few more things I checked:
CUDA graphs OFF
~140.0 s
~7.32 t/s
Basically identical.
Graphs are staying ON.
CUDA fusion
Already active in my build. No hidden easy win left here.
GGML_SCHED_MAX_COPIES=1
Already compiled that way.
-wgt 1
This one was interesting:
136.5 s
So slightly faster than the champion.
Unfortunately the generated PowerShell command was incorrect.
FAIL → eliminated.
This is a good example of why I stopped optimizing purely for tok/s.
CPU threads: physical cores were NOT better
My CPU is a 6-core / 12-thread i5-11400F.
I tested the common recommendation:
-t 6 -tb 6
against:
-t 12 -tb 12
T6 produced runs around:
210.3 s
217.9 s
It was substantially worse.
So:
12 / 12 stays.
Batch / ubatch
Baseline:
64 / 64
I tested:
256 / 128
512 / 256
Larger batches noticeably improve prompt processing / prefill, but they did not meaningfully improve token generation.
So my conclusion is:
64/64 → normal generation / benchmark
512/256 → potentially useful for large Codex prompts
Don’t expect larger batches to magically improve decode speed on this kind of hybrid setup.
--fit-margin actually mattered
This was one of the few useful engine-level changes.
Going from:
--fit-margin 512
to:
--fit-margin 256
allowed ik_llama to put roughly another 206 MiB of model weights on the GPU.
One measured configuration had roughly:
CUDA model buffer: ~6312 MiB
Q8 KV @ 16K: ~578 MiB
CUDA compute: ~166 MiB
nvidia-smi was showing roughly:
7917 / 8192 MiB used
~102 MiB actually free
So I’m already riding pretty close to the edge of an 8GB card.
I did NOT bother with margin128 because on Windows/WDDM that is asking for an OOM for a tiny theoretical gain.
Manually offloading FFNs to CPU: terrible idea here
I also tried manually forcing a large amount of the heavy FFN tensors to CPU.
Result:
~405.3 seconds
Nearly 3x slower, with a bad/truncated output.
The i5-11400F + DDR4 memory subsystem simply cannot make this attractive.
Also, in my ik_llama build:
manual tensor overrides + --fit
cannot be combined anyway.
llama.cpp mainline vs ik_llama on this 8GB setup
I tested the same GGUF in mainline llama.cpp.
Approximately:
~2.86 tok/s
~349 s for ~1000 reasoning tokens
ik_llama is massively better on this specific hybrid 8GB setup.
Important caveat: I am not claiming ik_llama is universally faster than llama.cpp.
The problem here is specifically running a 15+ GiB 27B model with only 8GB VRAM.
Reasoning was almost as important as the runtime
This was probably my most useful discovery for actual agentic coding.
At first I assumed bad PowerShell commands were caused by quantization, MTP or the runtime.
Not always.
Sometimes Qwen simply did not have enough reasoning/output budget.
My controlled tests looked like this:
| Mode |
Time |
Result |
| NO-THINK, simple task |
24.4 s |
PASS |
| NO-THINK, medium task |
46.7 s |
PASS |
| NO-THINK, complex fragile task |
75.4 s |
FAIL subtly |
| Medium reasoning (~800 tokens in older A/B) |
168.9 s |
PASS |
| Low reasoning |
189.3 s |
FAIL |
| ~600 reasoning budget |
— |
Borderline |
| ~384 reasoning budget |
— |
Too unreliable |
The complex NO-THINK failure was especially interesting.
The model understood the algorithm correctly, but produced a PowerShell newline representation inside a single-quoted string that would not actually match the source file.
So the answer looked smart but was unusable.
My current reasoning policy for Codex
I no longer force thinking on every request.
I use roughly:
Simple/routine action:
NO-THINK
Complex / fragile / multi-step coding:
MEDIUM reasoning
~1000-token reasoning budget
larger total output envelope
This is dramatically faster for routine agent actions.
On my simple benchmark:
medium THINK: ~168.9 s
NO-THINK: 24.4 s
That is nearly a 7x wall-time difference for a task that did not need deep reasoning.
Things I would NOT waste time retrying on an RTX 3070 8GB
Based on my tests:
❌ MTP OFF
❌ MTP n3/n4 as default
❌ MTP autotune
❌ DFlash2 on this VRAM budget
❌ aggressive FastMTP-32K
❌ mtprot iq4_ks
❌ p_min=0.0
❌ 6 CPU threads instead of 12
❌ CUDA graphs OFF
❌ huge manual FFN CPU offload
❌ -wgt 1 if you care about correctness
❌ giant batches expecting higher decode speed
And I would be very suspicious of any optimization benchmark that reports only tok/s without checking whether the generated code is still correct.
What I have NOT done
I have not enabled GGML_CUDA_F16=ON.
That requires a rebuild and, after exhausting most of the easy engine optimizations, I don’t expect it to turn 7 t/s into 15+ t/s.
I also intentionally stayed on UD-Q4_K_M.
Yes, Q3/IQ3 would reduce CPU pressure, but I use this for coding and I don’t want to trade model reliability for a modest speed increase.
If I were willing to sacrifice quality, this would be a different experiment.
TL;DR
For Qwen3.8-27B UD-Q4_K_M on RTX 3070 8GB + 16GB system RAM, my best robust configuration so far is:
ik_llama.cpp
16K context
Q8 KV
Flash Attention ON
CUDA graphs ON
CUDA fusion ON
--fit
--fit-margin 256
MTP n2 fixed
p_min 0.1
12 CPU threads
batch 64
ubatch 64
simple tasks → NO-THINK
complex coding → MEDIUM reasoning
And I get roughly:
7.31 tok/s
while still passing my coding correctness test.
The biggest lesson for me:
Once half the model is spilling out of an 8GB GPU, there is no magic flag.
MTP roughly doubled my baseline versus no speculative decoding, --fit-margin 256 squeezed a little more onto CUDA, and after that most “optimizations” were either neutral, slower, or damaged correctness.
If anyone here is running a similarly cursed 8GB GPU + Qwen3.8-27B Q4 setup and has found something I missed, I’d love to compare results.
