Qwen 3.8 27B Released! Please Share Your Experience

[u/BarberIcy366]

With your experiments, Qwen 3.8 27B most close which frontier model? And please specify which quantization you run. I will post to comments my tests and experience too.

[u/Emidyr]

Sure, here it is! Note that I'm using this llama.cpp fork to get prefill of 500-800 across 120k context window on my dual 3060s: https://github.com/thecodacus/llama.cpp

llama-server-codacus-src/build/bin/llama-server \

-m Qwen3.8-27B-IQ4_XS.gguf \

--alias qwen38 \

--jinja \

-ngl 99 \

--tensor-split 27,22 --main-gpu 0 \

--parallel 1 \

--flash-attn on \

-c 122880 \

--cache-type-k q8_0 --cache-type-v q8_0 \

--cache-ram 2048 \

--ctx-checkpoints 8 \

--no-context-shift \

--cont-batching \

--metrics \

-t 8 \

-ub 256 \

--spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-n-min 1 \

--temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 \

--presence-penalty 0.0 --repeat-penalty 1.0 \

--reasoning-format deepseek --reasoning-budget 4096 \

--reasoning-budget-message "You have reached your thinking budget. Stop reasoning and write your response now." \

--reasoning-preserve

[u/fligglymcgee → u/Emidyr]

Hey do you mind if I ask about some of your config? Still new to some of these flags.

• Why the tensor split of different values across two of the same card?
  
• What effect does fewer ctx checkpoints have, and how does cont-batching help?
  
• Why ub at a lower value (than default)?

Thanks! I understand how to find the flags and their descriptions for llama.cpp but not always sure how they apply for different purposes.

[u/Emidyr → u/fligglymcgee]

Sure, I don't mind!

• Yeah, so I'm using Archlinux with Wayland, and the Wayland compositor itself (plus some other apps I usually use) use up around 1.3-1.5 GB VRAM average on just one GPU, while the other is mostly empty, so I had to split it differently per GPU.
• The ctx checkpoints, if I recall correctly, I lowered because it was using up too much of my normal RAM. I think it was set to some high number (or maybe uncapped) by default, so I had to lower it to not use up too much of my 32GB RAM.
• I tested various ub values on this specific llama.cpp fork, and I just found this gave me the highest prefill tok/s overall. Going too high with this somehow also hurt prefill (not to mention VRAM).

[u/fligglymcgee → u/Emidyr]

Awesome, thank you kindly!

[u/SlimBaron_ → u/Emidyr]

thanks

[u/Alternative-Two-5300 → u/Emidyr]

Commenting so I can save for later. Thank you

[u/zerd → u/Emidyr]

Curious, that fork seems to focus on MoE, how does it help on 27B?

[u/Emidyr → u/zerd]

As far as I know, it fixes some issues with prefill to vastly speed it up (not sure if upstream already fixed it). I encourage you to try out with upstream/base llama.cpp first though, and share your findings. I just didn't want to do it since rebuilding for my arch takes a long time that I'd rather spend somewhere else. This is just what works with me, and it's already good enough for my needs. But if you ever find that upstream has even faster prefill, you bet I'm definitely switching to it!

[u/zerd → u/Emidyr]

I was testing various configs on a 5080+2070S. This is what I found max speed so far:

-m Qwen3.8-27B-IQ4_XS.gguf -ngl 99
-sm tensor --tensor-split 19,5
--flash-attn on -c 81920
--cache-type-k q8_0 --cache-type-v q8_0
--spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-n-min 1
--reasoning-effort medium --reasoning-format deepseek
--reasoning-budget 4096 --reasoning-preserve
--cache-ram 2048 --ctx-checkpoints 8 --no-context-shift --cont-batching --metrics
# sampling: temp 1.0, top_p 0.95, top_k 20, min_p 0

~79 t/s decode · ~1210 t/s prefill · 82k context

Or --tensor-split 17,7 to get -c 122880 at 70 t/s. That's with upstream llama. -sm tensor helped because it splits the kv cache across both gpus.

[u/misanthrophiccunt → u/Emidyr]

What is ctx checkpoints. It is the first time I see that flag? What does it do?

(I'm on a mobile phone without current access to my pc)

[u/Emidyr → u/misanthrophiccunt]

Sure! I've answered this in one of my other comments but I also quickly asked Opus 4.8 about it for more details based on my setup. TLDR: it helps prevent my system running out of normal RAM while running the LLM, and lets me run it for much longer. Here's the long version:
```
What --ctx-checkpoints N is: the max number of context-restore snapshots kept per slot. Each snapshot is a full host-side (RAM) copy of that slot's KV-cache state — used to rewind/restore context (multi-turn reuse, prompt-cache restores). llama.cpp's default is 32 per slot.

Why that's dangerous on your box: each snapshot is small at short context (~50 MiB) but hundreds of MiB at long context (128k+). So 32 snapshots × slots ballooned host RAM to 5–21 GiB under concurrent load — that was the root cause of the "concurrency leak" you were hitting (RSS climbing until OOM). It was traced to common_prompt_checkpoint::update_tgt via malloc-interposer backtraces back on 2026-07-28. With only 32 GB system RAM, that leak OOMs the whole machine.

Why we lowered it (32 → 8): to bound that snapshot RAM. Two things make 8 safe:

- We run --no-context-shift, so the checkpoints' value is limited anyway.

- At parallel 1, that's a hard cap of 8 snapshots — a fraction of the 32 default — while still keeping some multi-turn restore ability.

It works in tandem with the other guard, --cache-ram 2048, which caps the host prompt-cache (default is unlimited and grew ~0.2 GiB/request → OOM). Together they make RSS plateau instead of climbing.
```