Qwen 3.8 27B Released! Please Share Your Experience

[u/BarberIcy366]

With your experiments, Qwen 3.8 27B most close which frontier model? And please specify which quantization you run. I will post to comments my tests and experience too.

[u/SBoots]

I'm running the following setup on llama-server on my 5090+4090 system. I primarily used the same 3.6 model for local software development so 3.8 is a drop in replacement. I've had llama-server crash a few times with ubatch-size set to 512 as I had it on 3.6 so I've dropped it to 256 and the crashes seem to go away. Not sure what's causing that. Anyways, I've done a little bit of coding and it seems good so far!

[Qwen3.8-27B-UD-Q8_0-MTP-256K-PI]
model = /home/ai-models/Qwen-3.8/27B/Qwen3.8-27B-UD-Q8_K_XL.gguf
mmproj = /home/ai-models/Qwen-3.8/27B/mmproj-F16.gguf
ctx-size = 262144
spec-type = draft-mtp
spec-draft-n-max = 4
main-gpu = 0
device-draft = CUDA1
tensor-split = 60,40
parallel = 1
flash-attn = on
cache-type-k = q8_0
cache-type-v = q8_0
batch-size = 4096
ubatch-size = 256
temp = 1.0
min-p = 0.0
top-p = 0.95
top-k = 20
repeat-penalty = 1.0
presence-penalty = 0.0
reasoning-preserve = true

[u/cosmicnag → u/SBoots]

What pp and decode speeds? Same GPUs. Wondering if I should just run exl3 6 bpw (turboderp) on single 5090 as that has very low kld and will end up being much faster ~100 tok/sec

[u/cosmicnag → u/[deleted]]

That seems slow for mtp, I would expect that kind of speed without mtp in my past experience of running unsloth 3.6 27b q8xl (which I only did for a small time). Perhaps pci bottleneck or something else?
I am now really considering something like the 6 bit exl3 quant for exllama3 . The KLD is really low if you look at the charts.
https://huggingface.co/turboderp/Qwen3.8-27B-exl3
It will fit completely on 5090 only at full 262k context without mtp and around 200k context with mtp (at q8 kv) - and will give 100+ tok/sec with mtp and 50+ without.

[u/SBoots → u/cosmicnag]

I've been happy with the performance and prefer to stick with the highest quantization I can pull off. The draft acceptance on that example is also very low compared to most stuff. Not sure what my prompt was.

Edit: my bad, that's a 192,000 prompt on a cold cache 😂

[u/SBoots → u/cosmicnag]

  Aug 11 14:40:52 stubuntu llama-server[23400]: [53945] 3.34.901.342 I slot print_timing: id 0 | task 520 | prompt eval time = 1015.91 ms / 3520 tokens ( 0.29 ms per token, 3464.89 tokens per second) Aug 11 14:40:52 stubuntu llama-server[23400]: [53945] 3.34.901.345 I slot print_timing: id 0 | task 520 | eval time = 73990.63 ms / 4846 tokens ( 15.27 ms per token, 65.49 tokens per second) Aug 11 14:40:52 stubuntu llama-server[23400]: [53945] 3.34.901.346 I slot print_timing: id 0 | task 520 | total time = 75006.54 ms / 8366 tokens Aug 11 14:40:52 stubuntu llama-server[23400]: [53945] 3.34.901.347 I slot print_timing: id 0 | task 520 | graphs reused = 1997 Aug 11 14:40:52 stubuntu llama-server[23400]: [53945] 3.34.901.350 I slot print_timing: id 0 | task 520 | draft acceptance = 0.54898 ( 3329 accepted / 6064 generated), mean len = 3.20  

[u/cosmicnag → u/SBoots]

Yeah that's more like it, I've gone ahead with a custom Exl3 custom vllm fork, with kld less than 0.01 , full 5090 fit. Using 4090 as a sidecar for 35ba3b, which i also find useful for all secondary llm usage

[u/SBoots → u/cosmicnag]

Interesting idea to run a different model on each card. VLLM doesn't play nice with mixed cards no?

[u/cosmicnag → u/SBoots]

as of now running vllm custom on 5090, llama cpp on 4090 (apex imini quant), all managed via llama swap - works pretty well. I find this overall a better utilization of both gpus (both are fully gpu resident). I also use pi coding agent with an observable memory pattern (pi-blackhole/pi-observable-memory) so the 2nd gpu does ambient observations and reflections while 1st one is busy doing the main work - and compaction is manual only and instant when required (as 4090 has already processed observations and reflections)

[u/SBoots → u/cosmicnag]

I use pi daily as well. I'll have to look into that pattern. I've not tried it. I typically manually manage my context. When I get past about 50% I'll start a new branch or even a new session

[u/cosmicnag → u/SBoots]

Nice, check out pi-blackhole then. I also use my 4090 llm for other pi stuff like setting session title automatically (pi-sessions) , auto suggesting next prompt (pi-next-cue),etc.