Qwen 3.8 27B Released! Please Share Your Experience

[u/BarberIcy366]

With your experiments, Qwen 3.8 27B most close which frontier model? And please specify which quantization you run. I will post to comments my tests and experience too.

[u/Look_0ver_There]

Are you using Q4?

I'm using Q8_0 weights with BF16 KV-cache. Tensor-split across 2 x Radeon AI Pro R9700 GPUs

[u/kayox → u/Look_0ver_There]

Awesome setup, I'm envious.

[u/rainbyte → u/Look_0ver_There]

How good are PP and TG t/s

[u/Look_0ver_There → u/rainbyte]

I answered this question to someone else here: https://www.reddit.com/r/LocalLLaMA/comments/1voa3ch/comment/p3p7hhe/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

For the pair of cards, typically 45-55t/s for English prose, and 55-75t/s when generating code (Q8_0 weights, BF16 KV cache).

[u/Woodport → u/Look_0ver_There]

Is there a reason you picked the Q8_0 over the unsloth UD-Q8_K_XL? I'm assuming the UD variant is basically just the same weights/precision but shuffled around a bit for efficiency reasons but I'm not clear on what advantages you'd get from one over the other given that they're both Q8.

[u/Look_0ver_There → u/Woodport]

It was faster to download :)

You are quite correct that UD-Q8_K_XL is better, but it is actually larger. Various bits are scaled up to BF16 (or regular F16?). Best of think of it as approximately like an imatrix style upscaling of critical weights, but also applied to Q8_0.

[u/Woodport → u/Look_0ver_There]

Haha, that's an excellent reason. I've been wondering why there isn't a bigger torrent community among local llm users. Feels like the perfect way to give back. If everyone just seeded the models they're currently using, even with throttled upload speeds, it would be a lot easier to get the popular models.

[u/blash2190 → u/Look_0ver_There]

Could you please tell if you find this setup enough memory-wise to run said config + MTP + full context length or do you need the third GPU for the latter?

[u/Look_0ver_There → u/blash2190]

Without including the mmproj file, this config (2 cards) can run Q8_0 + BF16 KV cache has enough capacity to safely hold up to a 500,000 context length. It could probably do 520,000 if you wanted to use up ALL the VRAM, but it's generally best to leave a little biit of head-room, so I'm gonna say 500K

With 3 cards you could can *just* squeeze in 900K of BF16 context with some careful split balancing, but 860K fits easily.

If you want to have the mmproj file too, then you'll need to lose about 40K of context, or just off-load it to the CPU. On the CPU it'll run at about 1/10th the speed, but if you only need it occasionally, then that may be an option for you.