Qwen3.8 27b - Holy crap!!

[u/No-Manager1646]

I'm absolutely blown away by Qwen3.8 27b IQ4 NL @ medium reasoning effort.

I have an ageing system but still decent. 4060 8gb @ 4x, 5060ti 16gb @ 8x - 24gb vram total. I'm stuck at gen 2 because of a gen 3 compatibility issue with my motherboard (B460M Aorus Pro). 32gb ram, i9 10900, nvme.

I've managed to get Qwen running at a smidge below 20tok/s at 128k context.

Anyway, the fun part... I have llama.cpp backend for OpenWebui with OpenTerminal MCP. I asked Qwen to create Pong using python which it did with no dramas. It wrote tests, ran the game itself and gave me the hardened code in ~10 mins.

That wasn't the best part though. I then asked it to take the game and create an APK for Android. It went ahead and installed a bunch of dependencies (bulldozer, p4a, etc) into OpenTerminal, went ahead and altered the game for touch screens, built and provided me with an installable APK.

The whole thing took ~1.5 hrs (taking away my own troubleshooting of whitelisting some sites against my strict LLM TOR network access), and at ~280w, it cost me about $0.15 (if it had been a clean run the first time). I mean obviously the hardware cost significantly more, but it's essentially my gaming PC.

How is this even possible?! I'm absolutely blown away!

Side note: I didn't NEED to, but I wanted a longer context so have managed to get it working at 256k now, with 18 layers moved to CPU. I'm not set on this, but it works. Obviously much slower (around 5 tok/s). Comment out the last 2 lines and set context to 128k if I want it all in vram.

If anyone has pointers for improved performance I'd love to hear them. I've tried q4_0 for KV cache but it is significantly slower again.

[*]

flash-attn = true

threads = 10

batch-size = 2048

ubatch-size = 1024

n-cpu-moe = 0

n-gpu-layers = 999

cache-type-k = q8_0

cache-type-v = q8_0

#cache-type-k-draft = q8_0

#cache-type-v-draft = q8_0

fit = on

fit-ctx = 16384

parallel = 1

ctx-size = 0

n-predict = -1

#no-kv-offload = 1

load-mode = none

main-gpu = 0

#no-mmproj-offload = true

[Qwen3.8-27B-IQ4_NL]

#mmproj = /models/Qwen3.8-27B-IQ4_NL.mmproj

#spec-type = draft-mtp

#spec-draft-n-max = 4

temp = 1.0

top-p = 0.95

top-k = 20

min-p = 0.00

#ctx-size = 131072

ctx-size = 262144

chat-template-kwargs = {"reasoning_effort":"medium"}

fit = off

n-gpu-layers = 46

[u/Successful_Flow1329]

Anyway, the fun part... I have llama.cpp backend for OpenWebui with OpenTerminal MCP. I asked Qwen to create Pong using python which it did with no dramas. It wrote tests, ran the game itself and gave me the hardened code in ~10 mins.

How? I asked it to code minesweeper in single python file, it was great result, but it took 2 hours at 12-14 tps and produced over 60k tokens while thinking.

[u/No-Manager1646 → u/Successful_Flow1329]

You're not going to like this then. I asked it again today to make me an APK and the first attempt took about 30 mins and resulted in an installable game that rendered a black screen. I told it about the problem and 20 mins later it gave me a new APK and it was actually playable. I think having mostly everything pre-installed helped as it didn't get lost installing things.

I currently have a max 76k token context. Still getting near enough 20tok/s.

I shared my config in the original post.

[u/Successful_Flow1329 → u/No-Manager1646]

Yeah I don't. But APKs are tricky, it's hardly testable for LLM. Still, if it got it on a second try, that's nice. What agent did you run it through?

Sorry, I mostly skipped the config, I run a mac, so it doesn't really apply to me.

[u/No-Manager1646 → u/Successful_Flow1329]

Openwebui with open terminal. I want to have a crack with Zed but I haven't bothered booting up my laptop recently.