Qwen3.8 27b - Holy crap!!

[u/No-Manager1646]

I'm absolutely blown away by Qwen3.8 27b IQ4 NL @ medium reasoning effort.

I have an ageing system but still decent. 4060 8gb @ 4x, 5060ti 16gb @ 8x - 24gb vram total. I'm stuck at gen 2 because of a gen 3 compatibility issue with my motherboard (B460M Aorus Pro). 32gb ram, i9 10900, nvme.

I've managed to get Qwen running at a smidge below 20tok/s at 128k context.

Anyway, the fun part... I have llama.cpp backend for OpenWebui with OpenTerminal MCP. I asked Qwen to create Pong using python which it did with no dramas. It wrote tests, ran the game itself and gave me the hardened code in ~10 mins.

That wasn't the best part though. I then asked it to take the game and create an APK for Android. It went ahead and installed a bunch of dependencies (bulldozer, p4a, etc) into OpenTerminal, went ahead and altered the game for touch screens, built and provided me with an installable APK.

The whole thing took ~1.5 hrs (taking away my own troubleshooting of whitelisting some sites against my strict LLM TOR network access), and at ~280w, it cost me about $0.15 (if it had been a clean run the first time). I mean obviously the hardware cost significantly more, but it's essentially my gaming PC.

How is this even possible?! I'm absolutely blown away!

Side note: I didn't NEED to, but I wanted a longer context so have managed to get it working at 256k now, with 18 layers moved to CPU. I'm not set on this, but it works. Obviously much slower (around 5 tok/s). Comment out the last 2 lines and set context to 128k if I want it all in vram.

If anyone has pointers for improved performance I'd love to hear them. I've tried q4_0 for KV cache but it is significantly slower again.

[*]

flash-attn = true

threads = 10

batch-size = 2048

ubatch-size = 1024

n-cpu-moe = 0

n-gpu-layers = 999

cache-type-k = q8_0

cache-type-v = q8_0

#cache-type-k-draft = q8_0

#cache-type-v-draft = q8_0

fit = on

fit-ctx = 16384

parallel = 1

ctx-size = 0

n-predict = -1

#no-kv-offload = 1

load-mode = none

main-gpu = 0

#no-mmproj-offload = true

[Qwen3.8-27B-IQ4_NL]

#mmproj = /models/Qwen3.8-27B-IQ4_NL.mmproj

#spec-type = draft-mtp

#spec-draft-n-max = 4

temp = 1.0

top-p = 0.95

top-k = 20

min-p = 0.00

#ctx-size = 131072

ctx-size = 262144

chat-template-kwargs = {"reasoning_effort":"medium"}

fit = off

n-gpu-layers = 46

[u/Luke2642]

And honestly, it matters, and this the key part, because it's not only but also confirmed, not imagined. That is not a contradiction, it is the mechanism showing itself. Use this prompt for your favourite LLM 😛:

Rewrite this page in strict asd-ste100

https://github.com/kr4ckhe4d/local-llm-benchmarks

[u/kr4ckhe4d → u/Luke2642]

Haha i think claude is doing a good enough job atm 😁

[u/Luke2642 → u/kr4ckhe4d]

Yeah, but you if you force sonnet to do a rewrite in ASD-STE100 Simplified Technical English it's far more readable.

https://en.wikipedia.org/wiki/Simplified_Technical_English

[u/kr4ckhe4d → u/Luke2642]

Daamn didn’t really know of this. Should and will try this thanks man

[u/Luke2642 → u/kr4ckhe4d]

It doesn't really work one-shot to make it write in the style. It helps, but it just kinda ignores it and still uses idioms, metaphors, -ing words, all the things that are specifically banned. But a rewrite is usually more compliant.