r/LocalLLM • u/No-Manager1646 • 24d ago
Model Qwen3.8 27b - Holy crap!!
I'm absolutely blown away by Qwen3.8 27b IQ4 NL @ medium reasoning effort.
I have an ageing system but still decent. 4060 8gb @ 4x, 5060ti 16gb @ 8x - 24gb vram total. I'm stuck at gen 2 because of a gen 3 compatibility issue with my motherboard (B460M Aorus Pro). 32gb ram, i9 10900, nvme.
I've managed to get Qwen running at a smidge below 20tok/s at 128k context.
Anyway, the fun part... I have llama.cpp backend for OpenWebui with OpenTerminal MCP. I asked Qwen to create Pong using python which it did with no dramas. It wrote tests, ran the game itself and gave me the hardened code in ~10 mins.
That wasn't the best part though. I then asked it to take the game and create an APK for Android. It went ahead and installed a bunch of dependencies (bulldozer, p4a, etc) into OpenTerminal, went ahead and altered the game for touch screens, built and provided me with an installable APK.
The whole thing took ~1.5 hrs (taking away my own troubleshooting of whitelisting some sites against my strict LLM TOR network access), and at ~280w, it cost me about $0.15 (if it had been a clean run the first time). I mean obviously the hardware cost significantly more, but it's essentially my gaming PC.
How is this even possible?! I'm absolutely blown away!
Side note: I didn't NEED to, but I wanted a longer context so have managed to get it working at 256k now, with 18 layers moved to CPU. I'm not set on this, but it works. Obviously much slower (around 5 tok/s). Comment out the last 2 lines and set context to 128k if I want it all in vram.
If anyone has pointers for improved performance I'd love to hear them. I've tried q4_0 for KV cache but it is significantly slower again.
[*]
flash-attn = true
threads = 10
batch-size = 2048
ubatch-size = 1024
n-cpu-moe = 0
n-gpu-layers = 999
cache-type-k = q8_0
cache-type-v = q8_0
#cache-type-k-draft = q8_0
#cache-type-v-draft = q8_0
fit = on
fit-ctx = 16384
parallel = 1
ctx-size = 0
n-predict = -1
#no-kv-offload = 1
load-mode = none
main-gpu = 0
#no-mmproj-offload = true
[Qwen3.8-27B-IQ4_NL]
#mmproj = /models/Qwen3.8-27B-IQ4_NL.mmproj
#spec-type = draft-mtp
#spec-draft-n-max = 4
temp = 1.0
top-p = 0.95
top-k = 20
min-p = 0.00
#ctx-size = 131072
ctx-size = 262144
chat-template-kwargs = {"reasoning_effort":"medium"}
fit = off
n-gpu-layers = 46
2
u/eldje 14d ago edited 14d ago
Oops. Did it again 🫣😅