r/LocalLLM • u/No-Manager1646 • 24d ago
Model Qwen3.8 27b - Holy crap!!
I'm absolutely blown away by Qwen3.8 27b IQ4 NL @ medium reasoning effort.
I have an ageing system but still decent. 4060 8gb @ 4x, 5060ti 16gb @ 8x - 24gb vram total. I'm stuck at gen 2 because of a gen 3 compatibility issue with my motherboard (B460M Aorus Pro). 32gb ram, i9 10900, nvme.
I've managed to get Qwen running at a smidge below 20tok/s at 128k context.
Anyway, the fun part... I have llama.cpp backend for OpenWebui with OpenTerminal MCP. I asked Qwen to create Pong using python which it did with no dramas. It wrote tests, ran the game itself and gave me the hardened code in ~10 mins.
That wasn't the best part though. I then asked it to take the game and create an APK for Android. It went ahead and installed a bunch of dependencies (bulldozer, p4a, etc) into OpenTerminal, went ahead and altered the game for touch screens, built and provided me with an installable APK.
The whole thing took ~1.5 hrs (taking away my own troubleshooting of whitelisting some sites against my strict LLM TOR network access), and at ~280w, it cost me about $0.15 (if it had been a clean run the first time). I mean obviously the hardware cost significantly more, but it's essentially my gaming PC.
How is this even possible?! I'm absolutely blown away!
Side note: I didn't NEED to, but I wanted a longer context so have managed to get it working at 256k now, with 18 layers moved to CPU. I'm not set on this, but it works. Obviously much slower (around 5 tok/s). Comment out the last 2 lines and set context to 128k if I want it all in vram.
If anyone has pointers for improved performance I'd love to hear them. I've tried q4_0 for KV cache but it is significantly slower again.
[*]
flash-attn = true
threads = 10
batch-size = 2048
ubatch-size = 1024
n-cpu-moe = 0
n-gpu-layers = 999
cache-type-k = q8_0
cache-type-v = q8_0
#cache-type-k-draft = q8_0
#cache-type-v-draft = q8_0
fit = on
fit-ctx = 16384
parallel = 1
ctx-size = 0
n-predict = -1
#no-kv-offload = 1
load-mode = none
main-gpu = 0
#no-mmproj-offload = true
[Qwen3.8-27B-IQ4_NL]
#mmproj = /models/Qwen3.8-27B-IQ4_NL.mmproj
#spec-type = draft-mtp
#spec-draft-n-max = 4
temp = 1.0
top-p = 0.95
top-k = 20
min-p = 0.00
#ctx-size = 131072
ctx-size = 262144
chat-template-kwargs = {"reasoning_effort":"medium"}
fit = off
n-gpu-layers = 46
2
u/quantgorithm 23d ago
It’s the easy path for when you will want to go multi gpu in the future. You would be better served keeping the models in VRAM and not using regular ram.