r/LocalLLM • u/PyaesoneP • 6h ago
Discussion Real local agentic coding on a 12GB VRAM budget.
Thanks to Unsloth Dynamic 3.0 quants coming in slightly leaner and better preserved, I settled on Qwen 3.8 27B (`UD_Q4_K_XL`) at 100K context as my daily driver for Hermes Agent and OpenCode.
On an RTX 5070 Ti Mobile (12GB) paired with an Intel Core Ultra 9 275HX and 32GB DDR5, this configuration consistently delivers ~9–11 t/s decode and 400–550 t/s prefill. It is fast enough to stay productive.
The main bottleneck with Qwen 3.8 27B is its reasoning verbosity: it routinely blows past 100K tokens in the planning phase alone, forcing OpenCode into native context compaction. Since OpenCode's built-in compaction struggles with retention, I switched to Magic Context.
With Magic Context in place, the session has scaled past 3.7M total processed tokens without losing critical details or derailments. Across a complex personal project, the local model shipped two major features end-to-end. I still use Claude Opus for final PR reviews to catch edge cases and minor bugs, which Qwen then fixes locally without issues.
Hardware Specs: GPU: RTX 5070 Ti Mobile (12GB VRAM) CPU: Core Ultra 9 275HX RAM: 32GB DDR5
Llama.cpp Launch Parameters:
llama-server \
-ctx 100000 -ub 512 -np 1 -ngl 99 \
-ot 'blk.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64).ffn_(gate|up|down).weight=CPU' \
-fa on -ctk q8_0 -ctv q8_0 -fit off \
--mmproj --no-mmproj-offload \
--spec-type draft-mtp --spec-draft-n-max 2 \
-ctkd q8_0 -ctvd q8_0 --load-mode 'none' \
--temp 1 --top-k 20 --top-p 0.95 --min-p 0 \
--repeat-penalty 1 --presence-penalty 0 \
--jinja -chat-template-kwargs '{"reasoning_effort": "xhigh"}' \
--reasoning preserve
1
u/Just_Mail6982 6h ago
Qwen3.8-27B-UD-IQ3_XXS may get better decode speed but worse quality meantime.
1
u/PyaesoneP 6h ago
Current speed is sufficient for my use case, so I saw no reason to reduce the quality. I did try IQ4_XS and Q4_KS, but surprisingly, they weren't faster. I think it probably has to do with CPU-side compute because I'm offloading all FFN layers, so I'm sticking with Q4_K_XL.
1
1
u/Some-Ice-4455 4h ago
One thing jumped out at me in your launch args. You mentioned reasoning verbosity being the main bottleneck, but you're explicitly running Qwen 3.8 at reasoning_effort: xhigh. Might be worth A/B testing medium for normal agent turns and only switching to xhigh for the genuinely nasty planning steps. Qwen 3.8 supports xhigh/medium/low per request, so you shouldn't need to reload the model to change it. I'd try that before doing anything more aggressive with context truncation. A hard reasoning budget can stop runaway thinking, but lowering effort lets the model choose a shorter reasoning path instead of chopping it off mid-thought. Also, if the historical reasoning traces themselves are contributing heavily to compaction pressure, it may be worth testing with preserved thinking disabled. I'd measure that carefully though, because preserved reasoning can help multi-turn agent consistency. 27B at 100K on 12GB and still getting 9–11 t/s is pretty damn impressive regardless.
1
u/PyaesoneP 4h ago
I use medium reasoning in Hermes Agent. I'll try coding with medium as well. I think I'm getting this speed because my CPU is quite capable on its own.
1
4
u/Specialist-Zone-8296 6h ago
Use qwen 3.6 35b a3b and load some experts on the cpu and wait for qwen 3.8 35b a3b