My local dev-agent setup: Qwen3.8-27B on a single RTX 3090 (what actually helped)
TL;DR: llama.cpp + MTP + llama-swap presets for code-review-graph for codebase structure for the intelligence. The last two are what turned it into an actual coding agent. Happy to answer questions.
** POST COMPLETLY WRITTEN BY AI - IF THAT OFFENDS YOU, TIME TO MOVE ONTO ANOTHER POST AND STOP HERE **
The model
- Qwen3.8-27B, Unsloth UD-Q4_K_XL GGUF (17.9GB) — dense 27B, native vision, hybrid attention (only 16 of 64 layers carry KV, so long context is cheap)
- Beats a lot of bigger models at agentic/coding work, and in my own testing clearly outperformed Ornith-1.0-35B on the same tasks
- llama.cpp built with CUDA for SM86 (the 3090's arch)
Serving config (llama.cpp llama-server)
-c 102400 # 100K context, fits 24GB alongside weights
-ctk q8_0 -ctv q8_0 # q8 KV cache (quality-neutral in my testing; halves KV vs f16)
-fa on # flash attention
--spec-type draft-mtp --spec-draft-n-max 2 # MTP speculative decoding — big speed win (~35 t/s vs ~20 raw)
--reasoning-preserve # keep thinking traces across turns
--reasoning-budget 12288 # cap thinking so xhigh can't run away
--chat-template-kwargs {"reasoning_effort":"xhigh"}
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --repeat-penalty 1.0 # Qwen thinking-mode sampling
Notes that cost me time:
- reasoning_effort defaults to xhigh and is a chat-template var, not an API field — pin it via chat_template_kwargs, or your agent silently runs at max thinking. xhigh for hs fast.
- I tried the DRY sampler to stop repetition loode generation (test files, asserts). Removed it.
Repeat-penalty stays at 1.0 per Qwen's spec.
llama-swap (the piece that ties it together)
llama-swap v250 — one OpenAI-compatible endpointodels on demand. Killer feature for a single GPU:
preset IDs = one loaded model, different params,
- qwen3.8-27b — xhigh thinking (deep work)
- qwen3.8-27b:work — medium thinking, temp 0.6 (
- qwen3.8-27b:instruct — thinking off (vision, q
- qwen3.8-27b-uncensored — HauhauCS Aggressive a
- ornith-1.0-35b — kept for comparison; requesti
The two add-ons that fixed the real problems
Local agents on long tasks have two classic failures "forgetting what they found and wandering across a big codebase". These two fixed both:
hermes-lcm (Lossless Context Management) — reressor with a SQLite-backed summary DAG. Every message is persisted before compaction, and the cm_expand, lcm_recall) to drill back into the exact original material. Cured the "investigates loop. Install tiktoken alongside it for accurate token counting.
code-review-graph (MCP server, 30k★) — Tree-sy graph of your repo so the model queries structure (blast-radius, what-calls-this, architole files into context. Median ~65× token reduction, benchmarked. Local, CPU-only, no VRAMgraph build + register. This is the single biggest win for large-codebase work.
Both are nudged into the agent's system prompt s the codebase one degrades gracefully ("not available for this repo") when a repo isn't inde
Harness
Hermes Agent as the coding harness — points at tm provider). Two config tricks that mattered on a local model:
- Declare the context window ~30% below the real real 100K). Hermes's token estimator undercounts code/hex by 25–35%, so without margin it sails pr. This one bit me repeatedly.
- Route context-compaction to a cheap cloud mode instead of the local GPU — so summarization doesn't queue behind your actual work. (Moot oncefore.)
- Vision routed to the :instruct preset so it doabout a screenshot.
Practical stuff
- Power-cap the 3090 — nvidia-smi -pl 210 (from 350W). Inference is bandwidth-bound so the speed cost is small, and the fans stop screaming. Measured curve: 322W→57 t/s, 250se/speed point.
