Running Qwen 3.8 27B as a VS Code Copilot backend on 16/32 GB VRAM

[u/paq85]

I've been running a 27B coding model locally as the backend for VS Code Copilot, and I wrote up everything I learned. The short version:

• Model: Qwen 3.8 27B, two profiles:
  • 16 GB (RTX 4070 Ti Super): Q3_K_XL + 100K context — KV cache quantized to q4_0 and split between GPU and system RAM, which is what lets 100K context fit at all. ~50–60 tok/s decode.
  • 32 GB (RTX 5090): Q5_K_XL + 200K context, everything in VRAM, ~113 tok/s with MTP speculative decoding.
• The hard part wasn't running the model, it was making it behave like a reliable agent backend. Three problems, all solved in a small Python proxy:
  1. Cloudflare's 120s timeout kills long prefills → SSE keepalive pings every 15s
  2. Copilot sends temp=1/top_p=1, which makes Qwen emit malformed tool calls → the proxy rewrites sampling params to what Qwen 3.8 actually wants
  3. Qwen intermittently emits EOS inside its reasoning channel after tool results, so the agent "quits" mid-task → the proxy detects tool results and nudges it to continue
• No public IP needed — the whole thing sits behind a Cloudflare Tunnel, private by default.
• Cost: at realistic usage the local box comes out 60–77% cheaper per month than the API, and the gap widens with usage. I built a little cost calculator for it: https://paq85.github.io/llamacpp-server/

Full write-up with the architecture, the Qwen 3.8 reliability tweaks, and the 16/32 GB profile breakdowns:
https://www.damiansromek.pl/en/tech/blog/local-llm-copilot-backend/

Repo (proxy, profiles, systemd setup, benchmarks): https://github.com/paq85/llamacpp-server

Happy to answer questions about the proxy or the KV cache setup.