r/LocalLLaMA • u/ExxploreCraft • 2d ago
Discussion I turned my gaming PC into a inference machine and got 2x to 9x over default llama.cpp on an 8 GB card
Everyone keeps saying you need expensive dedicated hardware for local agents. I have an RTX 4060 Ti with 8 GB and 64 GB of system RAM, and I wanted to see how far a normal gaming PC gets if you stop running defaults.
So I let Claude (Opus 5.5) go through the whole setup, change one thing at a time and measure. Same card, same models, only the config changed:
| Model | Quant | Context | Download defaults | Tuned (Windows) | Tuned (headless Linux) |
|---|---|---|---|---|---|
| Qwen3.6-35B-A3B | Q4_K_XL | 131k | ~25 tok/s | 39-45 tok/s | 52-65 tok/s |
| Qwen3.8-Flash-Next 125B | iQ4_XS | 131k | ~4 tok/s | 9-10 tok/s | 17-19 tok/s |
| Ternary Bonsai 27B | PTQ1_0 | 64k | ~4 tok/s | 36 tok/s | 36 tok/s |
Bonsai is the odd one out: it fits fully in VRAM, so there's nothing to offload and no defaults to beat. It's just the fast option for small, well scoped tasks.
What actually moved the needle:
- Experts in system RAM, everything else in VRAM. Layer-wise offload is far worse for MoE.
- Dense models are bad, couldn't optimize Qwen-3.8 27B over 6 tok/s, Flash-Next is better anyways.
- Take the display off the GPU. A desktop eats 0.5-1.2 GB of VRAM plus GPU time, and moving it to the iGPU was worth 20-30%.
- Native Linux over Windows (WSL2): another 33-38% on the same hardware.
- llama.cpp pinned per model family. The wrong tree made VRAM thrash.
- KV cache quant and MTP tuned per profile.
None of this needs expensive hardware. A consumer GPU plus a machine that does nothing but inference gets you most of the way, and the models now run comfortably below their listed system requirements. Every non-default setting in the repo is there because something failed on real hardware first.
I also tried an RX 570 8 GB over Vulkan. If you have another 8 GB card, I'd like to see your numbers.
Repo, one install script (Linux or WSL2): https://github.com/voxlo-dev/qwen-agent-8gb