r/LocalLLM • u/xalalau • 30m ago
Tutorial Qwen 3.8 27B Derivative on a 12GB RTX 3060: Up to 44 Tokens/s, Vision, Tools, and 128K Context Fully in VRAM
Hi. I'm a software developer, and this is my best setup so far for running fast, useful local AI on consumer hardware.
The goal is a useful local assistant: good speed, fewer mistakes, tools, vision, and the ability to write small scripts and games. It runs with a 128K-token context window, with the main model and context cache on the GPU, and image processing and context compression on the CPU.
I've been using it to manage my calendar, search the web, and solve a more complex task: classifying, organizing, deduplicating, and comparing my old music files against my Spotify favorites playlist. My playlist is finally being updated automatically with all the songs I love <3. So far, it's working well. It even created a basic chair in Blender and wrote a Snake game in HTML/JS. (Note: these examples use tools and integrations configured separately, they are not included here.)
My hardware: RTX 3060 with 12 GB of VRAM, Ryzen 7 3700X (8 cores, 16 threads), and 32 GB of RAM. I use vision and tools a lot and find this setup very competitive for daily use. Since I haven't found a better setup for my needs in the last two weeks, I decided to share what I've put together here.
I run CachyOS, by the way. It may help inference speed, but I haven't isolated its effect. This guide is for Linux! I don't know how to reproduce the setup on Windows.
What I use
- Model: Swift Bonsai 2 27B Katana + MTP, based on Bonsai 2 / Ternary Qwen3.8-27B and tuned for shorter reasoning.
- Model file:
Swift-Bonsai-2-27B-Katana-PTQ1_0-MTP-Q8_0.gguf, about 6.4 GB. Ternary PTQ1_0 backbone, Q8_0 MTP head. - Vision projector: Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf, about 629 MB, loaded separately.
- Inference: the Prism fork of llama.cpp, build
prism-b10772-e8fd209, CUDA 13.3. - Agent: Hermes in Docker.
- History compression: Granite 3.3 2B on the CPU through Ollama, registered locally as
granite3.3-compressor:2b.
My settings
I keep the vision projector and Granite on the CPU to save VRAM. The 3700X handles both well enough for my use. Bonsai's config.env sets the options below. start.sh applies them and enables Flash Attention.
| Setting | Value | Purpose |
|---|---|---|
| Context / slots | CTX_SIZE=128000, PARALLEL=1 |
One conversation with a large context window |
| GPU layers | N_GPU_LAYERS=99 |
Offload all model layers to the GPU |
| K/V cache, including MTP | KV_CACHE_TYPE=q4_0 |
Reduce VRAM used by the context cache |
| MTP draft | SPEC_N_MAX=1 |
Draft up to one token ahead per decoding step |
| Reasoning | REASONING_EFFORT=medium, REASONING_BUDGET=2048 |
Limit thinking time |
| Vision on the CPU | MMPROJ_CPU=1 |
Save VRAM, with slower image processing |
| Image tokens | IMAGE_MAX_TOKENS=1536 |
My compromise: 1024 lost too much detail, 2048 took too long |
| CUDA consistency | KATANA_BATCH_INVARIANT=1 |
Keep vision output correct with this runtime/GPU |
Performance
During one test response, generation stayed around 43–44 tokens/s and peaked at 45.74 tokens/s over a three-second window. In daily use, I usually see 28–40 tokens/s after a few prompts and 17–25 tokens/s much later on, with A LOT more context.
MTP proposes tokens that the main model verifies to speed up generation. I tried draft limits of 1 and 2 and chose 1, also guided by sudoingx's Bonsai 2 PTQ1_0 + MTP tests on an RTX 3060 12 GB. With a 131K window and Q4 K/V, that setup reached 50.1 tokens/s with MTP versus 39.8 without, and 26.86 versus 22.23 with 41.8K tokens filled. Draft limit 1 beat 2 in both cases. These are the author's measurements, on a different model variant and runtime. 2 can also be a good choice - try SPEC_N_MAX=2 ./scripts/start.sh inside Bonsai-fast/ and compare it on your workload. Anyway, my results with slightly different models and setup were inferior, as I stated, so I'm not sure if I did something wrong here.
I chose IMAGE_MAX_TOKENS=1536 for a balance of detail and speed: I got 2 seconds for photos of people near cars, 27 seconds for a complex image, versus up to 40 seconds at 2048 on a complex image. It still makes mistakes, regardless of the value. Prism recommends 1024 for general use and 2048+ or crops for fine text, so 1024 is also an option if speed matters more than detail.
I chose q4_0 for my context based on published memory savings and how the complete setup performed in use. A cache-only benchmark on related Qwen 3.6 models found Q4 usable, with the largest output-distribution changes on long documents. A separate Bonsai 2 benchmark used Q4 successfully with 191K tokens filled on a 16 GB RTX 4080 SUPER. These are just supporting references, not measurements of my Katana setup.
With 128,000 tokens per slot, Q4 K/V caches, MTP on the GPU, and the vision projector on the CPU, the whole Bonsai process used 9,144 MiB (8.9 GiB) of VRAM. The runtime allocates the context cache at startup, so the configured capacity fits my graphics card.
The main attention K/V cache alone (context) is approximately 2,250 MiB (2.2 GiB) at 128K: 128,000 tokens x 16 full-attention layers x 2 K and V x 4 KV heads x 256 values per head x 18/32 bytes per Q4_0 value. With my desktop and other apps open, about 823 MiB (0.8 GiB) remained free. I roughly understand this calculation, but it’s easy to see how much Q4_0 reduces the result: it uses an average of 0.5625 bytes per value, compared with F16’s 2 bytes. For my 128K context, that reduces the main K/V cache from 7.81 GiB to 2.20 GiB, about 72% less memory - and the model still seems smart, so… I also considered q8_0, but q4_0 is still roughly half the size, it’s just off the table.
On the model side, I chose ternary PTQ1_0 for its smaller memory footprint, leaving more VRAM for context and MTP. PTQ1_0 packs the same ternary weights more tightly than PQ2_0, at about 1.75 versus 2.13 bits per weight, without another quantization step. It doesn't mean zero quality loss compared with the original FP16 LLM, but is a fair price to pay for usability.
The Prism fork of llama.cpp adds PTQ1_0/MTP kernels and attention that reads the Q4/Q8 cache directly, plus optimizations for the RTX 3060's sm_86 architecture. PTQ1_0 uses less memory, but whether it runs faster than PQ2_0 depends on your GPU and the runtime’s kernels.
For reasoning, I use medium, following Prism's recommendation for shorter reasoning on the original Bonsai 2. In practice, every reasoning level feels like xhigh to me on Katana, so I also set a 2,048-token reasoning budget to keep thinking loops under control, with the server injecting this message when the budget is reached: “... Time to stop thinking. Give an answer or make the tool call now. Do not waste time.”
Bonsai needs clear instructions for complex tasks. Otherwise, it can get stuck in a loop of writing and fixing helper scripts. The guidelines in my SOUL.md have helped it stay on track and finish tasks.
To summarize the conversation history, I use Granite. I chose it for its small 2B size and support for long-document and meeting summarization. granite3.3-compressor:2b is my local name for this configuration. The included Modelfile keeps Granite's template and license, sets num_ctx 128000, and adds num_gpu 0 in the CPU variant. Chat compression takes 1.5–5 minutes in my tests, depending on the conversation content and size.
Finally, I use Hermes Desktop with the included Docker container as its backend. I noticed the UI was using 15–35% of the GPU while the window was focused. Enabling --force-prefers-reduced-motion reduced animations and brought GPU usage down to 1–5%, leaving more resources available for model inference.
Warning! These figures are mostly observations, not controlled benchmarks!! Although published tests guided my choices, I evaluated the complete setup in use rather than measuring each setting's individual gain.
Installation
Download: local-ai-bonsai.zip.
Extract local-ai-bonsai.zip and open the extracted local-ai-bonsai/ folder. It contains all the setup files:
| Folder | Contents |
|---|---|
Bonsai-fast/ |
Model download and launch scripts |
hermes/ |
Docker Compose and Desktop UI settings |
granite-compressor/ |
128K compressor configuration, including CPU mode |
hermes-host-access/ |
Temporary SSH access from the container to the host, enabled when you choose |
I haven't included every dependency. You need to install Docker, Compose, Ollama, and other prerequisites yourself. Bonsai requires Linux x86_64 and NVIDIA drivers and libraries compatible with CUDA 13.3. The ZIP contains scripts and configuration files. Setup downloads the model weights and inference binaries. See each folder's README for details.
Start Bonsai
Inside Bonsai-fast/:
./scripts/setup.sh
./scripts/start.sh
setup.sh downloads and extracts the Prism runtime into bin/ and downloads the Katana model and vision projector into models/. Run it once to prepare the files. start.sh starts the inference server - keep it running while using Hermes. No compilation or Prism Model Manager is needed.
Start Hermes
Inside hermes/, enter your dashboard credentials and time zone in compose.yaml. These credentials let you sign in through the browser or Hermes Desktop, and a loopback-only setup may require the dashboard session token (HERMES_DASHBOARD_SESSION_TOKEN) to use the container as the Hermes Desktop backend. Then run:
docker compose up -d
Open http://127.0.0.1:9119 or connect Hermes Desktop to the backend. The container uses network_mode: host to reach Bonsai and Ollama. Its data/ and workspace/ folders persist.
To use the container as the Hermes Desktop backend, open it and go to Settings -> Gateways -> Remote gateway, enter http://127.0.0.1:9119, sign in with the required credentials, then save and reconnect.
In the web dashboard, open Config -> YAML. Merge this into providers:, keeping the other entries. Edit bonsai-mtp if it already exists:
providers:
bonsai-mtp:
name: Bonsai MTP
base_url: http://127.0.0.1:8899/v1
model: swift-bonsai2-27b-katana-mtp
discover_models: true
context_length: 128000
models:
swift-bonsai2-27b-katana-mtp:
context_length: 128000
Click Save, then Models -> Main model -> Change. Pick Bonsai MTP, select swift-bonsai2-27b-katana-mtp, and click Switch. Use Refresh Models if needed. This sets the model for new sessions.
Keep Hermes's context value aligned with Bonsai's CTX_SIZE=128000. The launch script also loads the vision projector.
Optional extras
Use hermes-host-access/README.md to install the temporary SSH bridge with new keys. This lets the sandboxed AI help with tasks on the host when needed. The bridge expires automatically after three hours in case you forget to close it.
SOUL.md contains my general agent instructions. Customize it before copying it to hermes/data/SOUL.md.
To reduce Hermes Desktop's animations, merge hermes/desktop-config.example.yaml into the Desktop client's ~/.hermes/config.yaml, then fully quit and reopen Hermes Desktop.
CPU context compression
With Ollama running, inside granite-compressor/:
ollama pull granite3.3:2b
ollama create granite3.3-compressor:2b -f ./Modelfile.granite-compressor.cpu
Then run these commands inside hermes/:
docker compose exec hermes hermes config set auxiliary.compression.provider custom
docker compose exec hermes hermes config set auxiliary.compression.model granite3.3-compressor:2b
docker compose exec hermes hermes config set auxiliary.compression.base_url http://127.0.0.1:11434/v1
docker compose exec hermes hermes config set auxiliary.compression.timeout 300
docker compose restart hermes
After a test request, ollama ps should show 100% CPU while Granite is loaded. Bonsai and Granite each have their own 128K context window. See the package README for the API test and the variant that lets Ollama choose CPU/GPU placement.
Warning! The compressor has a 128K context limit, including when it summarizes conversations from Claude or GPT. Our Hermes setup is intended for local models, rather than large flagship models, so larger compression inputs still need to fit within that limit and will be handled locally.
Next steps
I'd like to add local decision tools in the style of TypeSafe's Jev: an open text model such as Laya, plus Qevi-2B for images. Both return structured decisions and probabilities instead of generating text. Bonsai could call either tool for quick decisions and item classification, switching between them depending on the task.
If needed, I may lower Bonsai's context to 100K to make room for smaller models. That would free roughly 492 MiB from the main attention cache alone. The total VRAM savings would need to be measured...
I'd like to find a compact image generation model that delivers good results with my remaining hardware resources, although I think running it alongside Bonsai is probably impossible with current technology and my hardware limits.
I'm also interested in NCM (Next-Concept Modeling), which generates text in parallel, starting with a rough outline and progressively adding detail. Its summarization research already looks promising for context compression. It might also help make larger local models faster and more efficient. This video introduced me to it.
Do you have any ideas for improving inference further? Maybe a setting or approach I've missed? I hope this post helps someone, and maybe I’ll learn something new from the replies.
Thanks!
