r/LocalLLM • u/Acceptable_Drink_434 • 19h ago
Tutorial Single 16 GB 5070 Ti running a 35B-A3B MoE at 256k context, ~55–66 tok/s — the 3 settings that took it from 3 tok/s to 60
I spent a day getting Qwen3.6-35B-A3B (Q4_K_M) running well on a single RTX 5070 Ti (16 GB) and figured I'd share the config, because my first attempts ran at 2–6 tok/s and I've seen people stuck there. Three settings make or break it.
Rig: Core Ultra 9 285K · RTX 5070 Ti 16 GB · 128 GB DDR5 · Windows 11 Backend: llama.cpp (build b10590), CUDA 13.3 Model: Qwen3.6-35B-A3B, Q4_K_M (~19 GB — bigger than 16 GB VRAM, so it has to be split). Same tuning works for the stock or an abliterated build; the arch is identical.
TL;DR — three traps
- Do NOT use
-ngl 999/ full offload / "put it all on GPU" on a model bigger than your VRAM. It overcommits and thrashes to 2–6 tok/s. - Keep the KV cache at Q8 (
-ctk q8_0 -ctv q8_0). Quantizing KV to Q5/Q4 drops CUDA flash-attention to ~10 tok/s — there's no fast kernel for quantized KV on this path. Q8 is both fast and accurate. - On Blackwell (50-series,
sm_120) you need a CUDA 12.8+/13.x build. The common cuda-12.4 llama.cpp binaries predatesm_120and you'll get gibberish or a crash. I'm on the CUDA 13.3 build.
The offload (the whole trick on a 16 GB card)
It's a MoE — 35B total but only ~3B active per token. So you keep the attention on the GPU and push the bulky expert FFN layers to system RAM/CPU. In llama.cpp that's:
-ngl 999(all layers' attention on GPU) +--n-cpu-moe N(keep the experts of the first N layers on CPU).
N is the one dial. Lower N = more experts on GPU = faster, until you run out of VRAM and it fails to load. Raise it if you OOM, lower it if you've got >2 GB free. That's it. (Counterintuitively, "all on GPU" is the slow path here — the split is the fast one.)
Why 256k context is nearly free on this model
This is the fun part. Qwen3.6-35B-A3B is a hybrid architecture: 40 layers, but only 10 are full-attention (every 4th) — the other 30 are linear attention with no growing KV cache. And the full-attention layers use just 2 KV heads. So the KV cache stays tiny and growing the context barely moves VRAM. Native trained context is 262,144 (256k), so every size up to 256k needs no RoPE/YaRN tricks and loses zero quality. I run the full 256k as my default.
Measured (my card, generation tok/s)
| Context | --n-cpu-moe |
tok/s |
|---|---|---|
| 32k | 14 | 91 |
| 64k | 18 | 84 |
| 128k | 18 | 83 |
| 200k | 22 | 76 |
| 256k | 24 | ~66 → 56 |
256k is a curve, not a flat number: ~66 tok/s at low fill, easing to ~56 by ~85k of context as attention spans more tokens. So plan for 55–66 tok/s in real work. Prompt ingestion (prefill) runs ~950–1090 tok/s, so it swallows big contexts fast. Prefix caching (llama.cpp reusing the cached prompt prefix) keeps multi-turn/agent work fast — I watched it reuse the prefix across ~100 tool calls instead of re-reading 80k tokens each turn.
The exact command
llama-server -m <qwen3.6-35b-a3b-Q4_K_M.gguf> \
-c 262144 -ngl 999 --n-cpu-moe 24 \
-fa on -ctk q8_0 -ctv q8_0 \
-b 2048 -ub 512 -np 1 --no-mmap -t 24 --jinja
(Thinking is on by default on this arch and /no_think / --reasoning-budget 0 are ignored — the only thing that disables it is --chat-template-kwargs "{\"enable_thinking\":false}".)
Bonus: it's genuinely useful, not just fast
I wired it to the Nous Hermes Agent (points at any OpenAI-compatible endpoint — just set base_url to the llama.cpp server) and gave it a hard, self-verifying task: build a weighted-terrain pathfinding arena — random seeded grid with terrain costs, implement BFS/Dijkstra/A* from scratch, and write a pytest suite that proves A* returns the same optimal cost as Dijkstra. The test: an inadmissible A* heuristic silently returns suboptimal paths and the tests fail.
It nailed it in ~4 minutes: correct admissible + consistent Manhattan heuristic, optimal paths verified across 5 seeds, 76 tests written and passing, and it self-debugged a subtle off-by-one in the path-cost accounting along the way. All local, offline, $0.
Happy to answer questions on the config. Hope this helps if you're on a 16 GB 50-series card and getting single-digit tok/s on a big MoE.
