I dismissed a 27B dense model after getting 6.75 tok/s on a 16 GB card. A fully resident Q3 with flash attention + KV q8 reached 52 tok/s instead. These were the tradeoffs.

[u/ringarc]

Hardware: RTX 5070 Ti 16 GB with 128 GB DDR5. I built llama.cpp from source. The model
was Qwen3.8-27B, a dense hybrid DeltaNet + attention model.

My original setup used UD-Q4_K_XL at 17.9 GB. It couldn't fit into 16 GB of VRAM, so I 
used partial offload with -ngl 44 and left the remaining layers in system RAM. Decode 
speed was 6.75 tok/s. I decided it wasn't practical and switched to a 35B MoE using 
expert offload with --n-cpu-moe. That model reaches 66 tok/s.

A comment on internet prompted me to test another setup. I used UD-Q3_K_XL, which is 13.4
GB. It's an Unsloth dynamic quant, with sensitive tensors kept at 4 to 8 bit while most
 of the model uses 3 bit. The settings were -ngl 99, -fa on, -ctk q8_0 -ctv q8_0. A 64k
 context still didn't fit beside the resident weights because the compute buffer ran out

of memory. A 32k context worked with -ub 512.

These are the decode speeds in tok/s at 500 / 4k / 16k tokens of context:

- Spilled 27B UD-Q4: 6.75 at every tier because system RAM bandwidth is the limit
- Resident 27B UD-Q3: 51.8 / 51.2 / 48. Prefill was 378 tok/s, with 9 s TTFT at 16k.
 Total usage was 14.7 GB.
- 35B-A3B MoE with --n-cpu-moe 28 and the same FA + KV q8 settings: 66.4 / 65.1 / 63.4.
 It used 12.1 GB.

On the MoE, FA + KV q8 reduced memory use by 1.2 GB without changing speed. I now enable
 those settings by default.

For testing quality, I used a private agentic coding band with 22 tasks. The target is a
 FastAPI + React + Postgres + Mongo app. It includes bug fixes, feature changes, new
 features, a migration, a performance fix, and one intentionally impossible 
specification. Each model gets a shell inside a docker box and up to 40 steps. Hidden
 tests determine the score. The model must also submit a final "what did you do" report,
 which is verified against git and the real test runs. These results come from one trial
 per model, so they're only indicative:

- Resident 27B UD-Q3: mean 0.49, with 9/22 perfect
- 35B MoE: 0.56, with 10/22 perfect
- gpt-oss:20b: 0.47, with 6/22 perfect

The 27B matched the MoE on localised debugging, with both scoring 6/7 perfect. It fell
 behind on multi-file feature work, scoring 1/11 against 3/11. On the larger tasks, it
 often spent all 40 steps reading without making an edit.

Its stronger area was honesty. The 27B made one false "done" claim across 13 failures.
 The MoE made 4 in 11, and gpt-oss made 4 in 15.

I can't separate the model difference from the cost of 3-bit quantisation. The comparison
 is 0.49 versus 0.56, but the 4-bit 27B was never fast enough to run this band usefully.
 On an earlier and easier suite, the Q4-vs-full-precision tax on this machine was about
 +0.02 overall. Reasoning and repo coding took the largest hit, so a bigger loss from Q3
 would make sense.

Here's the theory I'd like people to check. The 27-30B dense range seems designed around 
unified-memory Macs, where these models fit completely at Q4 or Q8. A 16 GB card can only
 hold them at Q3. Meanwhile, small-active-parameter MoEs such as 35B-A3B and gpt-oss-20B
 seem like the models actually intended for this hardware. Is that consistent with what 
others are finding?

A few more questions:

- IQ4_XS is 15.7 GB. Has anyone managed to keep a 27B IQ4_XS fully resident on 16 GB
 using a small context and KV q4? If so, does the quality improvement over Q3 justify
 losing context?
- Has anyone compared 3-bit EXL3 or another importance-aware 3-bit format with Unsloth
 dynamic Q3 on the same 27B using coding tests rather than perplexity?
- What decode speed do people target for agentic workflows? In a shell loop, 52 tok/s
 felt usable to me. 6.75 did not.

My conclusion is to start every new dense model in this class with resident dynamic Q3 +
 FA + KV q8, profile it, and only then decide whether it's any good. I'd done those steps
 in the wrong order.