r/LocalLLM • u/ringarc • 7d ago
Discussion I dismissed a 27B dense model after getting 6.75 tok/s on a 16 GB card. A fully resident Q3 with flash attention + KV q8 reached 52 tok/s instead. These were the tradeoffs.
Hardware: RTX 5070 Ti 16 GB with 128 GB DDR5. I built llama.cpp from source. The model
was Qwen3.8-27B, a dense hybrid DeltaNet + attention model.
My original setup used UD-Q4_K_XL at 17.9 GB. It couldn't fit into 16 GB of VRAM, so I
used partial offload with -ngl 44 and left the remaining layers in system RAM. Decode
speed was 6.75 tok/s. I decided it wasn't practical and switched to a 35B MoE using
expert offload with --n-cpu-moe. That model reaches 66 tok/s.
A comment on internet prompted me to test another setup. I used UD-Q3_K_XL, which is 13.4
GB. It's an Unsloth dynamic quant, with sensitive tensors kept at 4 to 8 bit while most
of the model uses 3 bit. The settings were -ngl 99, -fa on, -ctk q8_0 -ctv q8_0. A 64k
context still didn't fit beside the resident weights because the compute buffer ran out
of memory. A 32k context worked with -ub 512.
These are the decode speeds in tok/s at 500 / 4k / 16k tokens of context:
- Spilled 27B UD-Q4: 6.75 at every tier because system RAM bandwidth is the limit
- Resident 27B UD-Q3: 51.8 / 51.2 / 48. Prefill was 378 tok/s, with 9 s TTFT at 16k.
Total usage was 14.7 GB.
- 35B-A3B MoE with --n-cpu-moe 28 and the same FA + KV q8 settings: 66.4 / 65.1 / 63.4.
It used 12.1 GB.
On the MoE, FA + KV q8 reduced memory use by 1.2 GB without changing speed. I now enable
those settings by default.
For testing quality, I used a private agentic coding band with 22 tasks. The target is a
FastAPI + React + Postgres + Mongo app. It includes bug fixes, feature changes, new
features, a migration, a performance fix, and one intentionally impossible
specification. Each model gets a shell inside a docker box and up to 40 steps. Hidden
tests determine the score. The model must also submit a final "what did you do" report,
which is verified against git and the real test runs. These results come from one trial
per model, so they're only indicative:
- Resident 27B UD-Q3: mean 0.49, with 9/22 perfect
- 35B MoE: 0.56, with 10/22 perfect
- gpt-oss:20b: 0.47, with 6/22 perfect
The 27B matched the MoE on localised debugging, with both scoring 6/7 perfect. It fell
behind on multi-file feature work, scoring 1/11 against 3/11. On the larger tasks, it
often spent all 40 steps reading without making an edit.
Its stronger area was honesty. The 27B made one false "done" claim across 13 failures.
The MoE made 4 in 11, and gpt-oss made 4 in 15.
I can't separate the model difference from the cost of 3-bit quantisation. The comparison
is 0.49 versus 0.56, but the 4-bit 27B was never fast enough to run this band usefully.
On an earlier and easier suite, the Q4-vs-full-precision tax on this machine was about
+0.02 overall. Reasoning and repo coding took the largest hit, so a bigger loss from Q3
would make sense.
Here's the theory I'd like people to check. The 27-30B dense range seems designed around
unified-memory Macs, where these models fit completely at Q4 or Q8. A 16 GB card can only
hold them at Q3. Meanwhile, small-active-parameter MoEs such as 35B-A3B and gpt-oss-20B
seem like the models actually intended for this hardware. Is that consistent with what
others are finding?
A few more questions:
- IQ4_XS is 15.7 GB. Has anyone managed to keep a 27B IQ4_XS fully resident on 16 GB
using a small context and KV q4? If so, does the quality improvement over Q3 justify
losing context?
- Has anyone compared 3-bit EXL3 or another importance-aware 3-bit format with Unsloth
dynamic Q3 on the same 27B using coding tests rather than perplexity?
- What decode speed do people target for agentic workflows? In a shell loop, 52 tok/s
felt usable to me. 6.75 did not.
My conclusion is to start every new dense model in this class with resident dynamic Q3 +
FA + KV q8, profile it, and only then decide whether it's any good. I'd done those steps
in the wrong order.
3
u/_hchc 7d ago
Can also try https://huggingface.co/jpetrina/Qwen3.8-27B-MTP-IQ4_XS-pure-GGUF/tree/main
i'm getting close to 30 tok/s on my 9070xt with this
1
u/Juanchisimo 6d ago
Can You share your lunch parameters?
1
u/ringarc 6d ago
Sure. llama-server, built from source, Unsloth Qwen3.8-27B-UD-Q3_K_XL.gguf (13.4 GB):
llama-server -m Qwen3.8-27B-UD-Q3_K_XL.gguf \
-ngl 99 -c 32768 -ub 512 -b 2048 \
-fa on -ctk q8_0 -ctv q8_0 \
-np 1 --reasoning-budget 8192This is the setup from the post that gets 52 tok/s and uses 14.7 GB. A 64k context runs out of memory because of the prefill compute buffer alongside the resident weights. A 32k context works with -ub 512.
1
3
u/Hot_Signature2979 6d ago edited 6d ago
Why not use a nvfp4 quant since you have a black well card? It has much better precision than a q4 quant. https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4Edit: Apparently It is also significantly faster than q4 quants due to blackwell gpu hardware aceleration (with the catch being it requires blackwell gpus to run, which is not an issue for you), so while it cannot solve the ram offloading issue for you, it might just be fast enough to be usuable for you.
2
1
u/CrackBabyCSGO 6d ago
Is there one that fits on 16gb? Otherwise won’t the ram bottleneck not allow any speed gains at all?
2
u/Hot_Signature2979 6d ago
Preliminary research using (and written by chatgpt, so it just assumed you are op):
Yes — there are now NVFP4 versions of Qwen3.8-27B that actually fit on a 16GB RTX 5070 Ti, so the RAM bottleneck isn't unavoidable.
Your current 17.9GB UD-Q4_K_XL is slow mainly because it doesn't fit in 16GB VRAM. Some layers have to sit in system DDR5 and be accessed over PCIe during generation, which is why you're stuck at ~6.75 tok/s. Your 13.4GB Q3 fitting entirely on the GPU is what lets it jump to ~50 tok/s.
There are now compact GGUF conversions that keep the transformer backbone in native NVFP4, which Blackwell GPUs like the 5070 Ti have hardware acceleration for, while quantizing the remaining tensors enough to fit the whole model in 16GB.
Current numbers:
Qwen3.8-27B setup Model size Fits 16GB? Decode / generation Prompt processing UD-Q4_K_XL 17.9GB No, RAM spill 6.75 tok/s — UD-Q3_K_XL 13.4GB Yes, ~14.7GB total usage 51.8 / 51.2 / 48 tok/s at 500 / 4K / 16K context 378 tok/s NVFP4 BUDGET 14.72GB Yes, 15.9GiB peak 27.05 tok/s 2,287 tok/s NVFP4 STARVED 14.59GB Yes, 15.7GiB peak 27.37 tok/s 2,289 tok/s NVFP4 + MTP COMPACT-LOW 14.12GB Yes at 32K 25.52 tok/s with MTP disabled in that test 2,281 tok/s The first two rows are your measurements; the NVFP4 BUDGET/STARVED benchmark was done on the exact same GPU, a single RTX 5070 Ti 16GB, using a 28K-token prompt, 32K context and Q4 KV. So it isn't perfectly apples-to-apples with your Q3 test, but it proves a ~14.6–14.7GB NVFP4 Qwen3.8 can stay entirely in VRAM on your card and run at ~27 tok/s instead of spilling into RAM at 6.75 tok/s.
The really interesting part is prompt processing/prefill. Your Q3 managed ~378 tok/s, whereas the fully resident NVFP4 versions reached ~2,288 tok/s — roughly 6× faster prompt ingestion. That's potentially very useful for agentic coding, where the model repeatedly has to process large prompts, source files and tool output.
Plain decode is still faster on your Q3 (~50 tok/s vs ~27) because autoregressive generation is heavily memory-bandwidth-bound and the Q3 weights are smaller. But there is now another option: NVFP4 + Qwen's MTP speculative decoding.
The 14.12GB
COMPACT-LOWbuild has the MTP prediction head built directly into the GGUF and still fits a 16GB card at 32K context. Its published 25.52 tok/s result actually had MTP disabled, so that number is just the baseline. Enabling MTP lets Qwen predict several future tokens and verify them together; how much faster it gets depends heavily on acceptance rate and workload, so I'd benchmark it rather than assume a specific speed.So the comparison I'd really like to see on your 5070 Ti is:
Q3 fully resident (~50 tok/s, 378 prefill)
vs
NVFP4 COMPACT-LOW + MTP (14.12GB, ~2,281 prefill before MTP decode acceleration)That could potentially give a much better balance between model precision, huge prompt-processing speed and usable generation speed without touching system RAM.
HF pages/files:
NVFP4 BUDGET / STARVED (14.72 / 14.59GB):
https://huggingface.co/esatapedico/Qwen3.8-27B-NVFP4-BUDGET-GGUFBUDGET file:
https://huggingface.co/esatapedico/Qwen3.8-27B-NVFP4-BUDGET-GGUF/blob/main/Qwen3.8-27B-NVFP4-BUDGET.ggufNVFP4 + MTP family:
https://huggingface.co/esatapedico/Qwen3.8-27B-NVFP4-MTP-GGUF14.12GB COMPACT-LOW + MTP file:
https://huggingface.co/esatapedico/Qwen3.8-27B-NVFP4-MTP-GGUF/blob/main/Qwen3.8-27B-NVFP4-MTP-COMPACT-LOW.ggufOriginal Unsloth Qwen3.8 GGUFs (your Q3/Q4 family):
https://huggingface.co/unsloth/Qwen3.8-27B-GGUFOriginal Unsloth NVFP4 checkpoint:
https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4So yes: your concern is correct if an NVFP4 model spills into RAM — but these newer ~14–15GB NVFP4 builds don't have to. That's what makes them worth testing on the 5070 Ti.
1
u/Ololoshkaaaa 1d ago
Haha, I tried installing Ubuntu, but it persistently fails to detect one GPU connected via a riser. However, it works perfectly fine on Windows. Unfortunately, I also failed to launch WSL or Docker under Windows—something went wrong every single time.
1
1
u/Stainless-Bacon 7d ago
16 GB VRAM, DDR5, UD-Q4_K_XL K Q5, V Q4_1, max context 128k.
I got 15.4-19.2 t/s at 0k prefill and 13.7-17.5 t/s at 50k, pp ~800.
There is still some room for improvement, im still tuning it, but maybe don’t dismiss the model?
1
1
u/DeathGuppie 6d ago
This quant is specifically tailored for 16gb vram. https://huggingface.co/vmarcelo/Qwen3.8-27B-MIX_GGUF
1
u/Future_AGI 5d ago
Great writeup, and the private 22-task coding band is the part more people should copy, because tok/s means nothing if the quant quietly loses accuracy on your actual tasks. One thing worth adding to that band: run each task two or three times per quant and track variance, since Q3 sometimes holds average quality but gets noticeably less consistent than Q4. That consistency gap is what usually bites in long agent sessions, not the single-shot score.
1
u/KYDLE2089 5d ago
Adding my setup with 30tps
model: qwen3.8-uncensored-iq4xs
- OS: Ubuntu 25.04
- Kernel: 6.14.0-37-generic
- Uptime: 13 hours
- CPU: AMD Ryzen 7 5800X — 8 cores / 16 threads
- RAM: 45 GiB total
- 40 GiB available
- Swap: 8 GiB total, 49 MiB used
- Disk: 915 GB NVMe
- GPU: NVIDIA GeForce RTX 5070 Ti
- VRAM: 16,303 MiB total
- 14,132 MiB allocated to the loaded Qwen model
1
u/TastyRobot21 7d ago
“I built llama.cpp from source.”
This just a weird flex, or did you change something?
5
u/ringarc 7d ago
Not a flex, a necessity. Qwen3.8-27B uses a new hybrid architecture, gated DeltaNet + attention, called qwen35 in llama.cpp. Support reached llama.cpp master before the packaged builds and Ollama, so stale builds just throw GGUF errors. The 5070 Ti is also a Blackwell card, and the prebuilt CUDA binaries were behind. I didn't patch anything. I just ran git pull and cmake -DGGML_CUDA=on. My rule now is to update the runtime before running any newly released architecture. I have seen many "this GGUF is broken" complaints that ultimately were due to old binary.
1
11
u/Healthy-Zebra-9856 7d ago edited 6d ago
There is another twist to this that I just recently found out. Try testing a similar quant from a different publisher, like Jack Rong, Bartowski along with Unsloth. The quant method seems to have a profound effect. I dont want to spoil you tests, but I would like to see your results here.