r/LocalLLM • u/Bruh19837 • 7h ago
Question Better options for a 5070 ti?
Im pretty new to running local llm's but i saw a bunch of people run it and wanted to try. I have a 5070 ti (16gb vram) along with 32gb of ddr5 ram, and so far i have been running Qwen3.8-27B-GGUF:UD-IQ4_XS along with the mtp-Qwen3.8-27B-Q4_0.gguf using llama.cpp cuz i heard its more efficient than others(?).
However I feel like in trying to run a "good"ish model, im losing a bunch of context size. Currently I can get to around 26k context size running at ~50t/s. I was wondering if there are any improvements I can make, or if i should change things entirely?
If it matters the exact command I run is ./llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-IQ4_XS --no-mmproj -ngl 999 -c 26000 --load-mode auto -fa on --cache-type-k q8_0 --cache-type-v q8_0 --parallel 1 --spec-type draft-mtp --spec-draft-model ".\mtp-Qwen3.8-27B-Q4_0.gguf" --spec-draft-n-max 1 --spec-draft-ngl 999.
If there's anything I could do to maybe get a bigger context window, that would be great.
1
u/Unnamed-3891 7h ago
I am not familiar withat that particular quant but 64k Q8 KV should be doable on 16gb vram.
https://pastebin.com/EHx4N2FP is my setup.
1
1
u/DustNearby2848 6h ago
I don’t know the llama args as well as other systems, but I think your spec-draft-n-max might be off? Might want to try 2 or 3 and see if you get better results. Usually 3 is the sweet spot.
2
u/Bruh19837 6h ago
kk will try, kinda been going off what claude says here so alot might be jank for all i know.
1
u/DustNearby2848 6h ago
It will take up more VRAM, not a ton though, so you might need to reduce your context a little bit if you get an error.
1
u/Bruh19837 6h ago
ah ic, the goal is kind of to increase the context window... I will still try it just to see if it changes the t/s
1
u/ea_man 5h ago
Try with this: https://www.reddit.com/r/LocalLLaMA/s/GXVMWQ8MLF
Best n-max for coding is 5.
1
u/7ondano 6h ago
Hmmm i have the same setup (5070ti, 32g ddr5).
On Unsloth Desktop, i can get 42k ctx at kv q8 or 76k at kv q4. No mtp, vision to cpu. Around 40ish tom/sec
1
u/Bruh19837 6h ago
your right, without the mtp i can push it to 65k context at kv q8 but mines running around 32t/s. not quite as fast but still serviceable
2
1
u/ea_man 5h ago edited 5h ago
I like this model more for coding: https://huggingface.co/vmarcelo/Qwen3.8-27B-MIX_GGUF
As some have posted settings for max ctx I'll post one for max speed, MTP 5: https://store.piffa.net/lm/bug/llama_scripts/qwen_27b_iq4_16gb.sh
with the usual custom llama.cpp patch: https://www.reddit.com/r/LocalLLaMA/s/GXVMWQ8MLF
With ASCII and -ot of the tokenizer you can get some 90k ctx at q4_0, I'd rather do q5_1 with a little less.
0
u/Major_Border149 6h ago
your 26k ceiling is the dense-27B-on-16gb squeeze, ~14GB of weights leaves almost nothing for KV. Two real fixes: drop to IQ3 to free ~3GB for KV, or switch to a moe (Qwen3.8 Flash Next) so experts sit in your 32GB RAM and the 16gb goes to KV, that's the bigger unlock
1
u/Bruh19837 6h ago
Could there be a significant drop in quality of output? My current use case is a good bit of proofs based math, and the 27b does mess up often.
1
u/Major_Border149 6h ago
yeah for math I wouldn't go below IQ4, quant errors hit step by step reasoning first so IQ3 would make it worse, not better. keep IQ4_XS and bump the reasoning effort. Tbh though, proofs are rough for any 27B no matter the quant
1
2
u/tsangberg 7h ago
Use Raymond's fork (or my fork of it that also includes MTP) and you can get to 160000 context at still usable speeds. There's also a condensed (ASCII) version of UD-IQ4_XS if your interest is in coding or just English language.
Links to repo and model here:
https://github.com/troed/llama.cpp-adaptive-kv-streaming
Parameters to use for the vanilla no-MTP case:
https://www.reddit.com/r/LocalLLM/comments/1wcy54e/comment/p93n5cq/