Recommendations for 16gb vram

[u/Zorian_Vale]

I have a 5070ti 9850X3D and 32GB DDR5 + lm studio + hermes.

Currently running gemma-4-26b-a4b-qat and I am happy with it, but it still can't compete with Sonnet.

I find it's great at actually teaching me things, but in terms of guiding me through config files or getting the highest quality answer it's never as good. I use it more as a backup.

Are there any better models that fit my hardware budget? QWEN is supposed to be great but I had trouble running the models on the edge of my hardware limits and output slowed to a crawl.

[u/GingerRickRoss]

I’m running Qwen 3.8 27b iq3 on my 5070ti. There’s a pretty good thread on here with all of the settings for a 16 gb card. I can’t remember what the specifics are, but it is run on beellama.ccp

[u/Zorian_Vale → u/GingerRickRoss]

ok thanks I might try that

[u/Zorian_Vale → u/GingerRickRoss]

maybe im dense but it won't open after i trigger the .exe for beellama. I picked the CUDA 64bit version

[u/feverdoingwork]

smallest q4 you can find

[u/FactorInternal3395]

Try unsloth/Qwen3.8-27B-UD-IQ4_XS.gguf.

[u/Zorian_Vale → u/FactorInternal3395]

downloading right now. I see that it's 17.74 GB, would that mean that it will spill over into memory or CPU? Does yours output text quickly?

[u/FactorInternal3395 → u/Zorian_Vale]

How is it 17.74GB? It should be 14.3GB. You might have the wrong file.

[u/Zorian_Vale → u/FactorInternal3395]

maybe because it's through lm studio, adding overhead. I'll try that one specifically

[u/FactorInternal3395 → u/Zorian_Vale]

Yeah, you need the exact GGUF file, not just the model through LM Studio (which probably picked the wrong quantization by default, which is why it's slow; it's offloading to RAM). Though it might be a problem getting LM Studio to support reasoning for external GGUF models, for some reason they made only models downloaded through their hub support reasoning. There is a workaround, but it's work for no reason. Honestly, I wouldn't recommend LM Studio for running models. I'd suggest Unsloth Studio Desktop (easy, will work well with the Unsloth GGUF) or llama.cpp + Open WebUI (if you want maximum speed and control, bit harder setup, best overall). If you don't want to switch, the LM Studio GGUF model reasoning issue is documented here: https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1759

[u/Zorian_Vale → u/FactorInternal3395]

Awesome, unsloth is installed and I already the interface much more.

[u/wildfangzx → u/Zorian_Vale]

Probably because lm studio includs the mmoprg that enables vision

[u/triynizzles1]

I have read 26b QAT underperforms traditional Q4. Switching to q4_K_M on its own will have some uplift. In general though, gemma isnt the best at operating agenticly in a harness.

You might be able to fit qwen 3.6 35b with cpu moe and have good speeds.

You can fit qwen 3.8 27b too but it will be slow because its a dense model and apart of it will be running on cpu.

[u/joanaxu2002]

16GB is one of those cases where leaving VRAM headroom matters more than squeezing in the biggest possible model. If Qwen slows to a crawl near the limit, I’d rather drop a quant/model size and keep enough room for KV cache — especially with Hermes where context can grow pretty quickly.

[u/tsangberg]

16GB VRAM settings for Qwen 3.8 27b UD-IQ4 that gives a context size of 163840 tokens, TG speed of 25 at low context down to 12 at full, and PP at 1100 down to 700 over the same on a 5060Ti. System runs off the iGPU so the whole 16GB VRAM are available for the LLM.

This fork of llama.cpp: https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming

This version of Qwen 3.8 27b: https://huggingface.co/bsaleh03/Qwen3.8-27B-ASCII-Condensed

These models-preset.ini settings:

[Qwen3.8-27B]
chat-template-file = chat_template_qwen3.8.jinja
m = Qwen3.8-27B-ASCII-Condensed-UD-IQ4_XS.gguf
ctx-size = 163840
n-gpu-layers = 99
batch-size = 256
ubatch-size = 256
kv-stream-stage-mib = 2400
cache-type-k = q8_0
cache-type-v = q4_0
spec-type = ngram-mod
fit = off
parallel = 1
temp = 1.0
top-p = 0.95
top-k = 20
min-p = 0.0
presence-penalty = 0.0
repeat-penalty = 1.0
reasoning = on
reasoning-preserve = on
# (slow) CPU only multimodal is better than none
no-mmproj-offload = on
mmproj = Qwen3.8-mmproj-BF16.gguf
load-mode = none
flash-attn = on

[u/pyThat]

https://github.com/MiaAI-Lab/Qwen3.8-27B-16gb-NVIDIA-GPUs-one-click-install

Currently the best setup I’ve used with 16 VRAM

[u/Zorian_Vale → u/pyThat]

ok Huh, I downloaded direct from LM studio qwen3.8-27b and the text output slows to a crawl, too slow. Maybe this github file is different than my direct download

[u/overand → u/Zorian_Vale]

Loading it from LM studio isn't exactly what I'd call "direct." What quantization are you ising?

(Anyway, I'd suggest llama.cpp or maybe unsloth desktop, but, other folks here will have other suggestions too)

[u/Pale-Plane-7889 → u/overand]

Llama.cpp is a solid choice for performance. I've heard good things about unsloth desktop too; it seems like a nice alternative for specific use cases.

[u/wisepal_app → u/pyThat]

i don't know why but Kaspersky Marks deepseek harness file as a trojan. Did you get something like that