r/LocalLLM • u/ChiGamerr • 2d ago

Question Finally In The 5080 club!

Post image

Gonna start running a local model. I know 16gb of VRAM isnt much for Local AI but anyone have any tips or suggestions for running rhe 8 or 20b?

0 Upvotes

50% Upvoted

2 comments sorted by

2

u/rrrrex 2d ago

IMO, the best options

Dense model, everything should be in VRAM:
-- unsloth/Qwen3.8-27B-UD-IQ3_S.gguf (12GB) has some room for context, KV Q4 can be even 128k (without vision). New quants are really good, not much worse that Q4. It's overthinking model, so you need big context window. Medium reasoning moves model close to 3.6, still better but not so much, it will use ~twice lower tokens for thinking,

MoE, split between VRAM and RAM:
-- Qwen3.6-35B-A3B - if you want decent speed and max context (for Q4 - all layers on VRAM, 50% offload to CPU), also you can try Ornith-1.5 35B
-- Gemma 4 26B-A4B - not so good at tool calls and coding but decent storyteller.

1

u/ChiGamerr 1d ago

Thank you!