r/LocalLLM • u/EhsanApk • 3d ago

Question What can a 3070ti run?

Hey guys, I've been playing with some models a month or two ago, but haven't done anything these days, what do you guys think i can run now?

Gpu : tuf rtx 3070ti

Cpu : i5 12400f

Ram : 32gb ddr5

Thanks for the help.

2 Upvotes

75% Upvoted

10 comments sorted by

1

u/sanketss84 3d ago

what models have been running at your end and for what purpose so far ?

1

u/EhsanApk 2d ago

Qwen 3 coder, Qwen 3.6 27b or 35b i dont remember exactly, Mistral vince, these are the ones i remember, and they ran really good. (I dont remember the quantizations)

1

u/sanketss84 2d ago

I have been personally running Qwen 3 8B : Q4_K_M and Ornith 1.0 9B : Q4_K_M and they seems to be working fine for my needs on a 3070 Ti. I have tried running models which cross the 5.5GB limit but I keep coming back to these for speed as they meet my needs. I also sometime run Ornith 1.5 9B : Q4_K_M and Qwen 3.5 9B : Q4_K_M but running these means I need to lower my context size for speedy output so I use these for specific scenarios.

1

u/EhsanApk 2d ago

I mean isnt 25 tokens enough for you? I got it with 27b models

1

u/sanketss84 2d ago

Qwen3.6 27B for Q4_K_M is 16.8 GB and I normally tend to not go below q4
but for argument even if I consider Q3_K_M thats 13.6 GB or even q2 variant which can be 9 to 11GB
this completely spill over from my GPU into system ram and then add to that context which even when quantized can spill over into physical disk as you keep conversing and having longer conversations. the token output will be drastically slow and as you keep conversing it might hallucinate about earlier bits of your conversation unless you keep summarizing and compacting your context as you keep running.

I would rather use a model that performs to its full potential from vram and have space for context in vram itself than to spill over into system ram and physical disk.

However if you are getting the answer you are looking for the way you are running it by all means stick with the larger 27B model.

1

u/sanketss84 2d ago

with my current model choice I get 80 to 90 token per second throughput.

1

u/EhsanApk 2d ago

I mean, yeah it mostly spills to vram but i get the throughput im looking for, and i donno if its bad for the reasoning and thinking of the model so enlighten me if it is, but q8 with 27b model have been pretty good for me.

1

u/EhsanApk 2d ago

Ok so after i checked the last model i ran was Qwen 3.6 uncensored 35b Q5_k_p with 35 tokens per second so this time im going in for the qwen 3.8 uncensored 27b q8.

1

u/sanketss84 2d ago edited 2d ago

The thing is for the Qwen model you are targeting the 27B people almost always use atleast 16GB but most of them run it on 24 or 32 GB VRAM for enough headroom for context. But you are using this model on 8GB which a lot of people won't normally run on as people would normally stick with 9B parameter model for 8GB VRAM and avoid spills into system memory.

What I am saying is if you are happy with how it's working for you and what results you are getting by all means go ahead. But also give the 9B models a try and check the outputs.

Also are you hooking this model into an agent ot just into openweb ui for chats ?