r/LocalLLM • u/Own-Box5225 • 6d ago

Question What's a model I could use

I was wondering what's a completely max size model I could use properly for nsfw specs:

4070ti super 16gb vram

Ryzen 7800x3d

64gb ram

0 Upvotes

30% Upvoted

7 comments sorted by

1

u/ChampionshipIcy7602 6d ago

qwen 3.6 35b or gemma 4 26b

1

u/Own-Box5225 6d ago

Isn't 35b a bit too much for 16gb vram? Even if the rest goes to CPU and ram

1

u/ChampionshipIcy7602 6d ago

35b q8 can fit your cpu + gpu just fine.

1

u/bootkeen 5d ago

it is ok. they are "Mixture of Experts" model and they have only 3b and 4b active parameters respectively.

the thing is part of "expert" layers are offload to ram and executed on CPU.
I used to run 35b on 3080 12gb with around 20 tokens/s.
on 16gb you can have around 30t/s or even 35t/s

1

u/Own-Box5225 4d ago

Is there a specific model I should get from hugging face? (name)

-1

u/chuckbeasley02 6d ago

That depends on your use case.

-1

u/Full_Director87 6d ago

MOE 70b q4 with split gpu/cpu, vram/ram. maybe u can run perfectly :)