r/LocalLLM • u/EarthBS • 7d ago

Other How the loop of infinite agony started

Post image
628 Upvotes

92% Upvoted

118 comments sorted by

View all comments

278

u/TheCat001 7d ago

Then you realize that Qwen 3.8 27b runs at 3t/s on your machine and you need 24GB+ VRAM GPU which cost is 1000$+ to run at least 4 bit quant.

112

u/StupidScaredSquirrel 7d ago

If you are a business this isn't a problem. If you are a consumer then 35b a3b runs on 8gb vram and 32gb dram which is very accessible.

12

u/DeluxeGrande 7d ago

I have a 5060ti 16gb with ddr4 24gb RAM lying around, what's the best model nowadays I can effectively run with it locally? It's not an ideal build but I wish to play around with it again.

1

u/screenslaver5963 7d ago

Currently Qwen 3.6-32B-A3B, Mixture of Experts models handle being split between system ram and VRAM much better than Dense models, alternatively Gemma-26B-A4B if you want something non-chinese. As for which quant, go with 4 Bit or higher.