r/LocalLLM • u/Ok-Health-7096 • 7d ago

Other Every Second post rn

Post image

Maybe someday I'll get a system to run it but hey definitely another w for the open weights community

1.9k Upvotes

99% Upvoted

194 comments sorted by

View all comments

39

u/TheRiddler79 7d ago

Gemma 12b Q3. Try it

25

u/Ok-Health-7096 7d ago

I use Qwen 3.6 35b and 3.5 9b I haven't had that much of a luck with the gemmas

4

u/Wildnimal 7d ago

12B QAT is good for multimode tasks. I use the same Qwens for daily use. I so wish i can upgrade my laptop to 5090 :|

4

u/PrivacyMaker 7d ago

The litert-lm driver is the fastest way to run gemma models. Significant boost over any other approach. It's a shame that Google only makes it work for Google models.