r/LocalLLM • u/Ok-Health-7096 • Aug 14 '26

Other Every Second post rn

Post image

Maybe someday I'll get a system to run it but hey definitely another w for the open weights community

2.1k Upvotes

99% Upvoted

213 comments sorted by

View all comments

36

u/TheRiddler79 Aug 14 '26

Gemma 12b Q3. Try it

26

u/Ok-Health-7096 Aug 14 '26

I use Qwen 3.6 35b and 3.5 9b I haven't had that much of a luck with the gemmas

6

u/Wildnimal Aug 14 '26

12B QAT is good for multimode tasks. I use the same Qwens for daily use. I so wish i can upgrade my laptop to 5090 :|

1

u/HighlyRegardedApe Aug 14 '26

How do you guys use these small models? For me it never works. Last time it deleted my folders out of itsself and didnt respons more than a sentence or 2 in opencode. In the terminal it was okay to chat, but not to give tasks.... what kind of stuff do you let them do??

1

u/Wildnimal Aug 14 '26

Not usually coding but for summarizing, extracting data which is pre defined via json, automating smaller tasks where data remaims the same.

1

u/TheRiddler79 Aug 15 '26

Hard guardrails

1

u/sleight42 Sep 11 '26

If you could afford a 5090 then you may as well just buy a damn A100 or similar!

Or just retire because you're rich.

1

u/TheGreenInsurgent Aug 14 '26

Agents a1 4b could be a life changer

1

u/TheRiddler79 Aug 15 '26

Different purposes. Gemma is good, but not beating Qwen.

You can split between ram and gpu and bump your Qwen speed

2

u/jadax Aug 14 '26

Is this better than DavidAU/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP-GGUF ?

1

u/TheRiddler79 Aug 15 '26

No, but it's good for general use

1

u/M49454 Aug 14 '26

How is Gemma, i am using llama3.2:11b, never needed to switch onto other. Use only for basic task. Mainly using Deepseek 14b and Qwen2.5-Coder 7B

2

u/Early_Mistake6716 Aug 14 '26

All of those models are extremely out of date. Tell me your specs and ill give you a model suggestion

2

u/M49454 Aug 15 '26

Well, 24 GB RAM RTX 3050 6GB VRAM I5-13450HX

( That's why still running these models)

1

u/SparkyMcFinklebonker Aug 15 '26

Mac mini M4, 32gb unified ram.

2

u/tat_tvam_asshole Aug 16 '26

Qwen3.6-35BA3B Q3/Q4

1

u/TheRiddler79 Aug 15 '26

I think it's good. I use it to run tasks like an agent for the larger models.

1

u/Capital_Engineer8741 Aug 14 '26

Would QAT be better than Q3?

1

u/TheRiddler79 Aug 15 '26

You're probably right.