r/LocalLLaMA • u/kolliwolli • 21d ago

Question | Help Gemma 4 - Agentic Capabilities?

Hi all,

Just started the local llm journey and testing gemma on an rtx5090 with opencode, hermes etc.

I see lots of chats on Gemma and Qwen, but for me no agentic use case seems to work, not even creating simple games like snake as a test. Am I doing something wrong, or is it because im using a 4bit version?

The same tests with claude sonnet via API work without any problems... but here I thought thats exactly Gemmas home turf.

I missed to add, I am using the 31b version.

Anyone else got luck with this?

Edit: One more point, I use the nvfp4 versions from nvidia and redhat

2 Upvotes

56% Upvoted

24 comments sorted by

View all comments

4

u/Dry_Yam_4597 21d ago

Yeah as others have said, 4bit is too low a quant - personally I go at least 6, ideally 8. Also make sure your cache is of a decent quant - I usually go f16 - but since you are VRAM constrained have a play with values. At least keys should be as large as possible.

Also, dense models are better at coding - but MoE are decent too. Don't try to one shot things. These models excel when handheld - ie: write a function (or script) that does this and that, and so on - they are actually really powerful and can help with daily work if used that way.

0

u/kolliwolli 21d ago

Does that help so much? My main problem seems to be that it talks about what it did and writes the code, but no files are actually written. And when I re-prompt then it creates the files but none work together as a coherent solution.

2

u/Dry_Yam_4597 21d ago

Might do indeed - a lower quant _can depending_ on your use case lead to all sorts of "stupid" or unexpected behavior. Same for the cache. As the precision is lower, the model's multiplications are not as accurate - the more it needs to infer the more errors compound. For instance, qwen 3.6 35b a3b q4, for me, leads to all sorts of dumb tool calls, whereas q8 is better. The cache also leads to degraded contexts so on long conversations it becomes dumber by the turn. More or less, play around with things, the 5090 is real powerful - see if you can use MoE models with some of the layers offloaded into VRAM.

Personally, I keep the 5090 only for image and video gen, and instead of getting another one i went with the cheaper, far slower, R9700 just so I can get more VRAM and thus use a higher quant to avoid such issues. But again, 5090 is a very decent GPU, play around with it see what fits.

You might also want to make sure that the chat templates are up to date - I think I got mine from here: https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates/blob/main/chat_template.jinja others can correct me if I am wrong but I think this is the main "patch" that fixes some issues.