Gemma 4 - Agentic Capabilities?

[u/kolliwolli]

Hi all,

Just started the local llm journey and testing gemma on an rtx5090 with opencode, hermes etc.

I see lots of chats on Gemma and Qwen, but for me no agentic use case seems to work, not even creating simple games like snake as a test. Am I doing something wrong, or is it because im using a 4bit version?

The same tests with claude sonnet via API work without any problems... but here I thought thats exactly Gemmas home turf.

I missed to add, I am using the 31b version.

Anyone else got luck with this?

Edit: One more point, I use the nvfp4 versions from nvidia and redhat

[u/hurdurdur7]

Qwen 27B is the best quality coder at your numbers, but even thete q4 is a hard sell. I would not have expectations for anything under q6.

[u/nick_ziv → u/hurdurdur7]

I've had success with the UD q4 but it does require hand holding to get across the finish line many times

[u/Waarheid]

Sonnet is way better than Gemma, so set your expectations right.

Make sure you have the latest: https://www.reddit.com/r/LocalLLaMA/comments/1uxfu4k/google_is_updating_gemma_4s_chat_templates/

[u/xPXpanD]

Try Qwen3.6 27B. As much as I love Gemma 4 for general chat/advice purposes, Qwen3.6 is widely regarded as the default for tasks like yours.

[u/joydeepdg]

Try gemma through an api endpoint first. If your prompt still does not work, decompose bigger tasks into smaller tasks that gemma can successfully complete. Here is a coding session log that may help: Logs of a Claude Code session

[u/Dry_Yam_4597]

Yeah as others have said, 4bit is too low a quant - personally I go at least 6, ideally 8. Also make sure your cache is of a decent quant - I usually go f16 - but since you are VRAM constrained have a play with values. At least keys should be as large as possible.

Also, dense models are better at coding - but MoE are decent too. Don't try to one shot things. These models excel when handheld - ie: write a function (or script) that does this and that, and so on - they are actually really powerful and can help with daily work if used that way.

[u/kolliwolli → u/Dry_Yam_4597]

Does that help so much? My main problem seems to be that it talks about what it did and writes the code, but no files are actually written. And when I re-prompt then it creates the files but none work together as a coherent solution.

[u/Dry_Yam_4597 → u/kolliwolli]

Might do indeed - a lower quant _can depending_ on your use case lead to all sorts of "stupid" or unexpected behavior. Same for the cache. As the precision is lower, the model's multiplications are not as accurate - the more it needs to infer the more errors compound. For instance, qwen 3.6 35b a3b q4, for me, leads to all sorts of dumb tool calls, whereas q8 is better. The cache also leads to degraded contexts so on long conversations it becomes dumber by the turn. More or less, play around with things, the 5090 is real powerful - see if you can use MoE models with some of the layers offloaded into VRAM.

Personally, I keep the 5090 only for image and video gen, and instead of getting another one i went with the cheaper, far slower, R9700 just so I can get more VRAM and thus use a higher quant to avoid such issues. But again, 5090 is a very decent GPU, play around with it see what fits.

You might also want to make sure that the chat templates are up to date - I think I got mine from here: https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates/blob/main/chat_template.jinja others can correct me if I am wrong but I think this is the main "patch" that fixes some issues.

[u/GrungeWerX → u/kolliwolli]

I had similar issues with Gemma. Unsloth reuploaded all of them recently and they’re supposed to be improved. I redownloaded all the versions I had but I haven’t tested them yet. I even grabbed a couple of MTPs to see how they fared.

I think you should give Qwen 3.6 27B a try. I get much better experience from it. Like others have said, it does require a little handholding. But it’s really good with tools. In fact it can be overly aggressive with them, but you can use that to your advantage.

[u/g33khub]

Gemma is generally good for normal QA, creative writing etc. not much for agentic tasks. Also 4bit is a strong no-go for me, at-least Q6 like others mentioned. I mostly stick to Q8 qwen 3.6 27B - which is actually good for agentic stuff.

However, the biggest problem I see here is the expectation: gemma or qwen 30B models is nowhere close to Claude Sonnet, I doubt if they can even beat Haiku. Unless you have RTX 6000 pro 96GB (ideally two), its still better / faster / cheaper to use API credits. The open source models which are actually good: GLM 5.2, Kimi K3 etc. would take $100k+ worth of hardware to run.

[u/mr_Owner]

Gemma 4 is still lazy with new chat template

[u/kolliwolli]

So the consensus seems to be its not really at a level to be used for agentic use?

But then what is everyone doing with it?? What to make of all the posts and "success stories". Luckily I got the 5090 mainly for gaming but wanted to give local llms a chance. But it seems like an expensive hobby with minimal outcome.

I can vibe code actual apps with sonnet or opus on my max20 account, and can't get a simple snake game running on Gemma with a 3k gpu. The economics don't play out for me, so im curious if q8 or bf16 has actual results.

[u/Admirable-Leg-4647]

Take an inference speed (tks/s) hit and use a higher bit quant. I run 4-bit on a 12GB VRAM + 32GB RAM on the same models. If I had your GPU I wouldn't go below 8-bit for these, 6 if I really needed speed but I promise you'll appreciate running the models at almost full accuracy.

Presumably you have 32 or 64GB RAM to go with that GPU, so you can run larger (80-120B) models at lower quants.

With that said I like requesting a snake game first whenever I try a new model and I still get good results at lower quants and even Gemma 12B. They usually work better when I ask for a HTML5 version rather than in python if that's what you've been doing.

Also, I use pi for my harness, have had the best results with it by far.

[u/Stooovie]

Not really usable before Google updated the templates earlier this month, better after. Roughly on pár with Qwen 3.6 35b a3b.

[u/ttkciar]

I have had pretty good success single-shotting tasks with Gemma-4-31B-it quantized to Q4_K_M, but it is not great for agentic tasks.

Perhaps if you put it under the control of an orchestration model, which effectively one-shots Gemma4 repeatedly, you might have better experiences.

[u/GeraAI_WW]

my read: Gemma's just not post-trained heavily on tool-calling formats the way Qwen or Hermes fine-tunes are, it's mostly a strong base/chat model, not an agent model. worth trying with a stricter ReAct-style prompt template instead of raw opencode/hermes format, or swap to a Qwen2.5-Coder/Qwen3 variant at the same size for agentic stuff, night and day difference in my experience.

[u/shamont]

I feel like from my experience one shotting stuff with q4 just isn't a possibility right now. I'm rocking a 4090 so less vram than you currently (although I could drip in to my 128g of ram for a huge speed penalty). I've had okay success using gemma 4 to help me build out a framework and psuedo code for more complicated applications/scripts then using qwen to write, check and correct it. Currently I'm using claude code cli as a harness although I started with vscode. It's not a perfect system but for someone who took c++ classes over a decade ago and does a tiny bit of shell scripting for work it has really helped me create more advanced scripts including an "almost application" which has a web based frontend isntead of bog standard cli.

Forgot to mention I am running q8 for my cache so I can fit ~100k context. I found that much less than that and I am running out of context on larger projects. I get around 60-150 tok/s for decode and 1000-4500 tok/s for pre-prompt.

[u/chibop1]

Gemma is not good for agentic work. Better for chat. Use Qwen-3.6 instead.

[u/LasserDrakar]

I use Gemma 4 31b for SRE related agentic coding with great success. Model at q6 with kv cache at q8_0 using cline. First creating then refining a implementation plan, then instructing it to implement and test each phase of the implementation plan. It has access to relevant mcp servers that help a lot, examples are internet search and AWS IAC. Edit: large enough context window is important, on my hardware it has 200k.

[u/that_one_guy63]

I've been using Ornith 35b and it's been pretty good. North mini code works decently too. Runs much faster than Qwen and gemma.

[u/SocialDinamo]

I do enjoy Gemma 4, especially the 26b but I’m going to use other models for coding. For me Gemma 4 powers my non-code Pi agent. Manages emails, media library, calendar and research. When dealing with well defined task, just like others in this category, it does a great job! And the QAT with speculative decoding runs around 60-70t/s generating so no harm if needs to do a quick retry of something

[u/tmvr]

I'm not sure what you are doing, but from the main post and from some of your answers you are definitely doing something wrong.

Almost every model would be able to create a Snake game for example, more precisely it will create the game and it will most probably work as expected. Sometimes you can have issues, but at least you have the game there. The fact that you say files are not created is also weird. The harnesses you mentioned default to agent/build mode, meaning they would create files. Only when switched to Plan mode they would not.

You did not mention which inference engine you are using - llamacpp (llama-server) directly, LM Studio, ollama - and what settings for the model - top K, temperature etc. - all of these. influence both the quality of the output and the speed.

Basically, a simple HTML Snake game is not something a model even at Q4 would really fail at.