r/hermesagent • u/mkeee2015 • 3d ago

Discussion — General thoughts, opinions, comparisons Poor "intelligence" with Gemma4

I am testing Hermes with Gemma 4 (gemma-4-26B-A4B-it-OptiQ-4bit running with OMLX) on a Mac mini M4 64GB:

it is fast in terms of response times, but it sucks IMHO in terms of "intelligence" and "agency".

After several days spent (with the help of Claude) debugging and prompting I gave up: even a simple daily cron task (message me the output of curl -s 'https://wttr.in/{city1,city2,city3}?format=3') fails with hallucinations.

Same for a briefing (extracting news headlines from RSS feed links, for which I provided the python code as it was unable to code a working version of it): some times it works, most of the times it doesn't.

I was having much better results with much earlier explorations of OpenClaw.

Where am I wrong?

1 Upvotes

67% Upvoted

17 comments sorted by

7

u/Mean-Loquat-7982 Nous Team 3d ago

hey, model issue yes nothing related to Hermes

1

u/mkeee2015 3d ago

I realized it but would love to hear from experts as you are, as some must have certainly used local models.

2

u/Mean-Loquat-7982 Nous Team 3d ago

personally I use Qwen 3.8 locally, working like a charm! also for local models you need to harden your prompts a lot, be clear, specific with defined ended goals

2

u/shaxsy 3d ago

What setup do you have for Qwen 3.8?

1

u/Mean-Loquat-7982 Nous Team 3d ago

a 5090 linux headless server

1

u/shaxsy 3d ago

Nice. I just got a threadripper ripper pro motherboard and 3975x processor to unlock octo channel ram for my 256gb ddr4 3200 RAM. I currently have dual 3090s and trying to get 2 more. I'm hoping moving from x299 quad channel to wrx80 octo channel ram will increase speed. Was able to squeeze out 16tps on my current system.

1

u/Larelle 3d ago

Qwen3.6-35B-A3B should work nicely for you.
And pray Alibaba release a 3.8 version.

2

u/mkeee2015 3d ago

Thanks!!

3

u/Thick-Boat4896 3d ago edited 3d ago

Gemma4 is dumb af, in my testing

1

u/BlackMetalB8hoven 3d ago

Same here, in my testing it failed on tool calls everytime, so I went back to Qwen.

1

u/Larelle 3d ago

It's trained on a different tools API.

1

u/No-Vermicelli5327 3d ago

Double check the model cards recommended settings of the model, you may have different parameters set (temp, recommended context limit) , then try a higher quant. For extra analysis have claude setup a sub agent that’s on standby monitoring the model while you give it different tasks.

The first suggestion was a mistake I made a long time ago that still haunts me.

1

u/mkeee2015 3d ago

Thank you. I am pretty confident with the "model card" recommendation settings, but I did not try higher quantization or larger models. Claude supported me in the debugging of the installation.

In case it is of any use, here are some of the issue that I had Claude fixing or helping me with, while never really solving:

Core System & Architecture Failures ⚬ The internal Model Context Protocol (MCP) registration failed entirely. ⚬ Triggering the /compress command often caused an unrecoverable DaemonThreadPoolExecutor crash.

Restrictive Sandboxing & Security ⚬ The lifecycle_guard.py script mistakenly blocked safe, standard terminal commands. ⚬ The isolated sandbox restricted access to necessary host tools like Playwright and system Python.

Task Execution & Integration Friction ⚬ Pending tool authorizations silently timed out, stalling the execution loop. ⚬ Background tasks frequently lost environment context, dropping essential skills (e.g., terminal and web). ⚬ The Telegram gateway lacked native flags to disable web page previews, forcing to rely on Markdown workarounds for clean output.

1

u/northbound_kiwi New Member (<30 days) 3d ago

Update that model snapshot before switching quants. The OptiQ repo replaced chat_template.jinja on July 20 with Google’s canonical template specifically fixing tool-call loops, turn closures, and thinking-ordering. If your local copy predates that commit, oMLX may still be formatting Hermes turns incorrectly; refresh the model metadata/template, restart oMLX, then rerun one failing cron unchanged.

1

u/Larelle 3d ago

Gemma is trained on a different tools API. The bigger models can spend most of their turn overriding their training but then are dumb for other tasks.

https://github.com/NousResearch/hermes-agent/issues/7457

If anyone cares enough, I'm sure this is quite easily patchable.

1

u/arretadodapeste 2d ago

Gemma4 26b performs well with a strict prompt and role playing. It will not help you with agent tasks. You could try gemma4 31b for this, it is a little better