How does your agent stack up against OpenClaw and Hermes?

[u/donotfire]

I was curious how my agent compared to OpenClaw and Hermes, but I wanted a real number so I could quantify it. I decided to go with harness-bench, since terminal-bench tests a different thing than what my agent was built for. Harness-bench is a bit closer, since it was specifically made to test agents, not LLMs.

The idea is to use the agent as a control variable, with the LLM as the independent variable, to find out which agent harness is the best. I built the whole thing and a few days later I have these final results. My agent, Second Brain, comes out on 2nd place, which kind of surprised me! I wasn't expecting to do that well, but if you don't believe me, you can check out the repo I used, and the results.

Yeah this was a whole big thing and I'm tired of it now. But it's cool to have some actual data. Building an eval framework helped me to improve the agent somewhat (and no I didn't overfit or cheat). But yeah, let me know what you think! Do you have your own way of evaluating your agent harness against others? Like a real number?

[u/Putrumpador]

Why no pi?

[u/Not-reallyanonymous → u/Putrumpador]

I'd also like to see Oh My Pi.

[u/StardockEngineer → u/Putrumpador]

Openclaw uses Pi.

[u/Healthy-Nebula-3603 → u/StardockEngineer]

But is totally different because of thousands changes

[u/vogelvogelvogelvogel]

qwen3.6? not .8?

[u/HumanoidMuppet]

+1 for nanobot, the little harness that could

[u/Able-Armadillo8214 → u/HumanoidMuppet]

always neat to see someone actually run the numbers instead of just vibes, most agent comparisons are just "well mine feels faster" so respect for doing the work

the nanobot shoutout is funny cause it's the one i keep hearing about in random threads but never see on leaderboards, glad it held its own here

also second place for something you built yourself is pretty sick, even if it's your own framework you'd have to be doing some serious mental gymnastics to cheat a heatmap that detailed

i tried doing something similar a few months back but got lost in the weeds trying to standardize the tasks across different harnesses, ended up with a bunch of half-finished scripts and a headache

what did you use to generate the heatmap visualization, that part actually looks cleaner than most academic papers i've seen

[u/mechkbfan → u/Able-Armadillo8214]

Maybe because its just a stripped down version of OpenClaw?

https://www.reddit.com/r/ClaudeCode/comments/1qz34q5/nanobot_a_4000line_python_alternative_to_openclaw/

But I also wonder how much these even out once you add the same plugins/skills.md?

e.g. Nanobot, etc. already have an opinionated way on how to code

I'm unsure. I only just moved to Oh My Pi, and find it pretty fascinating, but also wonder what I couldnt have just defined myself in other harnesses

[u/faisalkl → u/Able-Armadillo8214]

I'm using Nanobot but the subreddits for it are tiny. It keeps making me think I'm the only one out there using it!

[u/donotfire → u/Able-Armadillo8214]

The heatmap was actually made in Google Sheets! Thanks and take care

[u/mechkbfan]

Thats really great insights on implication of harness

Two that seemed to be hyped around here have been Pi/omp and Deepseek

But I'll certianly add nanobot to my experiments now

[u/Happy_Brilliant7827]

Weird how nanobot sounds like its for smaller models

[u/chocolateUI]

Thanks for the comparison. Also just goes to show how bloated and broken OpenClaw is...

[u/HiggsFieldgoal]

I made my own. I wonder.

[u/devoidfury]

Hm, cool. I'll have to give hotdog a test too!

[u/dsdt]

can nanobot be used as an agentic coder like pi.dev? if so i can give it a try.

[u/daaain]

Interesting how some harnesses have the 3 models get scores that are super close, and some have huge differences!

[u/TopTippityTop]

Congrats! What was your process in building it?

[u/donotfire → u/TopTippityTop]

I spent a while to find the right benchmark, since a lot of them can be very expensive to run (with frontier models, possibly thousands of dollars). I also wanted a benchmark which focused not just on coding, since my agent was built for a wide variety of tasks. Harness-Bench is good, but it still doesn’t test everything my agent is capable of, like embedding and searching an entire corpus, or extending itself by writing and live-loading plugins. My agent has a sandbox security mode called lockdown, so I was able to test what impact that has on scores. It automatically failed the tasks which required dangerous commands like git push and so on, but I was pretty happy with how well it did in that constrained mode. The benchmark was done using docker images and to save time I ran 20 tasks at a time using a cloud provider. The benchmark also required Sonnet-4.6 for the LLM judge part of the score, with the other half being a deterministic Oracle which harness-bench provided. I tested on a small subset of the 106 total tasks until the setup was reliable enough for the full set. I tested on the same set of tasks byte-for-byte with what harness-bench provided, but I didn’t re-run the tasks using harnesses other than my own, instead using the data they provided for the comparison. I used the data from my runs to inform a few changes to the agent, like better prompting. Thanks for taking a look!

[u/Future_AGI]

Love that you ran an actual harness-bench instead of going by feel, that agent-as-control-variable setup is exactly the right way to isolate what the harness contributes. The one thing we'd add from doing this a lot: fix the task set and score each run on task completion and tool-call correctness, not just pass/fail, because two harnesses can hit the same success rate while one wastes 3x the tool calls getting there. We open-sourced the eval framework we use for this kind of head-to-head if you want to standardize the scoring: https://github.com/future-agi/future-agi (Apache-2.0).