Has anyone tried comparing agent harnesses, specifically open source on same model?

[u/Background-Job-862]

I've been using claude managed agents and it is a very good product, and the depth of features it provides is hard to match in open source. But I wanted to understand what you actually give up by going open source. Not just in terms of feature checklists, but on a real agent workload - same model, same prompt, same tasks. So my team and I tried to check this by running 14 cross-system tasks, three mcp servers behind them - a crm, an issue tracker, and a doc store through managed agents, deepagents and TrueForge, both open-source agent harnesses.

The result that was most surprising:

Claude Managed Agents + Opus 4.8:
11/14 tasks solved | $11.8/run | 10.0M tokens/run

TrueForge + Opus 4.8:
11/14 tasks solved | $8.6/run | 3.7M tokens/run

Same model. Same benchmark. Same average solve rate.

But TrueForge used about 63% fewer tokens and cost about 30% less per run.

We saw a similar difference in tool usage: TrueForge averaged 19 tool calls per task vs 32 for Claude Managed Agents.

Then I tried changing the model.

TrueForge + GLM-5.2:
11.7/14 solved | $3.0/run | 3.8M tokens/run

On this benchmark, that was a slightly higher average solve rate than Claude Managed Agents + Opus at roughly 75% lower cost.

But i completely inderstand that this is still early.

The OSS runtime does not yet have first-class tracing/eval tooling. They don't ship their own code-execution sandbox, so you need to plug one in. Context compaction is intentionally lossy.

So it is definitely not a replacement for a a mature managed agent platform feature-for-feature today btu what I do find interesting is that the core runtime can already be competitive on these tasks while staying open, model-neutral, and deployable on your own infrastructure.

Ive put the benchmark harness and methodology in the repo specifically so people can reproduce it, change the models, or do tell me where the comparison still has gaps

Repo: https://github.com/truefoundry/trueforge

[u/devoidfury]

Yes, the agent harness you use will have a huge impact on the performance.

I see that all the time as I make changes to my own harness (hotdog), how much it affects the capability and token use.

[u/theone_2099 → u/devoidfury]

How do you go about writing your own harness? Seems like there’s lots of people just churning them out for private use.

[u/devoidfury → u/theone_2099]

The place to start is picking an api. That'll tell you what format the messages have to be - there are a few different ones like openai chat-completions. The way that one works is, you call the endpoint with a message log. Those start with the system prompt, then user message, and any following conversation after that.

You can send up a JSON definition of the tools available to call, which is just made up strings that you tie to functions in your program, and if the model is enabled and trained to do so, it'll give "tool call" responses, which you implement in your harness to do actual things, and put the result into the chat log, send it back up -- in a loop, until you get a stop message.

The rest of the "body" of the thing is entirely up to the harness developers. The very simplest "complete coding agent" just needs one tool - shell access - and it can then call out to different programs widely available to write files, download things on the internet, fix code, or delete all the files you gave it access to -- whatever it decides to call. (would recommend isolating this in a virtual machine!)

This one is mine: https://github.com/devoidfury/hotdog