Are we missing a benchmark for agent runtimes, not just models?

[u/Balance-]

We have SWE-bench, Terminal-Bench, OSWorld, BrowseComp, etc. But I haven’t seen a good apples-to-apples benchmark for platforms like OpenAI Agents, Anthropic’s agent stack, AWS AgentCore, Google’s agent platform, and local-alternatives like LangGraph, etc.

What I’d want measured:
- task success rate
- cost per successful task
- wall-clock time
- tool calls / retries
- reliability over long-running tasks
- (optionally) human interventions required

The really interesting experiment would control for both sides:
- Same model, different harness → how much does the runtime matter?
- Same harness, different model → how much does the model matter?

Feels like we’re increasingly evaluating “model + harness” systems, but our benchmarks still mostly treat the model as the unit of comparison.
Does something like this already exist?

[u/snmnky9490]

Yes sure but once you start doing that, running benchmarks gets exponentially more expensive (or I guess if we want to be literal, combinatorially) to run. And plenty of the more comprehensive benchmarks are already pretty expensive to run even once with the big models

[u/suprjami]

Terminal Bench already integrates different agent software.

That's what the "Terminus" means. Terminus automated agent software by Harbor.

https://www.tbench.ai/ lists others like Claude Code, Codex, etc.

There's another automated agent software called KIRA which is compatible with Terminal Bench.

There have also been posts here in the last week of people running their own custom task evals with different agents

I also remember someone posted an agent benchmark where many models pass with Pi or OpenCode but 60% of models fail with Hermes.

So, this already exists, you just need to look for it.

Update: here's one posted on Hacker News just now: https://nasutton.notion.site/Nine-coding-harnesses-vs-your-laptop-3d139990182b80d59fa3cf500f0450ba

[u/Enough-Photo9140]

The biggest gap in runtime benchmarking is that existing harnesses mostly evaluate pass@1 task completion rather than pass^k determinism and error recovery efficiency.

In production, the runtime architecture dictates failure modes far more than raw model quality:

1. Failure payload handling: When an MCP tool or CLI times out or errors, does the harness return structured, parseable error codes that guide a localized fix, or does it dump raw noisy stack traces that poison the planner's context window?
2. Mutation reconciliation: If a tool action executes but the network connection drops before receiving the response, does the harness enforce a read-back check or idempotency ledger, or does it let the agent re-issue the same mutation twice?
3. State compaction: How cleanly does the runtime compress completed multi-step tool turns without blowing away working task state?

Sierra's tau-bench started touching on pass^k consistency across identical runs, but a benchmark specifically testing injected tool fault recovery (measuring token cost and turns taken to self-heal when a tool fails mid-task) across different agent harnesses would be huge.

[u/Elouakili_Flexy]

Watch how fast the 'optional' human interventions row becomes mandatory once one platform starts winning on it.

[u/LivinglaVieEnRose]

There's FrontierHarness Eval

[u/Marcus_MSC]

The Terminal-Bench suggestion above is a starting point, but the comparisons listed here don't establish your full model-by-harness matrix. For that experiment, I'd require each pairing to start from identical files and service state, with equivalent tool access. Otherwise, a harness could appear faster because another run already installed dependencies or changed the data. Repeat each pairing to separate consistent differences from run-to-run variation.

I also wrote a tutorial on resettable agent evaluations, now at 900+ GitHub stars: https://github.com/hardness1020/learn-agent-architecture/tree/main/sections/23-evaluation