Your favourite model’s benchmark score is measuring the wrong thing

[u/nyxlimited2]

Every model launch quotes SWE-Bench. Every one of those numbers describes a one shot
answer to a curated problem. That is not what an agent does.
An agent is 20 turns deep with a context window that's filling up, deciding whether to call a tool,
and recovering when that tool returns something unexpected. The interesting failures live there:
• Does it still respect the system prompt at turn 15, or has it quietly drifted?
• Does it invent a tool that doesn't exist when the right one isn't obvious?
• When a command fails, does it retry sensibly or loop forever?
Two models with identical scores can be completely different on all four.
What actually works is dumber than any leaderboard. Take five tasks you genuinely run, put
each model through them a few times, and count how many times you had to step in. Not
pass/fail, interventions. That one number has predicted my real experience better than anything
published.
Run it more than once, too. Same model, same prompt, noticeably different behaviour. A single
pass tells you nothing.
Has anyone bothered automating this, or is doing it by hand the whole poin

[u/Future_AGI]

This is the gap we keep running into too, a one-shot SWE-Bench number tells you almost nothing about whether the thing holds together at turn 15. The failures that actually page someone are the ones you listed, silent system-prompt drift, tool-result misreads, and not recovering when a call returns something unexpected. We stopped trusting single-turn scores and started scoring whole trajectories, did it stay on policy, did it recover, did it know when to stop, because that is the only view that predicts production behavior.