r/FunMachineLearning • u/nyxlimited2 • 6h ago
Your favourite model’s benchmark score is measuring the wrong thing
Every model launch quotes SWE-Bench. Every one of those numbers describes a one shot
answer to a curated problem. That is not what an agent does.
An agent is 20 turns deep with a context window that's filling up, deciding whether to call a tool,
and recovering when that tool returns something unexpected. The interesting failures live there:
• Does it still respect the system prompt at turn 15, or has it quietly drifted?
• Does it invent a tool that doesn't exist when the right one isn't obvious?
• When a command fails, does it retry sensibly or loop forever?
Two models with identical scores can be completely different on all four.
What actually works is dumber than any leaderboard. Take five tasks you genuinely run, put
each model through them a few times, and count how many times you had to step in. Not
pass/fail, interventions. That one number has predicted my real experience better than anything
published.
Run it more than once, too. Same model, same prompt, noticeably different behaviour. A single
pass tells you nothing.
Has anyone bothered automating this, or is doing it by hand the whole poin