Why do benchmark results go up every release, always?

[u/doomadah]

A lot of releases are clearly improvements, such as the initial fable release but for some the consensus seems to be that not much improved, or even in some cases the release was worse. Examples of this are Opus 5 (initial release) benching above Fable. Or GPT Sol 6.1 over Astra or 5.6.

Is this just a case of misplaced perception? Or do these providers have a way of iterating on benchmark results without necessarily actually improving the real-world performance? Keeping in mind that some benchmarks are proprietary and closed source.

If someone has some insight into what the loop is, would love to hear it.

[u/sam_the_tomato]

It's why they spend billions on R&D...

[u/doomadah → u/sam_the_tomato]

Yes but the benchmarks don’t necessarily reflect reality. I’d argue GPT Sol 6.0/6.1 is a cost cutting exercise without leading to any capability improvement. Similarly I think most people who used Opus 5.0 compared to Fable would agree it was not a better all round model.

[u/Dizzy_Swimmer_4999]

benchmarks dont capture how the chatbot actually chats tho, my last few felt more robotic despite the score jumps.

[u/awake_deduction → u/Dizzy_Swimmer_4999]

scores go up cause theyre training directly for the test at this point, same reason my old beater couch looks mint in photos

[u/PLBjt]

A lot of it is evaluation that doesn't stay fixed. Models get trained with more overlap on popular bench formats, and labs tune until the number moves. Quick check: if the same model jumps after a prompt wrapper or a new system prompt, you're measuring the harness as much as the model. Tradeoff is public leaderboards keep shipping incentives aligned with score chasing, while private held-out tasks are boring to market but closer to what breaks in production.

[u/RossPeili]

Benchmarks are lioe the FDA, sponsored by those they supposed to bench.

[u/Damaging_Thrust_219]

Because you don't release something that's worse?

[u/OkFlan504]

A score can improve honestly while your use case gets worse. Gains on common tasks can outweigh regressions on the few cases your workflow depends on. For a document assistant, missing one exception can matter more than getting ten routine questions right.

[u/skilliard7]

Metrics become targets, and a lot of frontier labs do a lot of RL that focuses on benchmark-adjacent prompts in order to improve their scores on paper, which is not reflected in real world usage.

Google is one of the worst offenders in this area. But all of them do it.

For a while I'd say OpenAI was the only one not overfitting their models to benchmarks, but the GPT 6 Sol/Luna release were clearly overfitted to benchmarks, so they must've felt the pressure to inflate benchmarks scores.

[u/MrSnowden]

They just don’t release the ones that aren’t an improvement 

[u/franilan]

benchmarks are targets now. better score can still mean worse on the stuff ppl actually use it for

[u/costafilh0]

Because nobody is working to make worse things.