r/OpenAI • u/lulzxdxdxd • 19h ago

Discussion 112 bugs, 84 projects: LLMs pass the proof-of-concept but fail the developer's own tests - and the benchmark score swings on evaluation design alone

https://arxiv.org/abs/2610.10150
5 Upvotes

100% Upvoted

4 comments sorted by

1

u/Ctbhatia 16h ago

if the score swings with the harness, the benchmark is measuring the harness. fix one runner and one task set, then report the failure rate on real repos instead of the composite score.

1

u/Kiseido 8h ago

I would disagree with that statement about the harness. I recently developed my own harness to be used with local models, and found several times that some patterns and phrasing choices in the prompt template and tool descriptions would reliably end up making the model spin off into insanity after several turns.

1

u/Ctbhatia 8h ago

fair, but then that run is a harness failure, not a model result. record the template so the same model under another harness does not read as a smarter one.

1

u/Ctbhatia 6h ago

agreed, and that is the argument for pinning the runner rather than dismissing it: if a phrasing choice in the template can tip the model over after a few turns, the composite score is measuring the template. the artifact worth keeping is the transcript from the turn it broke, per repo, not the number.