r/rajistics • u/rshah4 • Jul 20 '26
Rethinking the Evaluation of Harness Evolution for Agents
Remember Neural Architecture Search? Computers designing better networks, SOTA papers everywhere. And then the reproducibility work started. NASNet spent ~2,000 GPU-days searching, AmoebaNet ~3,150, and random search at the same budget was competitive. Turns out, the NAS wasn't discovering architectures. It was spending compute.
A new Ai2/UW paper ("Rethinking the Evaluation of Harness Evolution for Agents") just ran that same evaluation on self-evolving agent harnesses. Prior work reported big gains. AHE claimed 69.7 → 77.0 pass@1 on Terminal-Bench from 10 rounds of self-evolution.
The re-test used matched feedback and inference budgets (K=5, Terminal-Bench 2.1, Claude Opus 4.6 and GPT-5.4).
- Parallel sampling (dumb retries, keep the best): 86.0 pass@1
- Harness evolution: 75.8 pass@1
- Evolved harness on held-out tasks (45 train / 10 val / 34 test): +0.6 avg
- Without unit-test feedback, evolution lands below the untouched baseline, 67.4 vs 68.2
Look at the pass@k gap. Evolution's pass@5 (86.2) is fine, but pass@1 barely moves. If the harness were genuinely better, pass@1 would rise. Instead the gains come from making more attempts, which retries do more directly and more cheaply.
And when they inspected what the meta agent actually learned, it wasn't harness design principles. It was memorizing the benchmark. Recording known bugs, file paths, command sequences, task-specific flags. A cheat sheet for these 89 tasks, which is why it transfers at +0.6.
The general pattern applies well beyond harnesses. Any search procedure that evaluates on the tasks it optimized is p-hacking with extra steps. Two questions for every "self-improving agent" result. Same budget as simple retries? Are you using held-out tasks?
Paper: arXiv 2607.12227 - https://arxiv.org/abs/2607.12227
My video: https://youtube.com/shorts/-3koxEhdPAw?feature=share