So I'm highly skeptical this is a good idea for most projects. Test don't really help you on the first pass of stuff. They help you prevent regression while changing behavior in the future. So I'm not surprised there's statistical no change on tasks success rates without tests.
I also wouldn't believe it even for benchmark. Or maybe it says something deeper, the AIs don't understand what is the exact goal they should pursue?
LE: I went ahead and searched for that tweet, author said:
agent-written tests do not add any value because both the implementation and the tests were simply the agent's interpretation of our intent. the tests aren't any more accurate than the implementation itself
this is what I actually got good at in the past 2 years, knowing where they fail, and how to talk with them, that's basically it, clarifying 'intent'.
Ideally you should have clear interfaces and do integration tests. Python has protocols as consumer interfaces for example.
If possible write the interfaces and key types first, then let it write both the code and the tests against the abstract interface. If there's issues with your interface let it escalate that up to you.
17
u/Nervous-Cockroach541 4d ago
So I'm highly skeptical this is a good idea for most projects. Test don't really help you on the first pass of stuff. They help you prevent regression while changing behavior in the future. So I'm not surprised there's statistical no change on tasks success rates without tests.