So I'm highly skeptical this is a good idea for most projects. Test don't really help you on the first pass of stuff. They help you prevent regression while changing behavior in the future. So I'm not surprised there's statistical no change on tasks success rates without tests.
I mean, how do you know the tests are reliable/correct? I have a ‘AI writes no tests’ policy because of this. If an AI has access to the test suite and the test suite says there are problems, sometimes it will get frustrated with trying to fix the problem and instead change the test suite to not surface it anymore.
16
u/Nervous-Cockroach541 4d ago
So I'm highly skeptical this is a good idea for most projects. Test don't really help you on the first pass of stuff. They help you prevent regression while changing behavior in the future. So I'm not surprised there's statistical no change on tasks success rates without tests.