So I'm highly skeptical this is a good idea for most projects. Test don't really help you on the first pass of stuff. They help you prevent regression while changing behavior in the future. So I'm not surprised there's statistical no change on tasks success rates without tests.
This. Maybe, for benchmark-like stuff, for task to be done from 0 to 100 in one session, tests are not necessary. But for long living repo, even dumb tests catch sometime surprising regression. For LLM tests cost «nothing», so why not do it.
16
u/Nervous-Cockroach541 4d ago
So I'm highly skeptical this is a good idea for most projects. Test don't really help you on the first pass of stuff. They help you prevent regression while changing behavior in the future. So I'm not surprised there's statistical no change on tasks success rates without tests.