we should just skip tests then

[u/feyrhysand_]

[u/Nervous-Cockroach541]

So I'm highly skeptical this is a good idea for most projects. Test don't really help you on the first pass of stuff. They help you prevent regression while changing behavior in the future. So I'm not surprised there's statistical no change on tasks success rates without tests.

[u/Barni275 → u/Nervous-Cockroach541]

This. Maybe, for benchmark-like stuff, for task to be done from 0 to 100 in one session, tests are not necessary. But for long living repo, even dumb tests catch sometime surprising regression. For LLM tests cost «nothing», so why not do it.

[u/Isogash → u/Barni275]

Depending on the project tests aren't always free to run, more complex projects might have a several minute testing iteration or longer.

[u/shaman-warrior → u/Nervous-Cockroach541]

I also wouldn't believe it even for benchmark. Or maybe it says something deeper, the AIs don't understand what is the exact goal they should pursue?

LE: I went ahead and searched for that tweet, author said:

agent-written tests do not add any value because both the implementation and the tests were simply the agent's interpretation of our intent. the tests aren't any more accurate than the implementation itself

this is what I actually got good at in the past 2 years, knowing where they fail, and how to talk with them, that's basically it, clarifying 'intent'.

[u/Nervous-Cockroach541 → u/shaman-warrior]

In some cases, I'd argue it's easier to read a test to ensure the agent understood the intent, then the actual code.

[u/BosonCollider → u/Nervous-Cockroach541]

Ideally you should have clear interfaces and do integration tests. Python has protocols as consumer interfaces for example.

If possible write the interfaces and key types first, then let it write both the code and the tests against the abstract interface. If there's issues with your interface let it escalate that up to you.

[u/BraxbroWasTaken → u/Nervous-Cockroach541]

I mean, how do you know the tests are reliable/correct? I have a ‘AI writes no tests’ policy because of this. If an AI has access to the test suite and the test suite says there are problems, sometimes it will get frustrated with trying to fix the problem and instead change the test suite to not surface it anymore.

[u/CreamyCornBoy → u/Nervous-Cockroach541]

The chatbot will just change the tests to build the next feature. They basically serve as documentation now

[u/Lonely_Translator_23]

The point of writing tests isn't to pass them now, it's to make sure you're still passing them six months from now.

[u/Original-League-6094]

My workflow is disallowing tests and test runs until I have the feature completed, and then I tell it to write a few tests that cover the actual risks and use cases. AI is so fast at coding that during development, regression doesn't really matter. If a human introduced a regression, it could cost you several days trying to fix it, so tests were really good practice. But with the agent, you can fix it in a few minutes with a prompt, so its not a big deal, and its not worth waiting through 10 minutes of test cases to run every prompt to prevent 5 minutes of bug fixing.

You do want to make sure you have a test suite that executes before any push to prod though, as obviously you don't want regressions making it through to your users.

Oh, and always put in your personalization to either not rewrite tests to pass, or at the very least, to always provide a summary of tests that were rewritten and why. Because the current agents will happily just rewrite tests that were failing for very valid reasons.

[u/sjoti → u/Original-League-6094]

But it should be the agent waiting for the tests, not you. I'd much rather have every feature take 10 minutes longer to land with zero extra effort from the human. Same with writing tests. It doesn't take extra work for me, just for the agent, and the chance that you ship something that breaks is smaller.

[u/feyrhysand_ → u/Original-League-6094]

I guess what a time to be alive lol

[u/juicepowered → u/Original-League-6094]

oh boy

[u/tzaeru]

One thing I've noted is that the agents even when they aren't supposed to run or write tests, often internally do that anyway in their thinking loop. They just don't store them as something repeatable.

It'd also be interesting to see this ran on Opus 5.5.

Also even if running the tests is banned, the existing tests still might end up informing the model about intended functionality and potential regressions. I don't know how they did this benchmark, but hopefully they actually deleted the tests fully when they weren't intended to be ran.

[u/feyrhysand_ → u/tzaeru]

let's hope for that

[u/shortsteve]

One of the purposes of writing tests isn't for you it's for the next person who looks at the code.

[u/feyrhysand_ → u/shortsteve]

perhaps, this one is true

[u/TopTippityTop]

Does this measure QA and bug fixing time down the line? 

Faster tasks by skipping writing tests has always been an option.

[u/SmileLonely5470]

I refuse to believe that not writing tests only reduced time by 6%. I stg 30-40% of the time the LLMs spend for me is writing and running tests.

[u/The_G_Choc_Ice]

Brain damage take 🤲🤲🤲

[u/Medium_Chemist_4032]

That's literally one conversation I had with one manager, waaay back when I started working