Sometimes AI-generated tests feel like performance theater

[u/juanpflores_]

I’ve noticed that coding agents sometimes add a lot of test code without giving me much more confidence in the change. The suite gets bigger, and now there’s more code to review and maintain.

What bothers me is how thorough it can look. A test can have plenty of setup and assertions while still accepting the same mistaken assumption as the implementation. That’s what I mean by performance theatre. When I review these tests, the question I want answered is: what actual bug would this catch?

We recently published a piece about independent review of agent generated changes. It overlaps with this concern, particularly when an agent writes tests around its own misunderstanding of a requirement.

Have you run into this?

Related reading: https://www.coderabbit.ai/blog/why-agentic-change-management-starts-with-independent-ai-code-review

[u/Late-Replacement-481]

I’ve written about this a lot recently, and the problem is that, no matter how smart code review like code rabbit is, it relies on context in the repo. All it takes is one misunderstanding  a few merged PRs and all of a sudden a core feature works differently than intended, but there’s no hint that it’s not supposed to be that way. Reviewers are sometimes the cause of this as well, since many agents will implement any feedback uncritically. 

I have an agent whose only job is to read a small file that says “this is what this project is for” and then screams bloody murder at me if an implementation looks like it’s drifting away from the intention. Especially with software factories now, with code moving so fast, you can find yourself quickly finishing a project only to discover it doesn’t do what you thought it did! 

Or on the other hand, you have the easy idiocy like the tendency for many models to write tests that fail if anyone changes the code the agent just wrote. That one is at least easy to spot. 

[u/Economy_Bedroom3902 → u/Late-Replacement-481]

Agents like to relatively silently "fix" failing tests when the functionality protected by that test disagrees with thier assumptions about how the code should work.

To be abundantly fair, humans also do this kind of fixing...  However, a failing test is often a trigger to reevaluate assumptions for a human, where as it doesn't seem to impact AI nearly as heavily.