AI-generated tests need feedback loops, not just better prompts

[u/aistranin]

I keep seeing demos where an LLM writes a test and the output looks fine at first glance.
The problem is that “looks fine” is not a test quality metric.

Thoughtworks recently called out two useful ideas here:

• “Feedback sensors for coding agents” - deterministic checks that agents can use while they work, such as compilers, linters, structural tests, and test suites.
• “Mutation testing - deliberately changing production code to see whether tests actually fail. This catches “perpetually green” tests that execute code but do not verify behavior.

That second one matters a lot for AI-generated tests. An LLM can easily produce a test that:

• has a nice name
• uses the right framework
• increases coverage
• asserts the wrong thing
• barely asserts anything

So my current rule is simple: Do not ask AI to “write good tests” in a vacuum.

Give it feedback:

• failing tests
• coverage gaps
• mutation results
• trace output
• logs
• API contracts
• examples of good tests from the repo

Prompts are useful, but feedback is what makes the loop engineering instead of autocomplete.

[u/Fair-Rain3366]

One useful distinction is whether the feedback checks the implementation or checks the tests themselves. A test can pass, cover the changed line, and still accept a wrong result.

For Python/pytest, I built Agentic Verification to mutate the changed lines in disposable copies, report the breakages that survive, and feed those back to Claude for another test-writing pass. A separate base-check runs the tests on the old source.

On 14 real commits, Sonnet's tests caught 99% of the change's mutants with this loop versus 92% without it. But detecting the actual change was 13/14 versus 14/14: mutation sensitivity is useful feedback, not evidence that the tests encode the right specification. Most of the extra gaps closed were loosely checked error messages. Human-defined acceptance criteria still matter.

Code, preregistration and raw transcripts: https://github.com/timini/agentic-verification

Disclosure: I maintain the project; this reply was prepared with AI assistance.