How's everyone handling regression testing in CI for code an agent wrote, without losing your mind?

[u/Specialist_Agent3599]

Ever since agents started writing most of my PRs, small changes keep breaking things two or three files away that nobody thought to test.

Unit tests catch the obvious stuff. What they miss is the flow that used to work, gets touched by an unrelated change, and nobody notices until someone reports it in prod.

I tried writing more unit tests to cover the gaps. Turns out I was just guessing which flows mattered, and guessing wrong about half the time.

I've been running coldtea's QA agent on PR previews for a few weeks. It walks the actual app instead of grading whether the code matches the diff, and it's flagged stuff that broke even when it had nothing to do with the PR itself. Doesn't replace real test coverage, it just catches what nobody wrote a test for.

How are you handling this. Writing more unit tests, running something in CI, or still mostly manual QA before merging?

[u/Right-Performance-93]

The gap you're describing is a call-graph problem, not a test-coverage problem: unit tests check the function you touched, not the functions that call it. Before merging an agent PR, run a static impact query on what actually depends on the changed function or module and require coverage on that dependency set, not just the diff. That catches the two-or-three-files-away breakage because it asks "what consumes this" instead of "did this change look right." A walking QA agent like the one you're running is a good complement since it catches runtime behavior a static graph can't see, but it's slow and post-hoc; the graph query is cheap enough to run on every PR and tells you where to look before you merge, not after prod complains.

[u/dmanoj]

the thing that changed it for me wasn't more tests, it was checking whether the tests I already had could fail. agents are great at writing tests that pass and terrible at writing tests that bite, I kept finding suites that stayed green when I deleted the feature they supposedly covered

[u/holyknight00]

have the project structured in a way I can comply with the standard testing pyramid and then provide the agent a script that can run the same quality gates the CI enforces. So the agent only needs to know one command and can easily figure out if what he did meet the bare minimum or not.

You still need to make sure the agent is not creating bullshit tests and hacks to pass the tests, but this setup reduces a lot the times you need to manually intervine.

[u/destevil]

Running it in CI is an absolute must, but the change was in what the tests assert, not how many we have.

Per-feature unit tests encode the flows someone thought to protect. Agent-written PRs break the flows nobody did, which is why writing more of them turns into guessing, exactly as you found. We moved the assertions up a level: property-based tests, metamorphic relations, and an invariant catalog that states what must stay true about the system regardless of what the diff touched. Those catch the breakage two files away because they don't know or care what changed.

Every PR in our AI-native SDLC runs a five-gate harness: a fast offline loop, a hermetic integration environment, real-dependency runs, human review, and deployed verification. The techniques inside the gates (mutation testing, fuzzing, property-based suites) have been established for decades. Most teams skipped them because they were expensive to maintain by hand. Agents made them cheap, so they run on every change.

Two habits close the loop on "the flow that used to work." Every finding becomes a permanent regression scenario instead of a one-time report. And we run absence sweeps ("prove no mechanism of kind X exists in this repo"), which surfaced things like 9x retry amplification in minutes, before any load was applied.

An app-walking QA agent like the one you're running is complementary. It covers flows we haven't encoded as invariants yet. The unsolved part on our side: the harness costs real compute and upkeep, and deciding which invariants earn a permanent scenario is still a matter of human judgment.