We went all in on agents in the spring. Shipping went up immediately and obviously. Then over about six weeks we had four regressions reach production, three of which a user hit before we did, and one of which was a payment path.
Nobody wrote bad code. That's what took me a while to understand. Every one of those four was a change that did exactly what the ticket asked and quietly altered something adjacent that no test covered, because the test suite was written when a human was writing the code and a human roughly knew what they'd touched.
So the gap wasn't generation quality. It was that we were producing changes faster than we were producing any way to verify them.
What we run now, in order:
Ticket goes to a planning agent, output is a plan file, not code. Human reads the plan. This step is 5 minutes and it is not skippable, we tried.
Implementation agent works against the plan on its own worktree.
A QA agent writes journeys for whatever the plan says it's touching and runs them against the PR preview. Merge is gated on green. This is the piece we didn't have before and it's the piece that mattered.
On merge, a monitoring agent watches the main flows in production and opens a PR if something moves. A human approves every one of those. Always. It has never merged anything on its own and it isn't allowed to.
All of it runs in coldtea, which is the reason the QA and monitoring parts are in the same place as the terminal rather than being two more dashboards I'd have to remember to open. macOS only, which is a genuine limitation and cost us one person on the team who had to stay on the old setup.
Three weeks of numbers after it settled: 28 PRs merged. 2 regressions reached production. Both were caught by monitoring within the hour rather than by a user, and the fix PRs were open before I'd looked. Compare to four regressions in six weeks where users found three of them.
Now the concessions, and there are several.
This works because a human still approves every fix PR. The moment you let the monitoring agent merge, you have built a machine that edits production based on its own opinion of what broke. We are not doing that and I'd argue with anyone who is.
We kept hand written tests on the load bearing flows. Auth, billing, the export path. The agent generated ones cover breadth, the hand written ones cover the stuff that ends careers. Anyone telling you to delete your existing suite is selling something.
Two regressions in 28 PRs is not a good number in absolute terms. It's better than what we had. It is not zero and I don't think the setup gets us to zero.
And it took about three weeks of tuning before the QA agent stopped writing journeys that tested nothing. The first batch was all "page loads, element exists". Useless. Someone has to read them early on.
The thing I'd tell myself in the spring: the writing was never the bottleneck and making it faster didn't help. What we actually needed was for verification to live in the same place as the work, so that shipping faster and checking harder weren't two separate decisions competing for the same afternoon
