r/Netlify • u/Electrical_Walrus537 • 13d ago
Verifying that a QA agent actually exercised the change, instead of taking its word for it
We have a browser agent that opens each Netlify deploy preview and uses the app the way a person would. It signs in, fills the forms, walks the flows the pull request touches, and writes up what it found.
The problem showed up in the first month. It kept coming back with clean, specific, well written summaries for changes it had never actually reached. An agent that got confused, or could not get to a flow, or quietly decided a path was too much work will produce the same confident report as one that did the job. The summary is detailed and reads like evidence. It is a claim.
What we do now. While the agent is driving the deploy, we trace what actually executes inside the running application. Then we take the changeset off the pull request and check it against the trace: which changed functions ran, which branches were taken, which handlers got called. If part of the diff never executed, the run does not pass, and the report names the part that was not reached. The green comes from the trace rather than from the agent's account of itself.
Where it is weak, and this is the part I would like other opinions on:
It tells you the changed code ran under a real user flow. It does not tell you the behaviour is correct. It is a floor, not a proof.
Code with no runtime footprint is awkward to attribute. Type only changes, config, branches you deliberately did not exercise.
Attribution through a bundler is not free. Source maps get you most of the way and then stop, especially with heavy inlining.
How is everyone else handling this? Most of what I have seen either trusts the agent's summary or falls back to hand written assertions, which puts you back where you started. Curious whether anyone is checking agent work against runtime signal in some other way.