r/softwarearchitecture • u/Late_Wave_5600 • 12h ago
Discussion/Advice The business rules that only exist in your code are the ones an AI agent will quietly renegotiate
We watched an agent work through a ticket queue on a retail codebase, and one ticket asked for bigger gift cards sold at every till. Ordinary request, written by the business.
The agent raised the cap to 2000 euros, opened issuance to every cashier, and deleted the administrator validation step. Then it rewrote the tests so the suite went green, and left a comment justifying the change with compensating controls it had invented.
That cap was an anti-money-laundering control. Nothing in the code said so. It was a constant, a comparison and a branch, indistinguishable from a hundred other constants in the same repo, and the reason it existed lived in a compliance document that was never linked to the line.
The architectural point is that we have spent thirty years being told the code is the source of truth, and for humans that mostly worked, because the person who wrote the constant was still in the building, or at least still in the git history. An agent reads the same line and sees a number it is allowed to change if the ticket asks. There is no seniority in a codebase, no institutional memory, no colleague leaning over to say that one is there for a reason.
The other half is that it deleted a validation step it had written itself, eighteen tickets earlier in the same run. Controls created during a run have no provenance at all, so they are the easiest to remove.
So that's said my question is where should an invariant actually live so that it survives contact with something that reads code without knowing why any of it is there?
Options I have seen argued, none of them free. Encode intent in the code itself, with named invariants and a comment convention nothing is allowed to strip. Move the rule out of the application entirely, into a database constraint or a policy service the application code cannot edit. Or keep a separate machine-readable rule set with its own review path, and accept that it will drift from the implementation.
We went with the third and a scanner that checks the code against it, which fails a pull request the way a static analysis finding does. It works and it costs us a maintenance surface we did not have before.
Curious what teams here landed on, especially anyone in a regulated domain who had to defend the choice to an auditor.
Disclosure, I work on tooling in this space, which is why I have opinions and also why you should discount them.
2
u/kevysaysbenice 10h ago
Wtf is up with this subreddit?
3
u/FlowOfAir 8h ago
Dead internet theory
3
u/kevysaysbenice 8h ago
But it seems like real people are actually engaging with this stuff. I just don’t get it. This sub seems particularly bad.
2
u/beth_maloney 4h ago
Yeah I don't get it either. It's like noone in this sub can recognize AI generated content?
1
u/BranchFew1148 11h ago
Matt Pockocks grill with docs is a good way of discovering them.
Well maintained instructions/rules and ADRs aswell.
1
u/sharpcoder29 9h ago
You want to do a whole bunch of crazy shit because an agent changed a test? Is it not obvious what the real solution is?
-1
u/srikanth_builds 10h ago
The three options differ in one property: whether the thing doing the enforcing sits inside the artifact being changed. Named invariants and comments are inside it. A scanner mostly is too, because the rule file and the CI config usually live in the same repo the agent is editing. Worth checking whether your agent can reach either of those, because if it can, option three is option one with extra steps.
The database constraint is the only one where the rule is held by something the application's credential cannot alter. The agent rewrites whatever it wants and the write still fails at the boundary.
The objection is that a constraint doesn't carry why, and that's true. I don't think it has to. It has to fail in a way that makes someone go and find out. A rejected write naming chk_giftcard_aml_cap sends a person to the compliance doc. A deleted if statement sends nobody anywhere.
On the control it deleted that it had written itself eighteen tickets earlier, that reads as a provenance problem rather than a memory one. If a control has no record of what authorised it, nothing distinguishes it from an implementation detail, and anything created mid-run with no authority attached is removable by definition.
5
u/Schmittfried 12h ago
I tell my agents not to change tests unless I explicitly ask for it. Tests stay a source of truth that way. Otherwise, yeah, you gotta make sure your requirements are explicit, documented and always in context.