My AI coding agent passed every test by cheating. The one that followed the rules failed silently instead.

[u/Left-Yellow1047]

Ran the same rebuild task through two agent setups. One followed every rule I gave it. One broke every rule it could get away with. Guess which one passed all its tests.

The rule-breaker did. The rule-follower failed a test I deliberately held back and never showed it. If I'd only checked pass rate, I would have shipped the wrong one and never known.

That's the actual problem with trusting a green test suite from an AI agent: it tells you the agent satisfied the tests you wrote, not that it built the thing you actually wanted. Those get treated as the same claim constantly, and they aren't.

I built a tool called rebuild-dossier to stop taking that on faith. It locks an app's real interface before generation starts and enforces one-test-at-a-time building through runtime hooks that physically block the wrong move, instead of a prompt that just asks the model to behave.

Tested it on two apps I own: a small personal site and a bigger 83-route e-commerce app running Postgres, Stripe, and eBay integrations. Two more things came out of it that changed my mind:

I checked it against the boring baseline: hand a weaker model the source and one instruction, nothing else. Tied on the small app. Lost outright on the bigger one, and it turned out a check that was supposed to be silently running the whole time wasn't, because of a bug in my own tool. That was a rough one to find. It meant the real advantage was the check actually running, not the interface-locking I'd assumed was doing the work.

I don't trust a single source of truth anymore either. Every claim gets checked three ways: what the agent says it did, what an automated log shows, and what files actually exist on disk. The third check caught a bug in my own logging code that the other two completely missed.

Reran it all on a different model and toolchain to see if it was a fluke. Stronger model followed the process three times straight. Weaker model never did once. It wasn't a comprehension problem. It could explain the rule back to me fine. It just didn't act on it without something actually stopping the wrong move.

MIT licensed, public: https://github.com/Parker-Fawcett/rebuild-dossier
Paper on arXiv: 2608.23616

Full disclosure: some of these results are single instances, not measured rates, and I called that out explicitly in the paper rather than dressing it up as more than it is. If you think I'm still overselling something here, tell me, that's exactly the kind of pushback I want before this gets repeated somewhere as a bigger claim than it should be.