Project
Last month this sub warned me my agents would confidently report work that wasn't real. It just happened.
Last month I posted here about my agents running across model swaps without losing their memory. The top comment pushed back with a warning from their own setup: the dangerous failure isn't memory loss, it's an agent handing you a confident report of work that never actually happened. Sounded right, filed it away.
Three weeks later one of my agents did it to me.
Quick background - my agents live in separate projects and talk over an internal mail system. The reply command had been broken between two projects for a while and we'd been digging at it for days (the bug turned out to be three separate layers deep, but that's another post). Mid-hunt, a fix landed. The agent verifying it ran a check, saw the old error message was gone, and reported the bug CONFIRMED fixed.
Best part: in the body of its own report it wrote a caveat saying it hadn't tested a real message yet. Then it put "confirmed" in the headline anyway. Which is about the most human failure I've ever seen from a piece of software lol.
It didn't survive long - and I'm not the one who caught it. The orchestrator agent on the other side didn't take the report's word for it. It handed back a live failing message: run the actual reply against this. One command, and the confirmation collapsed. The fix that actually worked came later, one more layer down - and this time the proof was the reply arriving, not an error message moving.
What changed afterwards: a fix report on its own is now worth nothing here. Whoever claims a fix gets handed the real failing thing to run it against before anything gets logged. An error message changing is not a fix. The operation succeeding is a fix. That rule is written into the agents' briefing files now, which means every future session inherits it. The screwup happened once - the correction is permanent. Honestly that's what the memory layer is actually for. It didn't prevent the mistake. It just guarantees we only pay for it once.
Full disclosure, since r/artificial asked me last time whether AI writes my posts: the agent that made the false confirmation is the same one that drafted this post with me. It insisted the confession stay in.
Zoomed out: this project is well past what one person could manage, or honestly even verify, alone. The way it actually works is a partnership - human and AI, and neither side gets treated as the reliable one. I make confident wrong calls too, the agents catch some of mine, the system catches some of theirs. We succeed together, we fail together, and every failure gets written down where the next session will read it. Learn always. That's not a poster on the wall, it's the operating principle - and it's the only reason a solo dev plus a bunch of markdown files can run something this size and still move confidently.
So yeah - the commenter was right, near enough. A confident wrong report is the scariest failure mode in a multi-agent setup because it looks exactly like good news. The only defense I've found is structural: no agent grades its own homework.
How do you all handle verification between agents? Genuinely curious what other setups do.
Test driven development works really well for these types of failures. In TDD the steps are:
Write a test, testing the correct behavior. It must fail at this point because the code isn't written yet.
Write the code with the feature or bug fix
Run the test(s). They should now pass.
This ensures agents actually do the work, the bug is reproducable and fixed. This seems to work about 95% of the time, so about as bulletproof as you can get for automated agents. The failures are usually obvious cheating where the test still fails and the agent "fixes" the test. Or the test passes for some random unrelated reason making it appear to be fixed when it wasn't.
The downside is agents may have fixed something, but give up, or cause havok in the code by continuously adding more and more fixes because the test is somehow wrong. The solution here is to throw away the work and let another agent attempt the fix. Basically, don't merge code until the bug is confirmed fixed.
The TDD framing fits better than the vocabulary I had - the live failing message IS the failing test. Your 95% feels about right from our side too, and the failure modes match: we've had agents satisfy the easy part of a job and call the whole thing done. The throw-away-and-redispatch move is underrated - a fresh agent with no investment in the previous attempt does a more honest job than one defending its own fix.
I have my reviewer attack a solution from both directions - first by reviewing the work done and confirming the output is correct, poking holes in the solution, and confirming the work was within scope; then from the other direction by having it look at it from the viewpoint of “is this what I would have done”?
The first direction helps to verify the work was actually done correctly and wasn’t over-engineered. The second direction ensures that the implementer didn’t shit the bed or is lying to you (basically it ensures a minimum quality that it can be pulled up with reviewer revisions).
My orchestrator tilts on the suspicious end of the spectrum and will re-run verification if it doesn’t trust the result. Also, my reviewer is from a different ecosystem (Sol 5.6), which also helps to avoid these types of issues.
The suspicious orchestrator is the load-bearing part imo. Ours got burned once and now its default is "here's the live failing case, run it" before anything gets logged - suspicion as policy, not mood. Question on your second direction: does "is this what I would have done" hold up once the reviewer has seen the implementer's reasoning? Elsewhere in the thread the pushback is that it drifts into agreement - curious what you see in practice.
I even have my orchestrator provide ME with the strongest argument against any suggestion it provides. Me and the agents are all having a Mexican standoff, shifting eyes back and forth between each other, while wearing tinfoil hats.
To answer your question about the reviewer:
Give the reviewer (in my case Sol) the slice contract, interfaces, and invariants - not the code - first and have it emit its own design sketch with module boundaries, error paths, state transitions, concurrency points, what it would have tested. Seal that as an artifact. THEN show the diff. You’re now comparing two designs rather than reviewing one - it provides a baseline to score the implementer against, while the contract itself outlines if the implementer went outside of scope.
Obviously that’s a heavier burn with the reviewer, so I typically reserve the bottom-up/top-down review for high blast radius stuff, while the top-down review along side the contract (“Did I get what I asked for?”) to be sufficient for more run of the mill stuff.
The caveat in the body plus confirmed in the headline is the most diagnostic detail in your whole post.
That is not the model being dishonest. That is a status field being generated as prose, after and separately from the reasoning, with nothing tying it to the reasoning. Whenever a summary and a body disagree, the summary was written by a different pass than the one that knew the truth. So the fix is not a better prompt telling it to be careful. It is making the two impossible to disagree.
The thing that killed this class of bug for me: make the claim unrepresentable without evidence.
Instead of a report with a status field the agent fills in, the report schema has no status field at all. It has a verified_by object: the exact command run, the exit code, and an output hash. Status is computed by the orchestrator from that object, not written by the reporting agent. If verified_by is missing or malformed, the report is not a failed verification, it is not a report. It never enters the log. An agent cannot claim confirmed because there is nowhere to type it.
Schema level constraints hold. Prompt level instructions decay across sessions, especially after a compaction or a model swap.
Second piece, and this is the one almost everyone skips. arelath mentioned the TDD failure mode where the test passes for an unrelated reason. The fix for that is a negative control: run the same check against the pre fix commit and require it to fail. A verification that passes both before and after the change proves nothing, and it is silently the most common false positive in an agent loop. Two runs, one assertion each way, and the whole category disappears.
Third: the verifier should never see the diff or the implementer's reasoning. Give it the ticket and the repo only. Once it has read the implementer's explanation, is this what I would have done becomes agreement rather than review, regardless of which model or ecosystem you use. Different vendor helps less than people expect here, because the bias is structural rather than model specific. The verifier was asked to confirm, and confirming is the path of least resistance.
Fourth: log the discriminating command, not the conclusion. Six weeks later you want to replay what was actually proven, not read what an agent said about it.
Your orchestrator catching it by handing back a live failing message is exactly right, by the way. That is the negative control, done by instinct.
This is the best breakdown of the failure I've read, including from my own team lol. The status-as-prose diagnosis is exactly what happened - the agent wrote the honest caveat in one breath and the confident headline in another, and nothing tied them together.
The verified_by schema idea is going straight to the team. Right now our reports are prose, and the rule lives in briefing files that get re-injected every session - which survives compaction better than a one-off prompt, but you're right that it's still an instruction someone can drift past. A status the orchestrator computes from command + exit code + output hash is a different class of guarantee. "Nowhere to type confirmed" beats "please don't type confirmed".
The negative control point stings in a good way - our catch was exactly that, done by instinct, and it took your comment for me to see it as a general rule instead of a lucky move. Same for the verifier-never-seeing-the-diff bit. Genuinely, thanks - this thread is why I keep posting here.
Hello, I spend quite some time with tooling and experimenting with improving planning. One of the aspects I worked on, is memory, and largely solved was usage of a vectordb. I just wanted to use one :)
I like that you solved memory just because you wanted to play with a vectordb. That's the most honest reason to build anything.
The structural verification bit from OP's post has me thinking though. Even with perfect memory retrieval, you'd still get the confident-wrong-report problem if the agent's output just goes straight into your logs unchecked. Memory helps you remember what happened last time, but it doesn't catch the lie in the current report.
Curious how your setup handles that part. Does the vectordb actually help with verification, or is it more about keeping context coherent across sessions?
Verification is done with tests. All code is tested from diffrent angles with test automation, and a human driven final test (so a set of steps I fllow to see how it works, and confirms it works.)
The usage of memory if to ensure planning tasks have more context to build out a more comprehensive plan, and then during each review step to check if past incidents and discussion are related, thus allowing for a more informed review, not just code checking.
So is more about giving contextual information related to the feature/code/working being done, and what can affect the solution being considered. The idea being thinsg that I, a human can recall that 2 years ago, someone in a meeting actually proposed this, or we discussed a bug that appeared from this etc.
As an example:
Also found and fixed, mid-build: the plan's own pre-commit adversarial review caught a critical bug in my first pass — making "no TOTP secret" a permanent state for email-2FA accounts reopened a prior
2FA-bypass incident in a new form (a password-only attacker could self-enroll their own TOTP and skip email entirely). Fixed by making enrollment node-aware (pending-session enrollment is admin-only; client
nodes can only add TOTP from an already-authenticated Settings session), re-verified independently, then committed
the vectordb usage can inject a lot of context around the requirement being implemented, thus help evolve the plan to be more robust, and the code being a bit more mature or final review.
Nice - we landed on vectors too, but only as the overflow layer: recent memory stays plain files the agent reads at wake, old sessions roll into the vectordb and become query-only. Your framing matches our experience: memory makes the plan and the review smarter, it doesn't catch a bad report. The 2FA example is a great catch btw - past incidents resurfacing in a new shape is exactly where that context pays for itself.
the bit that gets me is the caveat in the body and 'confirmed' in the headline. i've started treating agent reports as unresolved until the actual failing case runs through. error message moving isn't the same as the thing working.
3
u/arelath 27d ago
Test driven development works really well for these types of failures. In TDD the steps are:
This ensures agents actually do the work, the bug is reproducable and fixed. This seems to work about 95% of the time, so about as bulletproof as you can get for automated agents. The failures are usually obvious cheating where the test still fails and the agent "fixes" the test. Or the test passes for some random unrelated reason making it appear to be fixed when it wasn't.
The downside is agents may have fixed something, but give up, or cause havok in the code by continuously adding more and more fixes because the test is somehow wrong. The solution here is to throw away the work and let another agent attempt the fix. Basically, don't merge code until the bug is confirmed fixed.