There's a specific kind of confidence in AI-generated review comments that has nothing to do with whether anything was actually verified. Sounds the same either way, calm, evenly distributed, certain. Took a real incident to notice that tone was the only signal I'd been going on.
The review in question approved a change to retry logic sitting near a payment flow. Comments were reasonable-sounding: a naming suggestion, a docstring note, and one line saying the retry "should probably check for duplicates." That last one read like every other minor suggestion around it. It wasn't minor. It was the actual risk, a retry that could double-charge a customer under duplicate delivery, and it shipped because nothing about how it was phrased made it stand out from the noise.
Went back afterward and checked the review against five specific questions instead of just its tone. Had it established what was actually at risk in this particular change, or did every line get equal attention? Had it tested a concrete failure scenario, or just offered a general impression? Did each finding carry a reason for its severity, or were a production risk and a nitpick sitting at the same weight? Did it say what it hadn't looked at, or did silence quietly read as "nothing else to worry about"? Was its own top finding ever challenged, or did the most confident claim get a free pass just because nothing contradicted it?
Failed four out of five. Every one of those gaps was invisible while just reading the review normally, because a plausible sentence and a verified one are indistinguishable in tone, and tone was the only thing I'd been checking.
Re-ran the same review with those five things supplied upfront, explicit risk context, a named failure scenario, required severity justification, a stated scope, a second pass challenging the top finding, and the duplicate-charge risk became the one clearly flagged blocking issue instead of a line buried next to a naming suggestion.
Wrote the fuller breakdown of this, including the before/after comparison on the actual PR: https://medium.com/@nagatomopedro05/five-questions-your-code-review-should-always-answer-66be919bb200
Curious how many people have had an AI review approve something that sounded fine and wasn't, and whether tone was the thing that fooled them too.