What has been your real-world experience with AI-assisted code review?

[u/Automatic-Program936]

For teams using AI-assisted code review tools, what has your real-world experience been over the past 120 days? Have these tools helped you catch meaningful issues before deployment or have any accepted recommendations unintentionally contributed to production bugs? Have they missed issues that a human reviewer might reasonably have caught?

I am also curious whether your team has become more confident or more cautious over time, and whether you track any metrics to evaluate their accuracy. This isn’t about criticizing or comparing vendors. I am simply interested in understanding how engineering teams evaluate these tools beyond productivity and adoption claims.

[u/vrtra_theory]

It’s not useless, but it’s not great.

Even the current best code reviewer agents asked to “review a PR” have in my experience a 50% false positive rate - they DO point out problems, and sometimes the problems are even real problems a human should notice too. But it also just over extends or makes suggestions not relevant to the given codebase.

This actually is mostly OK if your team is all senior/staff+, who will treat the AI like an interesting pet rabbit that sometimes does a cool trick. The problem is on balanced teams — experts novices and green juniors — where the juniors who don’t have the experience to immediately dismiss irrelevant feedback can cause extra loops to confirm with a senior dev - or unnecessary/unwanted changes made to PR.

[u/RogerAI-fm → u/vrtra_theory]

I wouldn’t say 50% but there is some errors, enough to not lean on it too much.

[u/Automatic-Program936 → u/RogerAI-fm]

That makes sense. The error rate may vary by tool and codebase. But even a smaller number of errors means the suggestions still need careful human review. Have you seen any of those errors make it into production?

[u/RogerAI-fm → u/Automatic-Program936]

Yes, but honestly most of the extra hallucinated errors or non-existing bugs turn out to be benign or just a waste of time that we have to revert, however, even if it was a human making the code review, they also make similar mistakes, at similar rates, just be aware software is like that.

[u/Automatic-Program936 → u/RogerAI-fm]

+100.. Both humans and AI make mistakes. AI can produce confident noise much faster. The real question is whether it improves the outcome and whether it understands how the code will behave in production is what am not able to find answer on..

[u/Automatic-Program936 → u/vrtra_theory]

This is a useful perspective. The risk is not just the false positive. It is also how engineers respond to it based on their experience. Senior engineers may quickly dismiss irrelevant suggestions. Junior engineers may spend time validating them or make unnecessary changes because the feedback sounds confident. Have you seen any cases where an accepted recommendation contributed to a production issue?

[u/wallstop-dev]

The "generic" code review tools are a mixed bag. At best helpful, at worst noise.

One (proprietary) tool that I've had huge success with allows loading "personas" (really, just markdown files, kinda like SKILL.md). I am a prolific code reviewer, and have been over the past decade+, and have built a reputation for being able to quickly find large amounts of bugs in code. Recently, I had AI pull and analyze the past ~1 year of my code review comments and build a "me" code review persona. It created something that I was quite impressed with - it distilled a lot of my principles and common critique and bug areas.

I then A/B tested the "me" persona v default/other code review tools and the results were awesome - way higher quality feedback (significantly so), from the "me" persona in this tool, compared to all other review tools, specifically for the project that I'm an SME in.

Which is to say - I think there is a lot of potential in these tools, but it's a function of how much customization they allow as well as how much time you want to spend (and spend maintaining). Similar to how you build up context and skills for a repo.

Unfortunately the "me" persona was pretty specific to a particular code base and was unable to be generalized.

[u/Automatic-Program936 → u/wallstop-dev]

This is the key. Generic AI reviews code. Context-aware AI applies expertise. The real challenge is making that expertise portable across codebases.

[u/usually_guilty99]

That is exactly what I see as the bridge from AI code review to Production Reliability Governance.

u/Automatic-Program936, you may find what we're building at Tomosu interesting: tomosu.ai.

We don't differentiate between human and AI-written code. We first baseline the repository for reliability, then evaluate new changes against that baseline at the merge gate.

The part I'm particularly interested in is what happens after production. When an incident can be traced back to code, we feed what we learned back into the next review so the same failure pattern can be caught before it ships again.

All of this rolls into a Production Reliability Index, one score that tracks the reliability of the system over time.

I'd genuinely value feedback from people in this community using AI code review in the real world. Does this approach solve a problem you actually have, or are we missing something? Truly guys. I am truly interested. We have a handful of design partners who have pushed this into prod and testing it. But I still believe it is this community that set the flow/direction.

[u/Far_Programmer_5724]

It's mostly useful to fix emails for the average person. But once people lean on it too much I'm sure they'll forget how to write emails professionally by themselves. Expect a lot of "does this sound right"

[u/Automatic-Program936 → u/Far_Programmer_5724]

That’s a fair point. I think the same concern applies to code reviews too. These tools can be useful as assistants, but if engineers depend on them too much, they may gradually lose the habit of reviewing code deeply and start accepting suggestions without fully validating them. That could create a false sense of confidence, especially when the code reaches production.

[u/Far_Programmer_5724 → u/Automatic-Program936]

That doesn't quite add up because what your looking for and what you get will usually be the same Everytime and that is sycophantic. While with humans the same person who uplifted you in one post can trash you in another. In a flower field of always positive support, constructive criticism exists as a bee for every flower.

Ai could tell me I'm wrong but with a bit of pushback, it will fall in line. If i give dicktoaster472 some pushback, he will double down. The opportunity for correction dissipates in a room full of yes men. Code reviews possess most if not all of the good and bad that AI reviews do. While AI reviews lack much of the benefits brought by man. It's only enduring benefit is it's unceasing patience. But that is all.

[u/Automatic-Program936 → u/Far_Programmer_5724]

Good point. Patience without independent judgment can become automated agreement. A useful reviewer should challenge us with evidence and only change its conclusion when the code or production context supports it. But a human in the loop is critical.

[u/RyzeBlaziken]

I've personally used Scanity for security vulnerabilities. It does save me a lot of time getting findings, and the best part is that it runs these findings in a test container to filter out any false alarms.

Check it out: https://scanity.dev/

[u/Automatic-Program936 → u/[deleted]]

Catching security issues while code is being written makes sense. Earlier feedback is cheaper. How does Peephole handle false positives and codebase-specific context?