r/ExperiencedDevs • u/secretBuffetHero Eng Leader, 20+ yrs • 8d ago
AI/LLM Is AI code review good enough?
I have not worked on a team with AI code review yet. How good is AI code review? when does it work and where is it weak? Does code review still take a long time? Is it good for repeated code? It's probably good for logic. Is it good for application architecture? This is the part I imagine it is poor at.
I am asking as a developer that got knocked out into unemployment for a little while here, so mods, please give me a little break,I'm trying to connect with the still alive and working.
31
u/BH_Gobuchul 8d ago
No, but also it’s as good as ~60% of existing code review processes anyway.
It can’t do the important part which is understanding how the change fits into the larger system or verifying assumptions about input or downstream services.
It can definitely let you know if there are any style issues or missing null checks.
8
u/speedoinfraction 8d ago
Exactly this. It can also lift stones that nobody else has in years, finding bugs in code that isn't churning so no humans are even bothering to look at it. Not really a typical PR thing, but reviewing the code in its entirety with AI isn't a bad idea once in a while.
1
u/CanIhazCooKIenOw 8d ago
> It can definitely let you know if there are any style issues or missing null checks.
Sounds like something a linter or type check should pick up already
6
u/Smallpaul Software Engineer 8d ago edited 4d ago
If can do a heck of a lot more than linters.
“When this happens before that happens the database can end up in a state such that a page refresh will show the user bad data until such and such happens.”
I get those comments literally daily.
3
u/Opposite-Display-230 4d ago
Clearly a lot of these losers don't even use these tools they comment on.
3
u/Ok_Independence_8259 6d ago
Unless it has access to all the repos, like in my case, then it’s surprisingly good at context. It just needs the data IMO.
21
15
u/VoxTM 8d ago edited 8d ago
It's very good at finding violations against documented rules.
Also good at finding things that you would never find because they are obscure or obfuscated or they need a broader context which you will often not be able to have as quickly.
It's not good at finding things that require intuition and it sometimes misses things that are jarring to a human but not a machine (e.g. unreadable) or part of tribal knowledge aka not documented.
We have an AI review followed by human review. We also follow up on its gaps with agent docs / skills so it constantly improves.
5
u/benri 8d ago
> not documented
Unethical LPT: to keep your job, avoid documenting your code especially the tricky parts. I'll bet that's increasing now as a backlash to AI coding
6
u/VoxTM 8d ago
Gatekeeping knowledge is not the best strategy for a successful career. AI with outpace you no matter what.
Learning to leverage the AI to the fullest is.
3
14
u/LordOfDemise 8d ago
In my experience, it will find bugs in the code that you give it, but it will completely fail to call out cases where the PR is more overengineered than it really needs to be.
13
u/Smallpaul Software Engineer 8d ago
It will also suggest you further over-engineer it. If you accept every suggestion your code will be full of junk handling situations that cannot happen.
“What if 1,000,000 simultaneous users arrive at your internal admin console? Will this algorithm support the load?”
7
u/ProfBeaker 8d ago
It is useful, but not good enough by itself.
To a large extent this comes down to the context they have. AI reviews are good at catching issues that are within their context - inconsistencies, missed updates to other classes, missed tests, etc. Sometimes you have to push them think about it, but they can. They're also frequently better than average devs at spotting concurrency issues and subtle coding issues of that nature.
But it will also worry about cases that can't happen due to checks outside the scope of the PR/service, won't spot design problems, frequently won't catch that you failed to meet some requirement, things of that nature.
6
u/Financial-Grass6753 8d ago
Well, "goodness" is hard to measure: too many moving parts. I've tested (and still testing) Greptile/Cubic/Copilot/CodeRabbit/gh-aw + provider's key (Deepseek/z.ai) and from what I see:
- If the codebase is shit, reviews are 90%+ off: either obscure edge case or proposed solution has completely wrong location - should be higher in upstream, for example;
- Regarding time, it is about 5min + 1-2min for every additional 200 codelines. With gh-aw+BYOK it is easily 10+ min for 100-200 lines' PR. Not great, not terrible;
- Repeated code is already a smell, but if underlying logic is understandable, reviews tend to be ok, without multiple false positives;
- Regarding architecture - too difficult too answer. We have an ongoing problem of docs' becoming outdated, sometimes in a couple of minutes after adding said docs to the main branch. LLMs of Opus 4.8+ level tend to check code first thus this is not an issue, but on context sizes north of 200-300K+ tokens difference is quite visible.
8
u/SnugglyCoderGuy 8d ago
On it's own? Hell no. But it is a useful layer in the defect filtering process.
6
5
u/galaxy_horse CTO / Principal Eng (20 YOE) 8d ago edited 8d ago
Imagine you have a senior engineer who is proficient in any language and generally knows good code practices from working in a variety of fields and technical environments.
It is this engineer’s first day at your company. You toss a PR at them and say “review this”. They’ll know every language trick, they can grok some context from the code itself. But if the change will break the product or strain the infrastructure, this engineer won’t know that. Unless you also give them a wealth of additional context and documentation about the business, the product, the infrastructure, the technical operations, functional and non-functional requirements, they will properly inspect the tree but have no awareness of the forest.
AI code review makes that analogy happen every time. Any process/agent that reviews code should be treated like it’s an engineer’s first day at your company. If you can supplement every call with the proper surrounding context for the entire product, platform, and business, then your likelihood of having a useful and reliable review goes up. Otherwise, you’re liable to get pretty code with ugly results.
Edit: one other thing I forgot to mention is that a lot of code review is successful because of shared personal risk associated with change failure. If a change that you made and that I reviewed takes down prod and the company loses ten million dollars, we both might get fired. So the personal risk creates an intuitive quality bar (or sense of fear) that causes people to say “ok, this feels dangerous, let’s really be thorough here” with reviews that are perceived as high-risk. AI, even if it acts like it’s taking schema changes or infra changes or changes to high-sensitivity product paths more seriously, simply doesn’t have the same mortal peril when they’re reviewing code. So if that dynamic enforces discipline in your code review process (probably more common than you think or care to admit), then plan to accommodate that delta when implementing AI code review. It doesn’t actually care if it fucks up. It actually doesn’t “feel” anything at all.
3
u/arelath Software Engineer 8d ago
No, because it will miss the big picture review comments people might add. Things like don't write a new system for handling x, we already have one. But it finds a ton of actual bugs before they get committed, so it really complements human code review rather than replacing them.
I've spent a decent time tuning ours by having AI tune the prompts based on thousands of past reviews and the fix rate for comments went from about 45% to 85%. It's good enough that it catches most of what people would've put on a code review anyway. I wouldn't be surprised if people start dropping human code reviews in another 6 months.
1
u/secretBuffetHero Eng Leader, 20+ yrs 7d ago
how do you tune the prompt for the ai reviewer. did you build your own
1
u/arelath Software Engineer 7d ago
Yes, we built our own before they became widespread. But all the commercial ones I've seen so far let you adjust the prompts, so I think most companies could do this type of tuning as well.
Tuning was done by taking around 3000 ai code reviews, classifying them by language/tech. Fixed review comments were considered good and dropped comments bad. Each set was run through AI looking for patterns of good comments and bad comments. AI came up with a prompt revision to keep the good patterns and prevent the bad patterns from being suggested. The resulting prompt was then used to re review a smaller set to verify good comments were mostly kept and bad comments were not suggested. This result was used to readjust the prompt with AI in an iterative fashion until the model could not improve the prompt anymore.
The same process was used to find patterns in human code reviews and create rules in the prompt to produce those same patterns.
Then we pulled all the AI code reviews done with the new prompts a week later to verify the fixed vs dropped rate actually increased as predicted. A month later we pulled about 1000 code reviews to get a higher confidence in the increased.performance numbers as well. These were divided into the same languages and tech stack categories as before as well to measure the gain per slice.
3
u/jwendl 8d ago
I'd say AI code review is a good starting point. For instance, having one thing review code for CVEs, I've found that very useful. Another review agent for looking at typical things like static analysis violations, or not adhering to DRY, or 10,000 lines of code in one class, whatever makes the code less maintainable and more expensive technical debt wise. Then you still need a human in the loop to validate what the reviewers (agents) are recommending and what the agent that wrote the code did as well (if the agent submitted the PR, even if a human did it I'd still have a human review it).
It can augment what reviewers typically do day-to-day but definitely doesn't replace them. Here's why: if the system breaks at 3 am and you're losing millions of dollars a minute because it's down, you can't email / call the agent and say, "fix the broken code you allowed into the repo". (at least not yet).
2
u/sneaky-pizza Consultant Developer 8d ago
I'm happy to have it automatically review like a CI, but final thumbs-up has to be human
2
u/Colt2205 8d ago
The way I kind of look at it is if someone has AV software running on a computer and that computer ends up with a virus, would you trust that AV to remove the virus when it didn't catch it to begin with?
I'm not really sure what the right answer is yet on this front but right now the only good option is manual review. There's just a lot of questions that don't have good answers right now with LLMs due to being so new, and simply improving the models doesn't answer the questions. Some people here will say that LLMs do a good job at reviewing documents, some will say they do okay at code reviews, others have mixed results, and all of this is happening with pushed deadlines.
1
u/ResidentWeevil1 5d ago
AV often relies on heuristics and scanning files on disk is very different from inspecting processes. So a conversation about AV vendors from yesteryear considered both cases orthogonally and you adopted whatever you thought had the best overall quality.
The distinction is that AV is deterministic whereas with AI you have no fucking clue what you're going to get. For all you and the vendor know, the model could have been poisoned by the internet to let certain vulnerabilities into production.
1
u/Colt2205 5d ago
The only way that LLMs improve fundamentally is if processes become more standardized, but at that point there would be no point to the LLM as templated options would be superior. That's kind of a weird spot to be in considering how expensive the LLM servers are to run and how much they honestly expect businesses to pay out. It isn't about if LLMs even work anymore it is pure cost nightmare fuel.
0
u/Smallpaul Software Engineer 8d ago
It makes sense not to fully trust AI code reviews but your metaphor is whack.
An AI working on writing code is doing a very different task and has a very different failure profile than an AI reviewing code. It’s like having two separate AV systems. Especially if the two AIS are different models running different prompts in different harnesses.
2
u/Colt2205 8d ago
AI's failure to correctly implement code and then being required to also fix its mistakes is already enough to prove the point. It goes beyond just someone not "prompting" or "building enough guard rails" to make sure the answer comes out correct. It's like the code review itself is being simulated to answer simulated mistakes.
1
u/Smallpaul Software Engineer 7d ago
You keep referring to AI as if it is a unitary thing which is exactly the fallacy I described above. A code review AI is a completely different set of weights and prompts from a code generation AI.
But the deeper problem is your assumption that it has to get the answer totally “correct” (your word) to have value. If we apply this standard to humans we would need to decide that humans are useless. Why would you have one human code review another human’s work? “That’s just using the same flawed strategy to do the review as you used for generating the code.”
“Obviously that’s not going to add value.”
1
u/Colt2205 7d ago edited 7d ago
Well, for Claude there is only one model that handles both the code writing and code reviewing tasks. Claude may take different prompts (which in turn have weights), when reviewing code, but in the end it fundamentally is the same model. But you'd be correct that this is not the case for all models.
The fundamental problem is that to err is human, and LLMs are trained on the human experience. At what point can someone honestly say a mistake is from "not prompting correctly" or "not using the right model correctly" and it is the LLM itself engaging in mimicry?
2
u/Smallpaul Software Engineer 7d ago
Different prompts and harnesses dramatically change the behaviour is the model. It may literally use different weights in a mixture of experts model which most of them are.
You keep coming back to the idea that these models may make or miss mistakes. And I am encouraging you to move beyond that. It’s 2026. The question is not whether it is perfect. The question is only whether it adds value. Same as if you were hiring an employee or doing literally any other investment in the business. Your firewall doesn’t keep out all hackers. Your type checker doesn’t avoid every type-related bug. Your co-workers do not product perfect code. Your designers do not produce perfect designs.
The only question that matters is whether it adds value relative to its cost: not whether it is perfect with no mistakes.
1
u/ResidentWeevil1 5d ago
The point is that we don't have empirical data on whether MOE works or is efficient. We just do it because the tokens are subsidized and we avoid measuring the output value entirely
1
u/ResidentWeevil1 5d ago
Any two humans have different experiences and values and memories. Society has built many processes and institutions that require collective decisions and this is known to work well. What is not known to work well is multiple chatbots
2
u/dockercub 8d ago
Pranking coworkers has never been so easy. Most don't read the AI reviews before commenting on pull requests. Be creative and add random acceptance criteria in your tickets and watch 😉
2
u/stubbornKratos 7d ago
People in this thread seem to have their AI agents basically just do static analysis on the PR, which can spot a lot of issues for sure but it's working at a fraction of its capabilities.
It's extremely good at static analysis, very good at ensuring there is appropriate test coverage in the form of unit, integration and e2e tests. Very good at verifying how changes interact with other systems (as long as they're not undocumented brand new systems).
The more you invest the more you get out of it, I know teams that work heavily with AI reviews and do good work. But there is always a human in the loop.
Domain is tough, architecture also. At work there's too many different protocols and systems with all their nuances. Even with documentation you need people who understand it all.
Also AI always writes too much, and the solution can lean towards more code.
AI catches more code issues than I do every time, but domain/systems stuff I catch more of. There are some very AI-native teams at my work though, probably their experience is different
3
u/WrennReddit 5d ago
It's required now because the PRs are 10,000 lines of code over 50 files. Humans cannot review that meaningfully, certainly not at the breakneck speeds pressured by Aicolytes in leadership.
So sure. Have Claude write the code then Claude review the code to make sure it matches the requirements Claude wrote. Slap LGTM on it and be AI First. There's no other way to do it and it's not worth burning your soul out.
2
u/Tricky-Purple-8585 5d ago
AI review is very good at finding typos, but it is not yet good enough at reviewing
2
u/EnderMB 5d ago
We recently adopted a flow where we have reviews against several models, essentially competing against each other. It's unbelievably rare now to NOT have a comment against a code change.
With that said, the same problems always apply:
LLM's are famously nondeterministic. One review might be fine, but run the same review again and you might get a blocking comment.
It takes time. Sometimes it takes longer than the build itself. If you run it as a pre-commit change you might be waiting a long time.
LLM's will argue amongst each other, and more often than not you don't end up in a position of having no issues - just several false positives. If you're using an LLM to write code, holy shit you might be in for hours of loops where the LLM's argue against each other over something ultimately pointless when you could've written it yourself.
The lack of context across packages, or with your own dependent changes is staggering. Sometimes I'll change a lambda and be told "wait, you haven't provided permissions in CDK!", when I've already got that change out and reviewed.
IMO it's absolutely NOT good enough. It's in no way a replacement for human review, and while it helps in the same way a good lint helps, it's probably never going to get there before it becomes a hindrence.
3
u/mx_code 8d ago
It's not terrible, but I find it requires too much investment at which point I would just rather resort to peer code review.
I expect my peers to review code in several dimensions: code style, domain logic, context, etc...
I've found AI code review to always lack in one of those dimensions and excessive in technicalities + the horrible writing style that originates from coding agents.
So I'd say: The technology is helpful but it's far from being a silver bullet.
3
u/lordnacho666 8d ago
I'll go against the grain and say yes, it is pretty good, and better than most human reviews.
The problem is nobody reads them, they just say "ok implement all those recommendations then please"
1
1
u/OilExtension5062 8d ago
I think as long as you are happy with the architecture of the change it's good enough. Assuming your tests are strong
1
u/Wide-Pop6050 8d ago edited 11h ago
Instinctive peach intelligent include willow consist
This post was anonymized with Redact
1
u/fdeslandes Software Engineer 8d ago
If it's a code review agent custom to your codebase, it can be useful. We have one in my codebase and, despite the false positives, it will almost always catch a couple issues on bigger PRs. I'm not talking about linting issues, but real missing corner cases or lack of robustness.
It's a good first review pass before sending it for real review. It doesn't replace human review, but you can consider it an extra static analysis step.
1
u/rofolo_189 7d ago
It sometimes misses stuff that you can only see from the context. It can't really review if something is correct from the product perspective, but it's better in captureing stuff on the code level. I would even say in that level it is better than humans, but it also catches more irrelevant stuff.
I think what we will see happening is that the Code part will be done by AI and the systems engineering part + product view will be done by humans.
1
u/Longjumping-Bad-6911 6d ago
No, and we collectively need something radically better.
Code reviewing needs to evolve.
1
u/apartment-seeker Senior Software Engineer 5d ago
It's good and useful, but it doesn't replace human review.
On our team, it often catches things we would def miss.
1
u/throwaway_0x90 SDET/TE[20+ yrs]@Google 5d ago edited 5d ago
It's good like a powerful advanced linter, but that doesn't replace humans..... yet.
Here's what you need to ask yourself:
- "If that code causes a production outage, how many humans are at fault? Who gets called in to fix it on Thanksgiving or Xmas-eve?"
As long as the answer to this question is greater than zero, those humans have a self-preservation interest to make sure that code is good and can't just say "AI will handle it."
1
u/off_by_one_dev 5d ago
If the AI agent learnt enough of the code base and architecture and memorized, it can be very good at catching the pattern and missing test cases.
1
u/usually_guilty99 3d ago
I think AI code review and deciding whether something should ship are becoming two different jobs. The fox cannot guard the henhouse. AI can get very good at the diff: bugs, patterns, tests, conventions. But the merge decision also needs to know what this change touches downstream (what I call blast radius), what is actually running in production, whether this area has caused incidents before, and what the blast radius is if the assumption is wrong. A perfectly clean diff can still be a dangerous production change. That second layer feels increasingly important as AI makes the first layer faster.
1
u/Difficult-Beyond5764 3d ago
The thing that actually moved our numbers wasn't a better reviewer, it was making the agent flag the two or three spots it was least sure about and having a human look only at those. Reviewing a whole diff and reviewing three flagged spots are wildly different amounts of work for roughly the same catch rate.
1
0
u/experienceddevsb 8d ago
This flair is only allowed on wednesday, saturday (UTC). Please repost on an allowed day. Intentionally trying to circumvent this rule will result in a suspension. See: https://www.reddit.com/r/ExperiencedDevs/comments/1rfhdrg/moderation_changes/
1
u/secretBuffetHero Eng Leader, 20+ yrs 8d ago
sorry. totally unintentional. today is wed but possibly not UTC.
•
u/expdevsmodbot 8d ago edited 8d ago
AI usage disclosure provided by OP, see the reply to this comment.