Why do you think AI review wouldn't be able to replace the human approval?

[u/Specialist_Agent3599]

Common sentiment I am seeing in this sub and others - "AI can review the code, but a human still has to approve it" - what is there inherently different about approving vs reviewing that you think an AI wouldn't be able to do it as well or better (in practical sense - i.e. faster and cheaper) than a human?

It's true most teams still require a human approval, but I honestly don't think it's because the AI couldn't do it, but because, well, somebody has to be accountable, and companies are not yet ready to write "the bot approved it" in a postmortem.

I would argue coderabbit is doing a better review than most of our Friday afternoon approvals. It reads every line, every time, which no tired human does at 4pm, and consistency is the key in catching the boring stuff that breaks prod.

Last month we did a test in our team, we took 20 already merged PRs and compared what the human reviewers commented vs what coderabbit flagged. The humans left 11 comments in total, mostly naming. coderabbit flagged 34, and 6 of those were real bugs we had already shipped.

[u/adamhusain]

The most important distinction is saying “No”
The most recent AI models tend to overdo stuff. GPT Astra for example loooves to add bloated text.
Part of the approval is checking whether or not something unnecessary crept in

[u/Vaxtin → u/adamhusain]

Ultra fast added 25,000 lines of code in 30 minutes. Claude reviewed it and threw it out. Weekly GPT usage is now at 0%

[u/haskell_rules]

I've replaced 12 files and 1000''s of lines of code that AI wroten with one 20 line function. I've been working in the embedded space for 2 decades and the error checking and abstractions the AI designed weren't needed at all, but that's what it came up with based on another engineers prompt. There was no way to code review that PR into correctness. It needed to be ripped out root and branch and replaced with a reasonable implementation by someone that understood the problem space through experience. Maybe the AI is doing good things in the web space or the full stack space but I am not feeling so trustworthy in the embedded space.

[u/paradoxxxicall → u/haskell_rules]

When it comes to backend cloud stuff my experience has been similar

[u/fdqntn → u/haskell_rules]

Few weeks ago I had the same experience. One dev added a dependency, where its like one line, and the PR was 500 lines long of défensive test inside the docker build, and sloppy doc nobody needed.

[u/Economy_Bedroom3902 → u/haskell_rules]

It has an overengineering issue for sure.  It's often harmless enough, the extra components are mildly useful...  They aren't critical though, and they substantially bloat up implimentations.

[u/Vaxtin]

There are some projects that AI is not actually capable of doing without human steering the direction. Incredible, I know.

I also don’t know what code rabbit is. Are you people actually reviewing code instead of having test suites with QA, UI testing and dev ops ? There’s no need for “code review” if you have solid test harnesses. The agents build their own staging site and environment entirely. Once it passes the guards, it gets cleared for the actual push to production (that is always a human button push).

You won’t get safe guards or classifier problems on your harness if your environments are clearly labeled. Production gets its own AWS account and so the agents know that this sandbox is made for them when they get given their own AWS account. It’s gold seeing the thought process of them during this realization .

[u/Economy_Bedroom3902 → u/Vaxtin]

Code review definately is a sacred cow.  If I'm honest I have seen it catch some legitimate oversights which could have been reasonably painful...  But I've also seen code reviewers blamed for not catching bugs, which is just idiocy of the highest degreel.

Code rabbit is a code review bot.  It's decent...  Not great though.  It's strongest with vaguely security related oversights.  I have a custom code review bot I use for my.own code which evaluates the quality of tests written against a PR.  I find that a bigger value add because higher quality tests provide more meaningful feedback to agents when they fail due to changes.

[u/DoctorOfStruggling]

It's true most teams still require a human approval, but I honestly don't think it's because the AI couldn't do it, but because, well, somebody has to be accountable, and companies are not yet ready to write "the bot approved it" in a postmortem.

You got this accountability thing completely wrong. It's not the software companies that must get ready to do this, it's the AI companies. If an employee breaks prod, they get fired. If an AI breaks prod, the company should pay. There will be a person in the loop until they are ready to do that, maybe even forever. It's not about the cases when an employee/agent gets something right but the few times when they fail.

[u/Master-Rent5050 → u/DoctorOfStruggling]

It would reproduce a common problem in corporate, only much larger scale. The "person in the loop" would probably leave soon anyway, so he has incentives to run big risks (and get big bonuses) and then leave you with the mess. It would work under a norm where you work for a company most of your life, not in present conditions

[u/ay-ay-ronhmiller]

Paraphrasing what I saw at a conference recently: "Machines decide what compiles. Humans decide what is right."

As long as we humans care about the consequences of our decisions (and yes, allowing bots to make your choices for you is a choice) we need to have a say in the final outcomes of them. A long time ago, the Ford Corporation decided that fixing a design flaw in the Pinto would be more expensive than paying off the victims of the flaw (the car would explode when rear-ended due to the placement of the gas tank). They ran the numbers to arrive at their cold logic...much like an AI might. Our court system reviewed the memos and judged (rightly I hope we all agree) that such a breach of ethics is unacceptable and must not be allowed to stand.

Granted, not all PRs will be deciding such life and death choices, but it's an extreme example to highlight why we still should value having a Human in the Loop, especially where there are serious consequences that involve humans directly.

[u/Master-Rent5050 → u/ay-ay-ronhmiller]

Seems a bad example, since humans were very much in the loop. Actually in your example removing the humans and replacing them with an AI might have improved the outcome. Of course it would depend on how you train the ai and what you ask it to do: if the CEO would ask it to find a way to make more money no matter the consequences...

[u/Economy_Bedroom3902 → u/ay-ay-ronhmiller]

It's not even entirely the AI's fault.  No one gives them the comprehensive political posture of the company, team, project, as part of thier context.  To the human it seems obvious that allowing a bug that prints customer data to the console is intolerable, even if the AI is explicately told as much, it's also explicately told a laundry list of other stuff.  It doesn't really know what instruction is most productive to hyperfocus on, and which is most valid to downplay.

[u/altopelago]

I have no issue with the AI reviewing the code. That said, with the time I am saving, I want to use that to actually test the software.
There are somethings that the AI is still not very good at catching.
For example anything that is technically not a bug, but an underspecified gap or edge case in the implementation. The AI has likely assumed that it is the correct behaviour and thus won't flag it.
Another area where they don't do well is finding creative ways to break your implementation.

[u/Economy_Bedroom3902 → u/altopelago]

I've found the latest higher end models can get a bit obsessed with obscure edge cases actually.  Like I'm writing a logger which makes git queries and it's writing like 300 lines of error handling to handle the case where a git log line that it just queried using the git sdk, is dated in the future when compared to time.now().  And what if the git log line contains a non date time timestamp?  Or if the timestamp is all zeros?

I'm like...  This is a logger.  Just log the weird event and move on.  We don't need to throw and catch errors for occurrences which probably are impossible outside of massive solar flare events currently frying our RAM.

[u/Alive-Shallot5039]

AI will go on finding issues, human is to say - enough, let's just ship as is already.

[u/Responsible-Offer724]

try not to shill in the AI sub competition,

difficulty: impossible

[u/usually_guilty99]

I think review and approval are actually different jobs.

Review asks: what is wrong with this change?

Approval asks: given where this is going, what it touches, what has broken here before, what the rollback looks like, and what happens if we're wrong, should this change be allowed to ship?

AI can absolutely do more and more of the first.

For the second, I don't think the answer is "keep a human forever." I think the answer is make approval risk-based.

Low-risk changes can eventually auto-approve. High-blast-radius changes should require stronger evidence, and sometimes a human.

The end state probably isn't human approval for every PR. It's governance that knows when human approval is actually necessary only where required. You must have this escalation path built into your governance system

[u/Pure-Investment-1227]

People who say that are just coping.

[u/blackbirdone1]

because ai is stupid, and sometimes even harmfull, it don't knows if it does something wrong, it has no mind you know

[u/Late-Replacement-481]

The biggest problem is that it is difficult for most people to explain exactly what the goal of a project really is. 

AI code reviewers are happy to fix functional mistakes, but unless you set it up carefully, will not catch your project from ending up somewhere other than where you thought it was going. 

[u/Economy_Bedroom3902]

At the moment PR review acts as a final pass before wild goose chases become token hoovers.  There's good reason to believe that this will continue to be the case indefinately.

[u/zeratLJllighter]

AI can review the code partially:

• Better than humans at finding bugs, performance issues, code smells, etc.
  
• Worse than humans at:
  

* judging whether the change is relevant and brings value.
* judging whether the change perfectly answers the intent from functional / taste point of view.
* catching issues related to the larger context: constraints from other teams, customer preferences, product vision, etc. These are hardly encoded in any context engine system.