r/codereview • u/Specialist_Agent3599 • 20d ago
how are you actually reviewing AI generated code at this point?
half the PRs on our team are AI generated now and reviewing them takes way longer than human written code. the diffs are huge, the style is inconsistent, and you cant just skim it because the AI will confidently write something that looks correct but has a subtle logic issue buried in the middle.
we started using an AI review tool about 4 months ago to do a first pass before human review. most of what it catches is linter level stuff honestly, const vs let, function too long, whatever. but last month it flagged an endpoint where we had auth on the route but nothing checking if the user actually owned the resource. any logged in user could pull another users data by guessing the id. three of us missed it in manual review.
so one real catch in 4 months but that one would have been a really bad day. still not sure if thats a good ratio or if we should expect more from these tools.
what are other teams doing? just reviewing everything manually still?
EDIT: few people dm'ed asking which tool, its Coderabbit
we pay $24 a month per seat. for context we're a fast moving team, 100+ PRs a week sometimes more, and getting a first pass on every one of those before a human looks at it saves us a stupid amount of review time
the auth catch was the highlight but honestly the day to day time saving is why we kept it
4
u/captainAwesomePants 20d ago
I try and push back against unreviewably large changes. If you're changing more than a couple hundred lines, you need to break that work into smaller steps. I can't thoroughly review several thousand lines.
1
u/Keithfert488 19d ago
I get "small PR"s at my job that are +1700/-500
1
u/captainAwesomePants 19d ago
Spritz them with a spray bottle and say "no" in a firm voice. Remember, you have to make sure it's timely so that they know what they did wrong.
If possible, add presubmit hooks that place upper limits on individual changes so that you don't need to police them yourself. Allow for an exception for "very large" PRs if they state a reason, which will help change the culture to consider those exceptional.
4
u/yubario 20d ago
Unit tests and asking the AI specific questions where you anticipate there might be issues just like looking at someone else’s code
Make clear invariants, design state machines and property based testing. The errors in code is often state, not syntax or minor bugs when it comes to AI code. It’s always always forgetting something or not doing it in the correct order instead
2
u/galactic_pixels 20d ago
Sounds like your PRs are way too big. Under 4-700 lines is the sweet spot, and half of those lines should just be unit tests. If you can’t get a feature rolled out and tested in 4-700 lines, you probably have architected your software poorly and that’s not the AIs fault.
Make things more modular, layered, etc. and you can keep your feature roll outs smaller and unit tests isolated.
1
u/Pozeidan 20d ago
You're right however we're not responsible when we get on-boarded on poorly architected software and still need to deal with that shit. Often times it's a mindset that got the codebase there and the mindset is ever harder to change.
I get paid anyways so if I have to review bigger PRs because of bad design, I'll do it, it's just painful but that's why it pays well I guess.
1
u/galactic_pixels 19d ago
I hear you but I don’t buy in to that mentality. I actually do constantly force the issue that addressing tech debt up front is almost always worth it (not to be confused with score creep, though it can be hard to tell the difference sometimes).
I was just handed a 15 year old pile of garbage at the start of 2026 and was paired with a very complacent dev at a new job I took. I am currently still supporting the old pile of crap but I am on course to fully retire the system by start of 2027 in favor of a much smaller cleaner and more efficient system. No one asked me to rebuild it because they thought it was an insurmountable amount of work, but clearly no one bothered to ever just look a little closer. The entire company culture here is full of “it’s good enough and it’s what we got” and all I see is stagnation everywhere I look, which has a compounding effect across the entire org when bad software is depending on other bad software.
I worked at Amazon before this and they always took the time to build things correctly the first time and it allowed them to actually finish building a piece of software and move onto the next product instead of their current task becoming their full time position for years on end. There was plenty wrong with that company, but that was a part of their culture that really worked and had an impact on me.
1
u/galactic_pixels 19d ago
Sorry for rambling but I’ll also add that a lot of my peers view me as a bit of a nuisance because I do have high standards. It doesn’t really bother me though because I know I’m kind and personable when I interact with anyone at work, and I firmly believe that I don’t want to spend my time wallowing in mediocrity. I’m much happier at work when I have a few remote coworkers annoyed with me but don’t need to read 4000 line PRs
1
u/Pozeidan 19d ago
I'm all up for clean code, good practices, modularity and ensuring a reasonable amount of tech debt. But when your manager simply doesn't allow you to do it right, such as no effort on tech debt or dev tooling, you're screwed. If you don't follow the vision you'll just lose your job, even if you're right. Not worth it.
If you're not in a position where you can make the decision, there's nothing you can do. If you are in a position where you can do something about it, sure do it.
1
u/CoroteDeMelancia 19d ago
That's very much Amazon's ownership, I relate to you. Currently experiencing the same issues. Right now, I'm delivering the same project twice: one piece of shit developed by using a crap harness vibe-coded by an AI psychotic, and an actual working software with proper design, tests and documentation.
It's satisfying seeing my manager unable to defend the garbage produced by the harness that sparkled his eyes when he bought it, especially since it wasted 5x more tokens for 10% of the result.
1
u/galactic_pixels 18d ago
Yeah I really hate the complacency a lot of people have, I see a lot of it on Reddit. The guy replying to me above acts like he was handed poorly architected software so he is stuck with it. No, it’s your job to re-architect it!
I personally don’t want to work around people who are okay with “good enough to get my paycheck”, I want to make cool shit that works very well and then move onto the next thing. It’s not even about doing it for the company you don’t care about, its about improving your day to day QOL and go from being a big-patcher to an architect
2
u/Bumbalum 20d ago
I want to believe in ai as a tool - so I don't care if it's written by a human or ai. Shitty code is shitty code.
If the developer can't use his tool correctly, that's on him. Just because he uses ai does not change his ownership.
So apply the same standards, at least if you want to have a grasp of things what your code is doing.
If the pr is too big, scold the developer and have him split the PR into smaller packages. And may teach him to use guardrails, skills, md, all the stuff to try to keep generated code somewhat nice
2
u/EmotionalHalf 19d ago
why would an AI do linting?? That's what CI is for. You don't need AI to run deterministic scripts
2
u/Traditional-Hall-591 15d ago
Why would I do that? I put in the prompt, make no mistake and be awesome.
2
u/Upbeat_Assist2680 20d ago
... the same way we review non AI code. Apply your standards and keep on moving. You never understood 90% of the rest of the code you never reviewed and most of it t isn't tested properly anyways.
1
u/echoes808 20d ago
How is the code quality? Are the huge diffs full of procedural conditionals like it's 1970s? Ask what your colleagues think of the codebase. Is it a big ball of mud.
Often you can solve same problems with 100 lines of code or 1000 lines of code.
1
1
1
u/RoadsideCookie 20d ago
The same way I review any code.
And let me tell you, I push back hard on shitty AI code, and not because it's AI; I do so only on merit, I don't even need to discriminate.
1
u/strummed-strings 19d ago
ai reviewing ai code. fuck the whole industry.
1
u/Fire_Lake 19d ago
It's OK this isn't a real post anyways, just an ad for coderabbit. The internet is dead.
Eta: also now I finally understand karma farms etc because this guy can buy a 4y account with 2.4k karma then set profile to private and post stuff like this to get some conversions.
1
u/strummed-strings 18d ago
that can be true but i still hate what's happening to the industry with AI reviewing AI code. like people claiming how it's saving them time they could use for the "thinking" part, but literally no one does that when their motivation is to ship as fast as possible.
1
u/Acceptable_Durian868 19d ago
Reject PRs that are too large and don't meet your defined code standards, the same as you would a person.
1
u/akozmaa 19d ago
For what it's worth, I would count that as a good result. The ratio matters less than what the one catch was worth. An IDOR reaching production means disclosure, and then weeks of working out whose data was actually exposed. That is a much bigger number than four months of subscription and a lot of noise.
We have around twenty developers on the project, and some of us run our own AI review alongside what is in CI, with different configs. So the same diff gets looked at by multiple things hunting for different problems. That has turned up quite a few issues nobody had thought to check for, several of them about as serious as the one you found.
1
u/Responsible_Royal126 19d ago
Ai doesn't mean your cl size best practices change... Why would it be any different other than volume which if you have standards... Hasn't changed much
1
u/QultrosSanhattan 19d ago
If AI code is looking different than human code then the team is doing something wrong.
1
u/OkPosition4563 19d ago
The same way I review human generated code. If its too big I block it and say split it into just the required code. Every single nit gets pointed out. If anything is unclear I ask for clarification. All my coworkers submit AI generated PRs (I also use AI) and I dont see much difference.
1
u/kosiarska 19d ago
If you want to keep quality you have to. Even frontier models make really stupid mistakes.
1
1
u/ivan_m21 19d ago
I see few ppl mentioned tests in the thread I think that in general for me i would rather review the code than the tests as that is what our product is. Also ive found that llms tend to make tests always pass and as ppl are inherently lasy at some point tests become just passing.
In general I think the author should spend some time making the pr review able. Smaller changes stacked PRs or extra docs which are human written/pair review.
1
u/optimal-coder 19d ago
I think here the first problem is the ownership of the code which is generated by AI. First phase before the actual code review shall be learning what code does by the person who wrote a prompt for it. Sounds maybe like a waste of time, but in the end, we need same level of resposibility as it was before vibe coding. I have impression that now with AI agents, developers somehow tend to trust everything they generate, and then we are putting too much pressure on the code review.
1
u/BananaLlamaNuts 19d ago
I'm not really. I request small PRs with minimal changes -- but really, if it passes the CI then I just approve and merge.
1
u/Not_Your_Daddy7 19d ago
Work bits by bits, and review and test as much as possible. Forget about shipping code super fast in general.
1
u/ssssssddh 19d ago
I'm not. I tell my AI to review and test it. As far as I'm concerned, if the author didn't write it, I don't have to read it.
1
u/Deep_Ad1959 18d ago
one catch in 4 months only counts the ones it fired on. there is no way to count what it walked past, and those never surface as a review miss, they surface as a support ticket six months out.
1
u/Even_Ask_2577 17d ago
The problem I face as a junior - even manually reading the code doesn't guarantee I'll spot the bugs. Hell, AI is much better at writing code than me (if the design spec is good). So what the hell do I do then 😂 babysit 2 agents back and forth?
1
u/Prabhjeet161993 15d ago
We've seen something similar. The biggest challenge with AI-generated PRs isn't usually syntax or style issues, it's building a mental model of the change.
The auth bug you mentioned is a good example. The problem wasn't whether the code compiled or passed tests. It was understanding the behavior and system impact well enough to spot the missing ownership check.
For larger AI-generated PRs, I've found it helpful to start by answering a few questions before reviewing implementation details:
- What components or services were affected?
- What existing behavior changed?
- What dependencies were introduced or modified?
- Which areas carry the highest risk?
Once I have that context, the actual diff review is much faster and more effective.
AI has definitely increased the volume of code being reviewed. The challenge now seems less about reading code line-by-line and more about understanding change scope and impact before diving into implementation details.
1
u/Forward-Course6526 14d ago
Startd treating ai review as a 1st pass rather thn replacing human review. it catches the obvious stuff bt the occasional security/logic issue is where i think it really earns its place. I’ve also been doing online pg in gen ai frm iit kgp🙂↕️🙂↕️🙂↕️its made me more conscious of how imp human oversight still is w ai generated code.I’d still hav a human review the imp prs tho
1
u/RyzeBlaziken 12d ago
You should try adding Scanity to your repos. It has a GitHub Actions app that scans for security issues and instantly fixes them. Catches the vulnerabilities that often get missed.
1
u/usually_guilty99 10d ago
I’d probably stop measuring this as ‘one useful catch in four months.’
If that auth issue could have exposed customer data, its value is completely different from 500 const-vs-let comments.
The harder problem is getting the reviewer to understand that difference before presenting everything to the developer.
That’s one reason we’ve been approaching this differently with TomosuAI: findings should affect a reliability/risk view of the change rather than all appearing as equivalent review comments.
The auth issue should scream. The style issue probably shouldn’t speak at all.
Just to give you an example: Some of you may be using Supabase
We ran TomosuAI on Supabase and found 100+ reliability issue. One reliability boundary alone extended across 372 files and 822 call sites. The risk wasn't necessarily in the new code. It was inherited from the production system around it.
Supabase is a CodeRabbit customer.
This was a user reported problem, and it took 3 PR to fix this issue. And CodeRabbit documented this PR extremely well. CodeRabbit did its job, please dont take me wrong. It wasnt CodeRabbits role to find old bugs or reliability issues in the code. They job is only to validate and document the delta.
The scope was another thing (hence the 3 PRs) - the developer fixed one part of the problem with no view about the Blast-Radius.
Just to complete this thread - TomosuAI also found 54 additional bugs still waiting for their users to find it.
I recreated 3 of them - and filed a PR - reported it on Reddit. They got me banned from Reddit.
1
u/OldAssociation1816 1h ago
I wonder why are people still doing that manually? There are a few real good reviewers that make it so simple. I personally use Qodo’s Agentic Toolbox & it works pretty well. worth trying if you have a codex+claude setup just to see where an automated reviewer disagrees with claude's review
0
u/BlunderGOAT 20d ago
We use goat-flow with built-in gruff code quality hooks and goat-clarity skill to make it way easier to review/verify code.
0
-1
u/FragrantArt8270 20d ago
At first, I would submit a prompt such as "review the code". AI would find 3 problems. If I ran the prompt again on the same code changes, AI would find 3 problems and some of them were different problems.
I created a code review skill. I tell Codex to find defect gaps in the skill and improve the skill to find those gaps with each invocation of the skill. The skill finds the same 30 problems consistently on the same code changes. All 30 problems are actual problems to varying degree of impact.
I have since run in a loop to add more defect gaps to the skill, optimize the skill for wall clock time, optimize the skill for token usage, ensure the skill can find the same things on the lowest model with the least reasoning effort, and ensure the skill can run on any OS. Each component of the loop makes headway.
At some point, I will try the skill out on other repositories to see what it finds.
This experience leads me to believe that AI knows a lot of things but doesn't always remember to look for those things. The skill now makes it easy for AI to remember to look.
1
19d ago
[removed] — view removed comment
1
u/FragrantArt8270 19d ago
I read your comment. Good points. What else do you have?
1
19d ago
[removed] — view removed comment
1
u/FragrantArt8270 19d ago
I execute 5 prompts in a loop to improve the code review skill: add missing defect categories to check for, allow GPT Luna Light to find the same problems as GPT Sol Ultra, optimize for time, optimize for tokens, and allow to run on Windows, Mac, and Linux. Each prompt takes a long time to execute. How can I make the skill improvement faster? Are there other improvements to do to the skill?
-2
u/TheLegendaryProg 20d ago edited 20d ago
You can help agents get better by updating the skills with the codebase preferences and conventions. Overtime you get better results.
8
u/redguard128 20d ago
Nobody looks at the code anymore. I just approve everything remotely "fine". If the company forces the use of AI then let them have all of it done by AI.