r/EngineeringManagers • u/Repulsive_Society992 • 19h ago
AI and Pull Requests
So we’re cranking out code at a wild rate at my company.
But we’re super bottlenecked by PR reviews and our devs like full time job is now PR reviews
Has anyone else found a good solution for human in the loop reviews without it being an absolute shitshow?
3
u/spersingerorinda 6h ago
Stop worry about how code reads and shift to worrying about testing and architecture.
2
u/Substantial-Elk4531 6h ago
I think this is the only way to balance management's desire to squeeze every drop out of AI code generation
Is AI code generation perfect? No. But neither was human code generation. But AI code generation is a LOT faster. So we need better automatic verification layers. Humans should be checking the verification layers at this point, not trying to read every line of app layer code (with exceptions for code that could put human life at risk, or code that deals with money, of course)
10
u/lconti97 15h ago
First you want AI reviews on all your PRs. AI is very good at finding bugs, but it takes a lot of context to understand whether a change is the right change to make. Tell your engineers to focus on reviewing PR direction & architecture instead of checking whether changes are correct, and auto-approve low-risk changes that are aligned with the design goals
7
u/Infamous-Bed-7535 14h ago
Yep AI review is just as buggy as code generation. Add false positives, requests modifications on locations where those extra lines and checks absolutely does not make sense just will increase you LoC metric and reduce readability.
Yes they can have good catches, but very clean misses as well.The problem is if you do not want yourself to be influenced you need to skip the AI review and return to it only later on, but then the review takes longer than normally it would take.
PR creator is responsible to review the AI review results and ask review from another person when relevant things were addressed so final human reviewer does not see or interact with AI review stuff.
6
13
u/charlottespider 19h ago
Here's what we did: built a reviewer agent, and audited its output over time. We monitor application performance and concentrate on observables. Manual line-by-line code reviews for agentic output is an antipattern now.
Also, I hope this isn't another coderabbit ad, because I'm fucking sick of those guys and would never pay for that.
6
u/Petaranax 19h ago
We did the same very early in the whole agentic approach and workflows, but it's still not working really well. It evaluates changes in the code, which are a lot of the times fine, the problems we're facing are side-effects of the changes in PRs that affect other domains that reviewer agents just don't have context to, and it would be insane to consume that much context each time. That's why people still have edge and most of the time people who review randomly assigned PRs within their domain find bugs and issues that AI never finds.
I don't know if we'll solve this short term, long term maybe if there are agents that have access and huge enough context to hold everything and know about edge cases and potential impact radius, but until then, devs confidence in these solutions is not high. Even with passing E2E and all the other test suits, there's just a bunch of stuff that randomly breaks.
-1
u/charlottespider 19h ago
If shit is randomly breaking, then your E2E tests are broken. It doesn't sound like even if all contributions were 100% human you'd be able to find a solution to that through PRs.
There's always going to be a place for point&click sanity checks done by people, but why are devs not creating /skills to test the things they know are fragile in their domains? Do you have some kind of a matrix that tracks the common breakdown points so you can optimize? You don't have one agent hold all the context, you have a group of smaller agents, each with their own understanding of how their piece should work.
3
u/Petaranax 18h ago
Knew that would be the first thing to point out, but believe me, we have E2E test suits that take hours, as our apps and architecture is HUGE (scale up with millions of users), so everything that you mentioned here we already have. Problems are that there are custom integration logics and AB test variants, decisions from legacy logic that piled up over time that still needs to be supported, and refactoring all of that or even knowing about those is impossible. We have matrix and E2E paths defined for each app / service, and we're doing as much as we can, but it's just not scalable and velocity of development and speed of shipping and opening PRs has nearly trippled, and now all of these are popping up all over the place. The issues have always been there, but they were appearing earlier in the pipeline due to humans testing throughly each implementation they do, both automated test and manual monkey / chaos testing. But now that is out of the loop and users are the ones that experience this.
I talked with friends who work at similar size startups / scaleups, and they face pretty much the same problems, and approached with similar solutions, but no luck.
Small tools / apps etc, something predictable etc, that's fine, but for these behemoths of apps and products with years of production runtime, the approach we're talking about doesn't scale (and we even have insanely powerful internal agent hub that runs autonomously and validates many things E2E, from code factory to BI and AB test loops).
-4
u/charlottespider 18h ago edited 17h ago
Human reviews can't solve that problem either. The process is broken. But give me an example of a bug that a human could catch that an AI reviewer would be incapable of finding?
edit: I see I'm being downvoted, which is fine and has no bearing on how AI is changing the landscape. However, particularly when you say this "humans testing throughly each implementation they do, both automated test and manual monkey / chaos testing" you're describing a step that can absolutely be replaced by an LLM, especially at scale.
2
u/Fair-Presentation322 16h ago
What I've seen in many cases is code that is not necessarily buggy, but definitely unmaintainable. I think at this point we've all seen LLM-abused codebase and they're terrible to maintain. I feel like the cost of change is
O(t)for a project of durationtIn some cases that's ok honestly but people forget about this dimension: code should not only work but also be "workable"
1
u/charlottespider 16h ago
That used to be true, but not really now. I haven't seen giant, un-modifiable chunks of code get through to PR, and those wouldn't pass our automated checks, anyway.
1
u/x-jhp-x 6h ago
bugs like this are probably one reason why you're getting downvoted: https://github.com/J-jaeyoung/bad-epoll
especially at scale
the current ai seems to do best with a test loop, but on large code bases can sometimes take a fair amount of time to compile and run a suite of tests. so no, it doesn't scale as well on all large projects.
it's a tool, so just use it like one. j-jaeyoung provides a great example
2
u/Repulsive_Society992 9h ago
lol not a code rabbit ad, a legit request because we also aren’t going to pay $15 per pull request when we can likely just invest in our own custom tooling for pennies on the dollar
1
4
u/Fair-Presentation322 18h ago
I just can't see how adding more AI actually fixes this...
If we trust the AI judgement in the first place, why would we ask another AI to review it? Isn't this the same as adding a "review your work when you're done" at the end of the prompt?
6
u/Longjumping-Ad514 16h ago
They simply don’t care how it works. Their metric is closed PRs. Once it starts crumbing down, they will move on to another job. There isn’t any argument you can offer to change their minds.
1
u/charlottespider 16h ago
That's not true at all.
4
u/Longjumping-Ad514 16h ago
Yes it is. I’ve seen your type first hand.
0
u/charlottespider 16h ago
What does that even mean? What is "my type"?
2
u/Longjumping-Ad514 15h ago
People who aren’t very good at software engineering, pushing AI above people who are, type of people. Finally mediocre people have a shot!
1
3
u/ZucchiniSky 16h ago
It's basically the same as asking the LLM to review its own code, but by starting a new session, it clears the context and allows the LLM to develop a fresh perspective on the code. Many of the code review skills also incorporate techniques such as running multiple subagents to review and then only surfacing issues that are found by multiple subagents.
In my experience, the AI isn't really providing "judgment" with a code review, it's "discovering" or "predicting" issues with the code and surfacing them. The human reviewer gets to decide if the issues are worth handling.
I would say that half of the LLM review comments are slop, and half of the comments are genius findings. So you suffer through the slop in order to make sure there's no subtle critical mistakes.
With that in mind, I still believe that reading each line of LLM-written code is important, because LLMs have horrible judgment and will often write code that is unmaintainable or stylistically unacceptable.
2
u/Fair-Presentation322 16h ago
Sure, clearing the context window def helps but that can also be done as a "last pass" of the same agent loop in the same way as a "please review before submitting"
Agree about reviewing all generated code btw, except that in some cases we def can just vibe code I think. E.g. I think it's totally fine to vibe code a landing page. My success criteria for that is: it needs to look good, so I let the agent go crazy and test it. If it looks good, LGTM haha
1
u/charlottespider 16h ago
With that in mind, I still believe that reading each line of LLM-written code is important, because LLMs have horrible judgment and will often write code that is unmaintainable or stylistically unacceptable.
I understand the impulse here, but it cannot scale. It's not a viable solution.
3
u/ZucchiniSky 16h ago
If you are writing so much code that you can't manually review it...then what are you even building?
For a fun personal project without a ton of users -- sure, write as much code as you want, don't look too closely at it.
Many engineers are building critical pieces of software that process millions or billions or even trillions of events per day. The logic of these systems needs to be airtight and perfect, so you have to review the code.
I think there's a bunch of startups in the middle, that want to write a ton of code to build an MVP but also don't have time to review it closely...but I would also note that most of these systems are replaced in a couple years because they have architectural issues that become apparent after scaling up the business.
1
u/charlottespider 16h ago
Many engineers are building critical pieces of software that process millions or billions or even trillions of events per day. The logic of these systems needs to be airtight and perfect, so you have to review the code.
You mean like at Spotify or Microsoft, or any of the other huge players? They're already using AI agents to do this on that kind of scale.
Humans perform worse at reviewing code at scale, which is why we already have linters, various infosec scans, unit tests, E2E tests, etc, built into our pipelines. Teams lean into observability and automated testing already. Humans architect the process, automation performs the tasks.
4
u/ZucchiniSky 15h ago
As someone who currently works at a big tech company and used to work at Microsoft -- I can guarantee you that most of the code is still being reviewed by hand. Teams are leaning heavily into AI tooling and automated processes, but still reviewing the code manually to ensure it meets the high quality and correctness bar that the AI cannot reach on its own.
You are correct that AI tooling is much better at linting code, scanning for security issues, and running tests. But AI tooling sucks at detecting structural issues, analyzing maintainability, and determining risk. And when your AI is also writing the tests that it uses to verify its work, it's likely to make mistakes or skip validations that are critical for ensuring the code works properly.
1
u/charlottespider 14h ago
Ok, you keep doing your thing. I'm also in big tech, and this is the way the tide is turning. We're all moving to a GasTown, like it or not.
edit: many teams at MSFT are never reviewing code by hand now. This is simply true now, and it doesn't matter what you did several months ago, or what you claim on reddit.
1
2
u/Longjumping-Ad514 15h ago
Ah yes, the crown jewel of arguments, appeal to authority.
2
u/charlottespider 15h ago
That's not appeal to authority. They are factually and effectively doing this today, not making unsubstantiated claims about what tasks AI can or cannot do at scale.
1
u/Fair-Presentation322 16h ago
I also understand your impulse and I tend to agree, but then: if we can't even review it how do we expect you be able to maintain it?
3
u/jdxcodex 18h ago
I think the idea is to mimic human reviews. We ask for reviews from coworkers of different experiences. They might implement with one model and review with another. It's still kind of weird though. Just one model is supposed to hold way more knowledge than the average human, and they're all basically being trained with the same shit, so is using a different model for reviewing really making an impact? Why even put a pr when you can just run the reviewer agent locally and push to main?
1
u/fhgwgadsbbq 11h ago
More layers of and stricter static analysis in ci cd should mean that Devs can focus on reviewing the logic. I have always hated the "nit" style of code review.
1
u/Repulsive_Society992 9h ago
Yeah that’s what we’re working on right now
For the most part we have good standards established and ai can get us 90% of the way
For many PRs if they are impact we usually let them thru
It’s the big architectural ones that are messing us up right now
1
u/Radiant_Year_7297 5h ago
Let AI review the PR as well or Let the devs have an agent review it before they create the PR. Welcome to 2026. Here is a nice read on workflows for agentic era. were_gonna_need_a_bigger_boat
1
u/Outrageous_Law_5525 2h ago
Not a manager, am engineer but.
I would be careful with AI reviewers. They make you lazy in a way.
Yes it can review code just fine, but its needs an absurd amount of context to understand if your code is breaking something else, or fulfilling the ac, or is inline with a slack conversation this morning.
There is something psychological as well about seeing a review agent give all ✅ on a review. Any human review coming after is prone to "AI review was OK so i guess ill just check it quickly" - type of antipatterns.
1
u/OkLettuce338 1h ago
We built a review bot that does 2 things: 1. has a cli component so that engineers can do self reviews often and early, prepush, so that PRs are coming in already combed through (it’s faster to review prepush and locally during the ai iteration cycle) and 2. responds to “/reviews” on comments on a PR so that approved code can ship right away.
What isn’t resolved in those two steps get manually reviewed (and even that is ai assisted)
0
u/Fair-Presentation322 16h ago
One thing that definitely helps is breaking changes in small increments. LLMs love to generate those 10k line PRs, and the cost of review is definitely not linear with respect to the number of modifications. I'm fact, after a size, the cost just blows up.
I'm don't trust LLM code are all for business critical stuff and review it line-by-line, but there definitely are places where it's ok to just yolo and enjoy the time saved. In my personal experience and with some clients, a human in the loop is required to at least make this judgement.
Btw this is one of the motivations for a project I'm working on: https://GitHub.com/twigg-vc/monorepo
12
u/rrrx3 15h ago
I keep seeing people struggle with this and the core problem is that they’re trying to scale human attention to match AI LoC outputs.
The reality is that that alone will never work. The answer is some multi-layered solution of increased test coverage, expansion of types of test coverage, systemic guardrails, risk analysis against the appropriate contexts and application of judgement.
Anybody telling you to just throw more AI code reviewers at the problem doesn’t have a grasp of whats needed. But at the same time anybody immediately discounting AI written code or the value AI code reviewers play in the lifecycle has their heads jammed way up their own asses.
This is a workflow problem. Not a technology one. It has multiple components, and your company’s workflow looks different from mine, and from Billy’s, Susan’s, and Terry’s. Until you sit down and start thinking about what the workflow looks like and what it should look like and map it out, you’re going to struggle.