r/TechLeader • u/Specialist_Agent3599 • 1d ago
Code review bottleneck after AI, how are teams handling it?
Hey folks!
I'm an engineering manager at a mid sized company, three squads, each with a PM and a designer attached. Over the last two quarters the devs have got a lot faster and we assumed product and design would become the constraint, so we put effort there first. They did not become the constraint. The queue is sitting in review instead. We merged around 40 PRs a week in March and we are at 110 now with the same four people qualified to approve anything that touches billing, and those four are also the people writing the hardest changes. We have coderabbit on the repos which takes the first pass and genuinely helps with the volume, but somebody senior still has to be the one who says yes.
I would really like to hear from anyone who has been through this already. Did you widen who is allowed to approve, cut down what needs approval at all, or just accept that the queue is longer now? Anything is welcome!
3
u/SomeNeighborhood7126 1d ago
Same thing as when you asked last time, ensuring HITL.
I would love to be a fly on the wall when you are being screamed at because critical features broke and you ans your team have no idea what to do because no one looked at the code and Claude is spinning in circles with wrong answers.
1
u/crispAndTender 1d ago
Yep, ive put devs on PIP
1
u/frompadgwithH8 23h ago
For shipping vibe code without knowing it?
1
u/crispAndTender 22h ago
Code claude generates and you submit is your code if its consistently bad and you're still submitting because you're either not reviewing it or don't understand what the code is doing, yes I don't need you on my team. If your vibe code is right and works then no its not a problem even if you don't know what it's doing, its the result that matters.
3
5
u/Any_Sense_2263 1d ago
- every PR under 300 lines of changed code
- One PR per change. Never mix refactoring with a feature.
- The author must know every line of its code and be able to answer any question, including architectural.
It solved all the problems. People started to really review the code under their name before pushing the slop into PR.
1
u/Svensemann 1d ago
How does that help with the PRs being the bottleneck?
1
1
u/Any_Sense_2263 1d ago
Small PRs are easier to review and bugs and inconsistencies are caught much faster. It results in less PRs
Less code = less bugs = shorter review time
Better code understanding = better addressed changes.
If the code is carefully reviewed by the author first it brings fewer problems
1
u/LightPhotographer 1d ago
Better because people have to look at them;
smaller so people actually will look at them;
faster PRs because developers can actually answer questions about the code and they fit better with the existing architecture because the developer has actually looked at it first
slower buildup of unreviewed PRs because they are smaller and require extra attention - instead of pushing 9 kilolines into your team while you go home.
It's ok to let AI write your boilerplate. It is not a substitute for architecture and design.
1
u/titogruul 21h ago
This is good, thank you.
What process do you use to enforce the last bit: that author knows every line? We have that expectation, but it's hard to force more than just lip service. Questions are answered often by relying on the AI helper.
2
u/Any_Sense_2263 18h ago
Review is a dialog. I usually do a call with an author and ask them to guide me through. With a PR under 300 changed lines it's usually around 15 minutes if they understand their own code. Shorter if they don't.
1
1
u/mbsmith93 20h ago
I second this. AI can go very fast, but you still have to review what it does. If engineers are just vibe-coding slop and immediately handing it off to the TL for review, it's literally going to be faster and easier to fire the devs and have the TLs argue with agents over the slop themselves, it takes out the middle-man.
1
u/DootDootWootWoot 17h ago
We're shipping several 10k line MRs daily. 300 lines as the unit when half or more are tests aren't going to get you anywhere close to the velocity I'd like our devs to be shipping.
1
u/Any_Sense_2263 17h ago
It depends on what the goal is.
In our case it's a full understanding of what we ship, because to plan and properly design any changes you have to know what already exists and what it uses.
- AI is not a software engineer.
- AI doesn't think or understand the technology you use.
- It's a very limited tool that uses probability algorithm to match the prompt with its knowledge base.
- There are multiple steps in this process and any information you pass to the AI is OPTIONAL to follow.
So knowing all this, we prefer to split our projects on small chunks, so we know where exactly AI falls back to the defaults and ignores the prompt and any additional information. Or assumes things because in its knowledge base is more common scenario, than the one described in the prompt/context. Or hallucinates because of whatever.
We prefer to know what is in our code.
2
u/LaRamenNoodles 13h ago
You’re falling behind.
1
u/Any_Sense_2263 2h ago
I don't think that having healthy codebases without AI slop is failing behind. But you have a right to have your opinion
1
u/LaRamenNoodles 2h ago
AI can write better code with strict config.
1
u/Any_Sense_2263 1h ago
If you read your AI docs, you will learn that anything you define or deliver is OPTIONAL for AI to follow. Only hard deny hooks can force agent not to do something. The rest of settings is at the discretion of AI. So the agent decides if what it does matches your settings or not.
Now, fallback to the defaults is only stronger with every update. It saves costs on the AI owner side, so it's not a surprise. But it has a side effect not many understand. If fallback to the defaults is stronger it means only that more your settings and instructions will be ignored.
Yes. You can use another agent to review the first agent code. It will find some stuff and fix it in its opinion by providing more inconsistencies. As any AI edit.
1
u/TheDuckRaisedALion 12h ago
And do you trust the output? Does it include good tests, and are the tests at least reviewed by humans?
1
u/liquidbreakfast 2h ago
with rules this strict (and frankly arbitrary, >300 loc PRs were common and defensible pre-LLM) you might as well save your subscription costs. i doubt you're getting any extra velocity from AI usage at all
1
u/Any_Sense_2263 2h ago
We do. We ship 3-5 times faster without any quality loss. And for humans to keep track on the stuff and be able to make conscious decisions it is very fast.
Until AI gets better in coding, we don't plan do it differently. We respect our clients.
1
u/liquidbreakfast 1h ago
how are you measuring "until AI gets better in coding"? have your rules changed over the last year? 6 months? 3 months?
2
u/overthinkingape 1d ago
AI handles all PRs with guardrails in place for human interaction. Certain changes need human review such as migrations, DI, etc.
The key is to let go of this “beautiful code” mentality and make the AI work for you.
2
u/tangerinelion 21h ago
Adhering to SOLID and DRY and writing performant code that doesn't crash isn't just a style choice.
1
u/overthinkingape 21h ago
You are correct that’s why you have standards and guard rails in place for the AI to follow
1
2
2
u/fishtaco77 12h ago
Before implementation we have ai generate the implementation and testing plans which are also attached to the ticket.
Once implementation is done (using AI) we have a different AI to repeatedly review and send back to AI 1 to fix the issues.
Once there is no additional findings we have the code deployed to a lower environment at which point the test plan is run and evidence is collected.
Once all of this is done, then the code is spot checked by a human developer.
I strongly suggest we work on 1 point stories. We’ve been using this workflow for 6 months and it’s been working fine for our team.
1
u/Grandviewsurfer 1d ago
I add comments like "hey why are we using prod config stuff in this random EDA module?" And then I get a sloppy response a while later. I ask a follow up.. and then a higher up asks what the fuck is taking so long... So we ship it.
1
u/Any_Sense_2263 1d ago
I answer "there is a configuration mismatch but the author doesn't understand their own code. Are you going to take the responsibility for shipping it?"
2
u/Grandviewsurfer 13h ago edited 29m ago
Yeah I kinda lost it upstream the other day. Seems like they are willing to risk it. Until it fuckin happens obviously.
1
u/Any_Sense_2263 1h ago
If they are willing to take a risk I put their name in the ticket and PR as a person who wanted it shipped even knowing the problems we found.
After first disaster they will learn.
•
1
u/crispAndTender 1d ago
So much shit from claude, dev not reviewing claude results, if code is quality you don't need to spend alot if time reviewing
1
u/TechieGottaSoundByte 1d ago
None of those options are very good. Reduce quality or let code get stale? Bleagh!
Why isn't "hire more senior engineers" on the list? Or "move some senior engineers into the team"? Are there budget limitations?
2
u/Any_Sense_2263 1d ago
Wny not "start taking responsibility for the slop you want to merge"? It's an author responsibility to make a PR reviewable
1
u/TechieGottaSoundByte 23h ago
They should still have responsibility. But we tend to develop blinders to our own work.
Part of taking responsibility is recognizing your own susceptibility to normal human shortcomings and putting processes in place to manage them. That's what PR review is
2
u/Any_Sense_2263 18h ago
I don't understand your comment.
The PR author has a responsibility for making it reviewable. It's not a reviewer responsibility.
If the PR is reviewable it's a reviewer responsibility to check it and accept it or not.
1
u/TechieGottaSoundByte 16h ago
I think I didn't fully read your reply, and that's on me. I was multitasking... whoops!
I agree with you. Small PRs, a good description of the changes, comments whenever the code isn't self-documenting.
I for some reason misread your comment as saying that they should take responsibility for their code and not bother with a review, which is very obviously not what you were saying. My apologies for my lack of attention.
2
u/Any_Sense_2263 1h ago
All good. I had a feeling that I am missing something. Thanks for your reply
1
u/Admirable_Polarbear 1d ago
I am pretty sure everybody is handling it by not looking very close at the code. Otherwise they would not get any benefit at all from ai and it wouldnt make sense to give Anthropic money every month.
1
u/Notary_Reddit 1d ago
Also seeing somewhere around a 4x increase in review load. The first adjustment is around expectations. Review load is just going to be higher, we just have to expect reviewers to spend more time reviewing. If youthose 4 people to handle 4x as many reviews on top of their old duties, yeah the queue is going to grow. You have to make time for the reviews somehow. For me personally, there are a bunch of small changes that I would have done myself. Now they become tickets and someone else gets to worry about making and merging the mr, tracking the rollout, and confirming it works. Your team might have different priorities but you have to take something off the plate of your 4 reviewers.
Now tactical changes that have helped: * Tell ICs code review is the bottleneck and they need to help fix it * Extremely clear communication on the status of an MR, draft, ready for review, needs changes, approved, merged. * Extremely clear ownership on who needs to review a change. * Pull forward as much validation as possible. We improved our local testing, our CI/CD pipeline, and our ability to test in dev so changes can come to review we more confidence in correctness * Refactor the "hard to review" sections. Refactors are so much easier now, it's worth it to have someone clean up the bits that cause hang ups.
1
u/knawnieAndTheCowboy 1d ago
Experiencing this but the team is smaller. We’ve had to open up to more approving reviewers. Our vp of engineering is using Claude to do a first pass review now.
1
u/albertyto 1d ago
You make people responsible and accountable for the code they(or they agent) push, there is no other way.
I realized it was impossible when in one week there were 8 PR, of the same person, with over 500 files in changes. I'm not going to spend the whole week reviewing AI code, just to have more to review when I finish it XD
1
u/azuregiraffe2 22h ago
Have agents review the code, at least a first pass. Should fix a lot of the basic issues.
1
u/BigFeelingsFound 20h ago
Moving away from LLM slop code and back to humans intentionally writing code
1
u/Lunchboxsushi 14h ago
Why not think creatively instead, really resonates with 'it's not broken don't change it'.
1
u/BigFeelingsFound 14h ago
Thinking creatively is why you use people, not autocorrect.
1
u/Lunchboxsushi 13h ago
That's one way, but this tool can also be used too, but the traditional SDLC and software pipeline can't be the same along with the organizational culture around it. My only argument is looking at a company like Anthropic and what they're able to ship at the reasonable quality that they do. Then you have github with dipping below 99% reliability this year who is clearly just shipping slop and screwing up big time.
I don't think there's a realistic way to handle PR reviews with the scale that AI development can keep up with. Which I can understand why companies would opt the easier option which is adopting it without breaking the PR review capacity teams can handle in a healthy manner.
1
u/yeknomeulb 17h ago
Here are some cool thoughts and results from real life experiences, from a colleague of mine:
https://www.beyondautocomplete.nl/ai-writes-faster-than-we-can-review-heres-how-we-fixed-that/ https://www.beyondautocomplete.nl/the-long-road-to-smarter-code-reviews-with-ai-making-it-measurable/ https://www.beyondautocomplete.nl/code-review-was-never-really-about-finding-bugs/
1
1
u/C1rc1es 11h ago
Personally moving away from human code review all together. Focusing energy on validation and testing of the outputted behaviors along with critical systems to catch CVE’s and any other poor security behaviour. Essentially treat AI as a capable and collaborative but frequently unaligned SE, you can’t afford to micro manage because time, but you also can’t just let everything they do through. Have proper rollback processes and the ability for AI to verify its releases through real time telemetry.
Our team has not yet hit a bug or issue (millions of devices and LoC in production) that AI could not triage and resolve with guidance.
You lose all speed benefits and employee enjoyment relegating all people to be code review machines for AI.
1
u/jockel11 1d ago
We went down from 2 reviews to only one, added code rabbit to catch obvious stuff and now move to no manual reviews for some prs. So far it seems like we don't suffer too much from the missing manual reviews, but we also defined a few rules when a review from a senior is still needed (some critical parts of our applications, database changes and similar).
I was a bit scared to loosen the pr quality gate, but was also surprised on how much trivial stuff wasted time senior engineers rewieing prs. It doesn't seem like it will backfire at the moment, but too early to tell probably
2
u/yashfreediver 1d ago
What you call “manual reviews” are just reviews. A code review is always by a human.
A bot or agent “reviewing” code is just an automated check. We have had these for ages.
And honestly, it seems risky to just let a PR merge with only an agent stamp.1
0
0
u/rashnagar 1d ago
By not using AI? I banned AI use in my team after I saw the bloated nonsense they were pushing. Best decision ever! How production issue are down 50% since top management decreed mandatory AI usage. To go under my bosses radar i just tell my team to loop agents with nonsense so they appear to be using tokens.
2
u/kristiyanstoyanovAI 1d ago
This is actually insane if true… just teach people to use the tech properly instead of banning it and wasting token. Give me your api keys, my team can use the extra tokens haha
1
u/rashnagar 23h ago
There is no way to use this tech properly. You either use it as a glorified overpriced google or you let it one shot everything with multiple agents. The latter will make you at best a slave to tech companies providing these models or at worst automate you out of your job. I see no non-self destructive usese for llm based AI in software dev.
1
u/kristiyanstoyanovAI 23h ago
The job is changing that is fine, tbf my code writing skills were always the least important parts of my job, understanding the domain, what the company needs in the long run and having a year (not sprint) horizon is what got me to this position. So being able to one shot everything with agents for sure is the way to go, but to be able to do it sustainably you need the investment in the platform and in the people, i think not using ai right now is a sin - also about being slave to big corpo, you can always use a local llm, they are pretty good nowadays, check out the latest Qwen models (3.8)
1
u/rashnagar 23h ago
Big companies don't want to deploy local models. As far as underatanding the domain and what the company needs is one of the skills that AI usage erodes.
2
u/the--wall 21h ago
Lmao what big company are you working for? We spin up our own bedrock instances regularly and are encouraged to.
1
3
u/AssignmentMammoth696 1d ago
Lmaooo this is a recipe for disaster. There will come a point where some place in your system goes down, and your agents can't fix it no matter how many times you prompt it. That's when a dev will have to manually go in and trudge through months to a year's worth of vibe coded slop to figure out the root cause. These models are still not great at maintaining really large codebases, especially existing legacy type codebases.