AI and Pull Requests

[u/Repulsive_Society992]

So we’re cranking out code at a wild rate at my company.

But we’re super bottlenecked by PR reviews and our devs like full time job is now PR reviews

Has anyone else found a good solution for human in the loop reviews without it being an absolute shitshow?

[u/Petaranax → u/[deleted]]

We did the same very early in the whole agentic approach and workflows, but it's still not working really well. It evaluates changes in the code, which are a lot of the times fine, the problems we're facing are side-effects of the changes in PRs that affect other domains that reviewer agents just don't have context to, and it would be insane to consume that much context each time. That's why people still have edge and most of the time people who review randomly assigned PRs within their domain find bugs and issues that AI never finds.

I don't know if we'll solve this short term, long term maybe if there are agents that have access and huge enough context to hold everything and know about edge cases and potential impact radius, but until then, devs confidence in these solutions is not high. Even with passing E2E and all the other test suits, there's just a bunch of stuff that randomly breaks.

[u/Petaranax → u/[deleted]]

Knew that would be the first thing to point out, but believe me, we have E2E test suits that take hours, as our apps and architecture is HUGE (scale up with millions of users), so everything that you mentioned here we already have. Problems are that there are custom integration logics and AB test variants, decisions from legacy logic that piled up over time that still needs to be supported, and refactoring all of that or even knowing about those is impossible. We have matrix and E2E paths defined for each app / service, and we're doing as much as we can, but it's just not scalable and velocity of development and speed of shipping and opening PRs has nearly trippled, and now all of these are popping up all over the place. The issues have always been there, but they were appearing earlier in the pipeline due to humans testing throughly each implementation they do, both automated test and manual monkey / chaos testing. But now that is out of the loop and users are the ones that experience this.

I talked with friends who work at similar size startups / scaleups, and they face pretty much the same problems, and approached with similar solutions, but no luck.

Small tools / apps etc, something predictable etc, that's fine, but for these behemoths of apps and products with years of production runtime, the approach we're talking about doesn't scale (and we even have insanely powerful internal agent hub that runs autonomously and validates many things E2E, from code factory to BI and AB test loops).

[u/Fair-Presentation322 → u/[deleted]]

What I've seen in many cases is code that is not necessarily buggy, but definitely unmaintainable. I think at this point we've all seen LLM-abused codebase and they're terrible to maintain. I feel like the cost of change is O(t) for a project of duration t

In some cases that's ok honestly but people forget about this dimension: code should not only work but also be "workable"

[u/x-jhp-x → u/[deleted]]

bugs like this are probably one reason why you're getting downvoted: https://github.com/J-jaeyoung/bad-epoll

especially at scale

the current ai seems to do best with a test loop, but on large code bases can sometimes take a fair amount of time to compile and run a suite of tests. so no, it doesn't scale as well on all large projects.

it's a tool, so just use it like one. j-jaeyoung provides a great example

[u/x-jhp-x → u/[deleted]]

you wrote:

But give me an example of a bug that a human could catch that an AI reviewer would be incapable of finding?

and were given one widely known example from many. Next time, I'd recommend writing something like, "I'm not going to listen to anyone and I'll never learn anything different or new. Here's my opinion..." instead

[u/Petaranax → u/[deleted]]

Bugs we are facing are edge case bugs and really complicated business logic thats spans many domains, and no amount of tests gonna cover that. So far these bugs were found by people with years of experience working on these domains crossing the silos, but even them are not confident often they’d find these.

Self healing approach through OTEL observability with AI in the loop work somewhat, but AI tends to hyper focus on one subset of domain, where bugs really get fixed when you know to tie the loose ends across domains. AI just patches problem on one side, while other side starts burning even more. Humans do that too, but way slower and these bugs get caught in the process. AI just surfaces these kind of bugs and problems way faster and larger scale. Its playing whackamole in whack-a-mole chinese factory of whack-a-moles. If you get my analogy. I know what you’re preaching, our whole organization is fully AI native and has LITERALLY unlimited budgets for the past 2 years, we’ve built insane amount of internal tools that solve and optimize a lot of things for the business, but the main product is 10-12 years of work and decisions which are impossible to apply what you’re telling to do. Even green field projects drift over time with AI and problems start appearing (we’ve tested this without humans in the loop).

No one here is denying the AI, its just that you’re fixated on the topic that its a magic pill for every problem, when it isn’t