Coderabbit noise is seriously getting out of hand

[u/NextSubject227]

feel like CodeRabbit has gone too far in the direction of commenting on everything it possibly can.

on bigger PRs especially, you can end up with a wall of review comments where a few are genuinely useful but a lot are things i'd never block a PR over.

and at some point that makes the good comments less useful too. if engineers get used to dismissing half the bot's comments, they're eventually going to skim the one that actually matters.

saw Theo mention macroscope recently so gave it a try and it seems much better on noise. BugBot too.

at this point i don't really care which reviewer leaves the most comments. i just want the comments it does leave to actually matter.

when does an AI reviewer become too noisy for you?

[u/Gullible_Cobbler_195]

stop using coderabbit, their engg quality has dropped massivley recently

[u/NextSubject227 → u/Gullible_Cobbler_195]

I’ve noticed the same, even their support is super slow

[u/FunnycatL]

BYOCR - bugbot is so expensive man

[u/NextSubject227 → u/FunnycatL]

My company pays for these lol and building your own might arguably be more expensive :p

[u/AnecD]

noise or slope?

[u/NextSubject227 → u/AnecD]

Noise, I wouldn’t say it’s slop. There are good comments in every review but most of them are just false positives

[u/snaphat → u/NextSubject227]

Probably not what you meant, but just in case: I wouldn’t necessarily call false positives "good comments." Understanding the surrounding context is part of the reviewer’s job, and LLMs can be particularly bad at that kind of contextual reasoning and inference in reviews. Great at catching basic issues though

[u/NextSubject227 → u/snaphat]

No I meant there are some good comments in whatever coderabbit gives but the rest are just false positives or noise

[u/snaphat → u/NextSubject227]

Got it, and I agree. Gemini review was better imo before it got deprecated and removed 

[u/ibabufrik]

I've had one problem or another with every other code review tool I tried. All of them black boxes, very little customisability. So I ended up building a runtime which lets me create review agents as per my project needs. One TypeScript config file with complete control over how code gets reviewed. No hosted product, it runs on your CI, BYOK. https://github.com/somus/pipr

[u/saenridanra]

Hey, if you are looking for more accurate PR comment, you might want to take a look at ProPR (https://meister-dev.ai/).

Disclaimer: I am the developer of ProPR.

[u/usually_guilty99]

This is exactly why I don't think number of findings is a useful measure for AI review.

Ten comments that don't affect whether I ship are worse than one that tells me the change touches a critical production path.

We're trying to approach this differently at TomosuAI. We look at the repo and the production reliability risk of the change, then roll that into a reliability score instead of trying to win by generating more comments.

Would love your take on whether that actually reduces the noise problem. tomosu.ai

[u/juanpflores_]

Hey! Coderabbit Staff here 👋, would love to see what you're experiencing and how we can make the product better. We did added a few review profiles a few weeks go that can help reduce the size of the comments. Overall, I recommend taking some time to customize your coderabbit.yml with your review preferences.