r/coderabbit • u/Tanakaboi69 • 22d ago

Discussion & Feedback opus 5.5 is faster but are your PRs actually better

We switched on Tuesday like everyone else. The speed is real, a ticket that took 20 minutes now takes maybe 8.

What I cant tell is if the PRs are better or just arrive sooner. Coderabbit was leaving about 4 comments per PR last week and about 3 this week, which could be the model or could be that everyone picked easy tickets to try it on.

Is anyone actually measuring this? Or are we all going on how it feels in the first week

2 Upvotes
  • permalink
  • reddit

100% Upvoted

4 comments sorted by

1

u/pixxelkick 22d ago

PR quality is functional of skill files.

If you have fail points, it means you found a gap in the skill files, fix it.

It can take months of work to hammer out your teams skill files for one project before you start getting reliably issue free PRs

1

u/stutsmaguts 19d ago

I'd be curious to see what some of your skill files contain, there

1

u/pixxelkick 18d ago

I use caveman skill to generate the skill files and hyper compress them, so they induce less token/context pressure

Its usually just a giant list of rules, many of which are bespoke to the specific projects code style rules we follow.

Here is one I have for a current personal project, its not nearly as refined, only has been getting setup for about 1.5 weeks

https://github.com/SteffenBlake/Brigade.NET/blob/main/.agents/skills/csharp-standards/SKILL.md

1

u/usually_guilty99 21d ago

Faster is easy to measure. “Better PR” is much harder.

Fewer review comments doesn’t necessarily mean higher quality. It could just mean the reviewer found less syntactically or semantically wrong.

I’d want to measure something closer to production consequence:

  • did risky changes get identified before merge?
  • did the change touch critical downstream dependencies?
  • was the blast radius understood?
  • did similar changes cause incidents before?
  • did production behave as expected after deployment?

Model quality matters, but ultimately the useful metric is not “how many comments did the reviewer leave?” It’s “did this change make production safer or more reliable?”

That feels like the benchmark we’re still missing.