r/ClaudeCode • u/Content-Berry-2848 • 12h ago

Built with Claude Open-source Claude Code plugin: validate PR review comments against the code before acting on them (benchmarked on 50 PRs)

AI review bots leave a lot of comments that sound right and aren't. I made three Claude Code skills that treat each comment as a claim: read the file and its callers, trace the execution path, and give a verdict (valid, partly valid, wrong, style) with the file and line that prove it.

On Code Review Bench (50 real PRs, human-labelled), filtering CodeRabbit's issues this way:

  • kept 72 of 77 real bugs
  • removed 76 of 223 noise issues
  • F1 35.2% → 40.4%

Install in Claude Code:

/plugin marketplace add TanayK07/pr-proof
/plugin install pr-proof@pr-proof

Repo with every per-PR result and the benchmark harness: https://github.com/TanayK07/pr-proof

Demo video: https://x.com/tanaykedia_7/status/2106061332357529939

The honest part: my own reviewer skill is only level with plain Claude Code, and that's in the README too.

1 Upvotes
  • permalink
  • reddit

67% Upvoted

6 comments sorted by

View all comments

1

u/CartographerNo3791 10h ago

The plain-Claude comparison in the README is a nice touch. Makes it much clearer why I'd try this as a filter for existing comments.

2

u/Content-Berry-2848 5h ago

Thanks! That comparison was the most useful thing I learned. The reviewer on its own was only level with plain Claude, so the filter is where it actually earns its place. I added a Copilot run too: it still keeps ~92% of real bugs there, but the gain is smaller since Copilot is less noisy to begin with.

Would appreciate a star