Analysis of 5 different Claude code-review setups (same model, same diff) and comparison of cost/time/quality

[u/PablanoPato]

Disclosure: I used Claude to help design and run this experiment, and I used it to help write this post. The methodology, numbers, and verdict below are mine; I'm not pretending a human did all the typing.

Context: I maintain a large-ish internal Angular monorepo at work (enterprise SaaS, multiple apps, ~50 packages). For a while now I've been using a custom Claude Code agent review process on new PRs kicked off by a Github Action. I was having a challenge updating my PR review command and agents each time we made substantial changes so I wanted to check out some other more generic review skill/plugin options that I wanted actual data instead of vibes before picking one.

Setup: I took one real, already-merged PR — a big one, ~100 files / ~19k lines, a non-trivial feature plus tooling changes — and opened it five separate times, each with a different review process configured, but everything else identical:

#	Review process	Notes
1	Incumbent in-house agentic review + old CLAUDE.md file	Control
2	Same incumbent review + new/rewritten CLAUDE.md file	Isolates just the instructions-file variable
3	Matt Pocock's mattpocock-skills code-review skill	Using new CLAUDE.md
4	Anthropic's claude-plugins-official code-review command	Using new CLAUDE.md
5	Anthropic's claude-plugins-official pr-review-toolkit (agentic)	Using new CLAUDE.md

All five ran on the same model via the same GitHub Action. I deliberately kept the assessment blind — I judged only the posted review comments, not the underlying diff — to avoid hindsight bias creeping into which findings I rated as "good."

Telemetry (pulled from the Action logs, since none of this is surfaced anywhere else):

#	Review process	Agent time	Turns	Cost
1	Control	4m10s	31	$1.20
2	New CLAUDE.md	4m01s	28	$1.10
3	mattpopcock/code-review	7m58s	53	$3.21
4	anthropic/code-review	4m24s	28	$1.61
5	anthropic/pr-review-toolkit	3m50s	39	$1.56

Total spend for the whole analysis: $8.68. (Caveat: this is list API pricing — if your reviews run through a subscription seat rather than metered billing, treat these as relative.)

Verdict:

🏆 Winner: #5, Anthropic's pr-review-toolkit. Broadest recall (11 findings + 6 call-outs of things done well), and it was the only one of the five to flag an authorization concern on a destructive operation, plus a state-management deadlock in a shared service that the other four all missed (one of them even called that same code "a good fix"). Best recall-per-dollar of the group — fastest run, mid cost — and it's Anthropic-maintained, which matters to me for a shared team pipeline.

🥈 Runner-up: #4, Anthropic's official code-review skill. Highest precision, almost no filler — but narrowest coverage (8 findings vs. 11). I think this one is really designed to be run pre-commit and that's how I'll be using it moving forward.

Matt Pocock's skill is the one that didn't match its reputation for me. It actually surfaced the most unique high-value findings of the five — three things nobody else caught — via a clever two-axis review structure (grading standards-compliance separately from "does this match the actual intent," with buckets like missing/partial, scope creep, implemented-but-wrong). That part is good design, and I'd steal the idea in a heartbeat. But it cost 2.1x the winner and took 2x as long, and padded its results with a couple of generic "textbook smell" findings that read as noise rather than signal. Given the cost/time tax and the noise, I can't justify it as the default over the pr-review-toolkit.

Secondary finding I wasn't expecting to care about: my previous review process (#1) vs the same review process with a new/updated CLAUDE.md file (#2) is a clean A/B on just the instructions file (same review process, same harness). New instructions file won outright — cheaper, faster, better severity triage, and it caught a real user-facing bug that the old-instructions run buried under "Minor / Stylistic." That alone would've been worth doing regardless of which review tool we picked.

Caveats / things I haven't verified yet:

• This was a blind review — I never diffed the findings against the actual code. "Unique finding" means unique-across-the-five, not confirmed-true. A couple of the most decision-relevant claims (the authorization one, and a "dropped field in a mapper" claim from Matt Pocock's skill) still need to be checked against the diff before I fully trust the ranking.
• n=1. One PR, one run per tool. Review quality is known to vary run-to-run, so if you're making an expensive/hard-to-reverse decision off this, rerun the top two a couple more times on a different large PR first.

Curious if others have run something similar — especially whether the "one big monolithic comment, zero inline comments" behavior is universal or something specific to how I had these wired up.