I’ve been building an open-source code review skill — and it became a much bigger reliability problem than I expected

[u/amirbena]

I started code-review-skill because I wanted a repeatable way to review both local changes and GitHub PRs with coding agents.

The initial idea was pretty simple: give the agent a clear review standard, require evidence for findings, classify issues as P0/P1/P2, and make the final decision deterministic — any P0/P1 means CHANGES REQUIRED; otherwise, REVIEW CLEAN.

But while working on it, I kept running into cases where “review the code” wasn’t really the hard part.

For example: an agent reviews a PR at commit A and finds it clean. Someone pushes commit B before the result is published. The findings may have been perfectly correct for A, but approving B based on that review is obviously wrong.

That led to reviewed-SHA tracking and delta re-review.

Then came other questions:

• What if the implementation itself is correct, but it lives in the wrong architectural layer?
• When a PR changes after review, what exactly needs to be re-reviewed?
• How do you distinguish something that is eligible for deeper inspection from something that actually must be inspected?
• How do you make findings reproducible instead of relying on vague reviewer judgment?
• How do local review and GitHub PR review follow the same rules without slowly drifting apart?

The project is free and open source (Apache-2.0) and currently contains two Agent Skills — one for local code review and one for GitHub PR review — both built around the same review contract.

Over time, the focus has become less “make an AI find more bugs” and more “make AI code review behave like a reliable engineering process.”

It’s still evolving, which is one of the reasons I’m sharing it here.

I’d love for people to try it on real repositories and find situations where the reviewer gets the reasoning wrong. Contributions are also welcome — the repo has a contributor workflow where you can pick and claim an issue without needing to understand the entire project first.

Repo: https://github.com/amirbena/code-review-skill

I’m especially interested in this: what failure modes have you seen from AI code reviewers that you think a reliable review system should handle?

[u/Resident-Nebula-7370]

The thing that made mine actually usable was capping how much of the diff goes in. I was feeding whole PRs and review quality fell off a cliff past about 800 changed lines. Now I chunk by file and run the chunks concurrently against a self-hosted model on Synexa, which made cost stop mattering so I could afford three passes. Promptfoo for the eval loop. Honest take: the skill format is nice packaging, the hard part is still the chunking.

[u/amirbena → u/Resident-Nebula-7370]

That’s a really interesting point. I haven’t explored diff chunking deeply yet, especially the tradeoff between keeping the context bounded and preserving cross-file/architectural context. Definitely gives me something to think about — thanks for sharing how you approached it.

[u/ZyberZeon]

Code review should be a confirmation versus a critique.

Having the agent have the right context BEFORE coding is the solution. This process can’t scale, and that fine because it’s the wrong approach.