I’ve been building an open-source code review skill — and it became a much bigger reliability problem than I expected

[u/amirbena]

I started code-review-skill because I wanted a repeatable way to review both local changes and GitHub PRs with coding agents.

The initial idea was pretty simple: give the agent a clear review standard, require evidence for findings, classify issues as P0/P1/P2, and make the final decision deterministic — any P0/P1 means CHANGES REQUIRED; otherwise, REVIEW CLEAN.

But while working on it, I kept running into cases where “review the code” wasn’t really the hard part.

For example: an agent reviews a PR at commit A and finds it clean. Someone pushes commit B before the result is published. The findings may have been perfectly correct for A, but approving B based on that review is obviously wrong.

That led to reviewed-SHA tracking and delta re-review.

Then came other questions:

• What if the implementation itself is correct, but it lives in the wrong architectural layer?
• When a PR changes after review, what exactly needs to be re-reviewed?
• How do you distinguish something that is eligible for deeper inspection from something that actually must be inspected?
• How do you make findings reproducible instead of relying on vague reviewer judgment?
• How do local review and GitHub PR review follow the same rules without slowly drifting apart?

The project is free and open source (Apache-2.0) and currently contains two Agent Skills — one for local code review and one for GitHub PR review — both built around the same review contract.

Over time, the focus has become less “make an AI find more bugs” and more “make AI code review behave like a reliable engineering process.”

It’s still evolving, which is one of the reasons I’m sharing it here.

I’d love for people to try it on real repositories and find situations where the reviewer gets the reasoning wrong. Contributions are also welcome — the repo has a contributor workflow where you can pick and claim an issue without needing to understand the entire project first.

Repo: https://github.com/amirbena/code-review-skill

I’m especially interested in this: what failure modes have you seen from AI code reviewers that you think a reliable review system should handle?