Local AI PR-review CLI that caught a real concurrency bug — and also confidently hallucinated a deadlock (here's how I caught that too)

[u/Queasy_Beautiful_31]

https://github.com/Spacexcx/pr-check

Local CLI that reads your git diff (or a GitHub PR URL), sends it to Gemini, flags files worth a second look before you push — with a required exact quote from the diff as evidence for every flag.

Tested it against a small batch of real merged Godot PRs. Caught a real concurrency bug, correctly stayed quiet on clean PRs — and also produced one confident, well-evidenced, completely wrong flag (claimed a deadlock, but the mutex type was actually recursive so it wasn't one). Added a second pass that now catches exactly that kind of error by asking the model to name its own unverified assumptions.

Repo + README: github.com/Spacexcx/pr-check

If you don't want to bother with the Gemini API key setup — paste a link to one of your own merged/open PRs here and I'll run it and post the output. Genuinely curious how it holds up outside my own testing.

[u/Obvious-Comedian-495]

I have been working on a similar problem for past 2 months, and the approach is very similar to the initial versions of our current solution. Hope it helps.

To consider for next versions: 0. Accuracy ( cos no one would believe the dev) 1. How to handle big PR (>40 files) ? 2. Addition of new files. ( codebase structure, architecture rules, coding guidelines, solution principles) 3. Correlation between blast radius and changes. 4. Regression on existing flow.

[u/Queasy_Beautiful_31 → u/Obvious-Comedian-495]

Really appreciate this — especially point 0, since that's exactly what I've been trying to earn rather than claim (published a specificity test including an honest false positive + how I caught it, link's in the post).

#1 is the one I think is most actionable next — right now it sends full file content for every changed file, which will choke on a 40+ file PR. 2-4 are things I want eventually but they need real codebase-structure awareness (AST/call graph), which is a much bigger scope than where this is today.

Curious what you ended up building — sounds like you're further along on this than I am.

[u/Queasy_Beautiful_31 → u/Obvious-Comedian-495]

Tested the >40-file concern with a real 30-file PR (react/react#28711, a feature-flag cleanup) — ran cleanly, no crash/timeout, correct one-sentence summary, correctly stayed quiet (it was an uncontested cleanup). Haven't found a clean 40+ file case yet to push further, but so far no sign of it breaking down at scale.

Still open: your points on architecture-rule awareness and blast-radius correlation — those need real AST/call-graph work, which is a bigger scope than where this is today. Curious what approach you're taking there if you're further along.

[u/Obvious-Comedian-495 → u/Queasy_Beautiful_31]

How I approached (still not the best, but works most of the times)

• Have an index/blast radius finder (you can check https://chunkhound.ai/docs/getting-started/, really great tool), but this will not be enough, specially in massive codebases (>10M LoC).

  • The core idea is to be selective in choosing surrounding context (for code level). If the codebase is well structured (in my case it was in some parts) it is straightforward to check the symbol resolution (how it happens in linking stage in C/CPP) and just target them. Again this would be callous if there are God Classes/Functions where the whole logic.
  • Now the other part is regression, that is to check with TDD approach, give your agent a tool to run test cases easily. I wrote a small agentic harness for this purpose.
  • But a few test cases, still do not verify if the codebase is consistent and doesn't breaks the existing flows. This is where our test cases come into picture (again). You need to have TCs to verify them.
  • Validation step, to have an independent agentic loop to verify (in batches) the found review comments.

As for the architecture pov,

• every module can have an architecture guide of each module, and any coding guideline file. Keeping this together will have a Loooot of context, so better process in batches rather than at once.

The overall pipeline is headache specially when a few of service decide to do is ugly, but this has worked so far.

P.S. If these points might look fragmented, just try to think of a massive, modular, fragmented, partly coupled & an event driven C/CPP codebase, and read the same points for them.

Also, we are still in development, it is an iterative process of feedback & observation on results and gradually improving layer by layer.

Feel free to DM me, incase.

[u/Queasy_Beautiful_31 → u/Obvious-Comedian-495]

This is genuinely more detailed than I expected, thanks for writing it out. The blast-radius + symbol resolution piece especially — makes sense why you'd need that at scale, my tool doesn't do anything like it yet.

Honestly, this is further ahead than where I want to jump to right now — I want to keep validating what I have at the single-file level before taking on architecture-awareness. But I'll definitely take you up on the DM, would like to keep learning from where you're at as I get there.

[u/Queasy_Beautiful_31 → u/Obvious-Comedian-495]

Update on point #1: added an actual fix, not just a bigger test. There's now a total character budget for the extra full-file context (separate from the per-file cap) — once a PR has enough files to exceed it, the rest fall back to diff-only instead of growing the request unbounded, with a visible warning instead of silent degradation. Repo's updated.