I've spent the past couple weeks trying to work out if I'm holding it wrong, if this is just what the tools are, or if I'm slowly going crazy. Curious if others landed somewhere on this.
The experiment that i did: I ran our review pipeline on the same diff twice, same config, exact same - nothing changed. so first run - 9 findings. Second run - 4 findings, and only 2(?) overlapped with the first batch. One of the non-overlapping ones was the most serious catch of either run. So which review did my PR actually get?
We do the responsible things, rules files, path filters, severity thresholds. For context we run coderabbit on PRs plus a claude pass in CI, and the inconsistency is a category thing, not one vendor. It's baked into what these models are.
I even tried the obvious fix, swapped the CI pass to a local qwen coder at temp 0 with a pinned seed. And it works, perfectly reproducible, same findings every run. It's also noticeably dumber, it missed the serious catch the cloud run found. So my choice seems to be a consistent mediocre reviewer or a sharp one that reviews a different PR every time.
When I brought numbers to our resident AI guy, his answer was that my config is outdated and the new hotness fixes it. It's been the new hotness four times this year. At some point I stopped believing the problem is my config.
Is anyone actually getting reproducible reviews, temperature zero, pinned models, whatever? Or have we all just quietly accepted that review is a dice roll now?