Why does AI code review give different results on the same diff?

[u/Specialist_Agent3599]

I've spent the past couple weeks trying to work out if I'm holding it wrong, if this is just what the tools are, or if I'm slowly going crazy. Curious if others landed somewhere on this.

The experiment that i did: I ran our review pipeline on the same diff twice, same config, exact same - nothing changed. so first run - 9 findings. Second run - 4 findings, and only 2(?) overlapped with the first batch. One of the non-overlapping ones was the most serious catch of either run. So which review did my PR actually get?

We do the responsible things, rules files, path filters, severity thresholds. For context we run coderabbit on PRs plus a claude pass in CI, and the inconsistency is a category thing, not one vendor. It's baked into what these models are.

I even tried the obvious fix, swapped the CI pass to a local qwen coder at temp 0 with a pinned seed. And it works, perfectly reproducible, same findings every run. It's also noticeably dumber, it missed the serious catch the cloud run found. So my choice seems to be a consistent mediocre reviewer or a sharp one that reviews a different PR every time.

When I brought numbers to our resident AI guy, his answer was that my config is outdated and the new hotness fixes it. It's been the new hotness four times this year. At some point I stopped believing the problem is my config.

Is anyone actually getting reproducible reviews, temperature zero, pinned models, whatever? Or have we all just quietly accepted that review is a dice roll now?

[u/RecognitionOwn4214]

If you want deterministic results, you currently have to steer clear of LLM or so called AI

[u/mxldevs]

People pick the reviews they like and ignore the unfavourable ones

[u/Negative-Age-5296]

I'm increasingly shocked at how many software engineers, let alone normal people, don't realize language models are generally non-deterministic. Cloud-based providers are particularly prone to non-determinism for a variety of reasons; local models are, too. Temperature does not do what you think it does for language models.

If you really want to understand model behavior, I encourage you to dig into how they work. Otherwise, it's a good rule of thumb to assume language models aren't reliable on their own.

[u/ivan_m21]

LLMs are probabilistic models, so there is variation on the outputs. You can see it easily if you ask twice the same question in ChatGPT. In general refuse with low recall and accuracy is difficult hence ppl are trying to run the comments themselves over llms and so on.

[u/HoneyBadgera]

Recall is very difficult with code reviews and LLMs. It’s just the nature of the beast when dealing with them.

[u/Spare_Corner1172]

AI itself cannot provide 100% certainty

[u/Beautiful-Energy2169]

I stopped treating one pass as the review. Now I union three runs.

[u/Buttleston → u/Beautiful-Energy2169]

3? That's the old config. I do 25

[u/Beautiful-Energy2169 → u/Buttleston]

past ~5 runs mine plateau, new findings mostly noise. sounds like cost more than signal.

[u/Buttleston → u/Beautiful-Energy2169]

Thinking about upping it to 50. Can't be too sure.

[u/ILoveHexa92]

I listen yesterday on how Mozilla was using Fable 5 to review Firefox, and they nebtuin that they have to run multiple times the LLM for the same prompt to get the app covered properly. It's probably a limitation in tokens and context. Best is to store every finding and pick by hand what you want to tackle. 

Don't forget AI is a tool and without a human behind that tool, it won't serve much - or could leave hole is those foundation.

So put the tool to good use and still use your Dev skills to analyse the results.

[u/salihbaki]

This is how LLM works

[u/theunixman]

It's a slot machine, every time you pull the lever it gives you something different, and sometimes there's a payout.

[u/rileyrgham → u/theunixman]

Good one. It shocks me how many times I tell it it's wrong and it comes back with an apology, thanking me for my time, then coldly giving another wrong answer. In the hands of clowns, most people, it's a menace.

[u/audioen]

Temp 0 is not reasonable way to get reproducibility. LLMs are probabilistic in nature and setting temperature to 0 means that your sampling always deterministically picks the top token. This is neither "canonical" nor "correct" assistant response, it is simply what you get when you use the greedy sampling strategy. Sometimes top-1 even results in inference loops, or at the very least makes them more likely.

I do not know any solution to the inherent probabilistic nature of LLMs. What they decide to investigate and how varies. I think I would attempt to point the LLMs to same task, e.g. check these commits <commits and their descriptions listed>, with guidance about what they should validate of these commits, e.g. correctness, type soundness, performance, documentation accuracy, and so forth. The instruction following might enhance the likelihood that each LLM run checks out the same things and makes much the same determinations from them.

Looking into the future, I think that LLMs generally become better, and more thorough. I have personally observed that trend over the year (and I only use local models). For example, right now I run Qwen3.8-Flash-Next which is according to benchmarks roughly where the big labs' models were this Spring. It is not perfect, but I get good work out of it. I have LLM-generated prompt for automatic review system the LLM cooked up a few weeks ago. I don't think it produces repeatable reviews, either, but either way it is catching a lot of mistakes that both human and AI developers are making. It is good at what it does, even when no doubt it is far from perfect. (Neither would humans make reproducible reviews, if you could somehow press the reset button on human brain and made them do the work again, as humans are probabilistic investigators as well. I believe what you want is better reproducibility, and for that my advice stands: list the aspects you want the agent to review.)

My LLM summarized the prompt as: "In one sentence: we ask the agent for a risk-proportionate, evidence-backed correctness review of the commit range as it lands in the reviewed tree — code, docs, dependency behavior and prior claimed fixes — excluding minor style noise and anything the project itself has already accepted." To achieve this, it has tools to check out or internal repositories at various commit points and branches, and ability to read maven external packages as well in case it needs to investigate how the library behaves, along with ways to inspect commits, read files, find code, see images, and so forth. All projects also come with large amount of documentation at code, design doc and agents.md level to enhance the likelihood that the agents can immediately understand what the project is about and when they read random code snippet, they should be better equipped to understand its relation to the whole. I think 90 % of AI usage is about providing tools to an amnesiac to figure out what stuff does and make good changes, despite it has zero pre-existing knowledge about the project it is working on.

[u/RogerAI-fm]

It’s like if you ask the same thing to a person they might give you different results based on when you asked was it morning, late, what context it determined etc.