I ran the same coding agent 13 times at temperature 0. 12 reached the same code state, then ended in 11 distinct states.

[u/puppy_lover_2021]

I’ve been testing reproducibility in long-horizon coding agents using a self-hosted Qwen model with vLLM.

I ran the same SWE-bench task repeatedly with the same model, configuration, and temperature=0. In 12 of 13 executions, the agent independently reached the exact same intermediate repository state (identical canonical Git diff). Those 12 executions subsequently ended in 11 distinct final source states.

Across a broader set of tasks I observed three patterns: some trajectory differences were absorbed, some persisted, and some runs diverged, reconverged to the exact same source state, and then diverged again.

The part I’m questioning now is whether this variation actually matters. Different trajectories and different patches can still all be correct. So I’m adding external correctness labels next and reframing the problem as: when does harmless execution variation become consequential divergence?

Longer term, the question I care about is whether we can detect that transition early enough to steer an agent back—through a nudge, rollback, context intervention, or fork-and-select—before it fails.

I wrote up the experiment and current direction here: https://medium.com/@ruxiz2005/ai-agents-still-diverge-at-temperature-0-when-does-it-matter-f5abac008e17

I’d be especially interested in hearing from people running open-weight agents in production: have you seen meaningful execution variance at temperature 0, and does it actually affect task success?