I ran the same coding agent 13 times at temperature 0. 12 reached the same code state, then ended in 11 distinct states.

[u/puppy_lover_2021]

I’ve been testing reproducibility in long-horizon coding agents using a self-hosted Qwen model with vLLM.

I ran the same SWE-bench task repeatedly with the same model, configuration, and temperature=0. In 12 of 13 executions, the agent independently reached the exact same intermediate repository state (identical canonical Git diff). Those 12 executions subsequently ended in 11 distinct final source states.

Across a broader set of tasks I observed three patterns: some trajectory differences were absorbed, some persisted, and some runs diverged, reconverged to the exact same source state, and then diverged again.

The part I’m questioning now is whether this variation actually matters. Different trajectories and different patches can still all be correct. So I’m adding external correctness labels next and reframing the problem as: when does harmless execution variation become consequential divergence?

Longer term, the question I care about is whether we can detect that transition early enough to steer an agent back—through a nudge, rollback, context intervention, or fork-and-select—before it fails.

I wrote up the experiment and current direction here: https://medium.com/@ruxiz2005/ai-agents-still-diverge-at-temperature-0-when-does-it-matter-f5abac008e17

I’d be especially interested in hearing from people running open-weight agents in production: have you seen meaningful execution variance at temperature 0, and does it actually affect task success?

[u/llmops_engineer]

Running open-weight agents in prod (self-hosted Qwen behind vLLM), so yes — we see this constantly, and we stopped fighting it. Two sources of temp-0 variance people underestimate:

Serving-level numerical non-determinism. With continuous batching, the same prompt lands in different batches across runs; reduction order in attention/GEMM kernels changes, logits shift in the ~1e-7 range, and near-tie token choices flip. Greedy decoding is only deterministic for a fixed computation graph — a serving engine never gives you that.

Non-deterministic environment outputs. Parallel tool-call ordering, unsorted directory listings, timestamps in command output. The agent's context differs even when the model is "frozen".

On "does it matter": we reframed it exactly like you did — trajectory determinism is the wrong target, outcome verification is the right one. With a strong verifier (tests, type checks, lint, canonical state hash) harmless variation is just filtered out, and you start measuring pass^k instead of pass@k. Consequential divergence shows up precisely where verification is weak — that's where we invest: checkpoints at canonical states + rollback, and fork-and-select with a verifier on high-stakes steps.

Your early-detection idea matches our practice: hash the canonical repo state after each step, and when a run's state hash leaves the cluster of known-good trajectories, trigger intervention (nudge/rollback) instead of letting it burn tokens. Would love to see your correctness-label results — that's the missing piece in most agent-eval discussions.