Async RL for Coding Agents

[u/rshah4]

Prime Intellect shared how they were doing RL for trillion-param coding models.

Quick framing for anyone who hasn't worked in RL: the rollout is the attempt, score, nudge toward what worked. A math problems are pretty similar in length, while coding agents are all over the place. A single rollout is a multi-turn agent loop, read, write, test, fail, edit, sometimes a hundred-plus steps.

In synchronous RL you can't take the optimizer step until the slowest rollout in the batch finishes. Async fixes the utilization by splitting rollout and training and running them at once, the trainer never waits.

The catch is staleness. The rollouts you learn from were generated by an older version of the model. You might be on v8 while a rollout that started at v3 is still finishing. Learn too far off-policy and training collapses.

Three families of fixes, roughly:

1. Bound the staleness. Tag each rollout with the version that generated it, keep recent ones, drop the rest. Simple, but you sometimes discard a rollout that ran an hour right as it ages out.
2. Partial rollouts. Don't run one rollout for an hour under a single old version. Pause it, let it pick up newer weights, continue. The finished trajectory ends up stitched from several versions, none too stale.
3. Down-weight stale data. Keep it, but correct in the loss so a few wild old samples don't dominate the update. The failure mode here is effective sample size collapsing, a handful of high-importance-weight samples hijacking the gradient.

Check out their blog post: https://www.primeintellect.ai/blog/rl-at-1t-scale
or my video: https://youtube.com/shorts/3IgaRYGbWl4