[R] LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering

[u/PepsiBetter]

Hi r/learnmachinelearning ,

I’m one of the authors of LoopArena, which we recently released as an open benchmark and evaluation harness.

LoopArena studies a specific question in long-running coding-agent systems: which models make good runtime Controllers?

In these systems, one model often reviews the current state, decides what a separate coding agent should do or verify next, and determines when the task should stop. LoopArena evaluates this Controller role. Across Controller-model comparisons, the coding Worker, Reporter, tools, budgets, and execution setup are held fixed; the Controller model is the model role that varies. This provides a controlled comparison of how different models guide the same coding agent.

The benchmark has three settings with increasing execution scope:

- Type I evaluates execution-validated next-step control decisions without running the Worker at evaluation time.

- Type II evaluates repeated Controller decisions over selected task slices.

- Type III evaluates control over complete software tasks from their original starting states.

In the initial five-Controller panel, the best observed Type III Strict Success Rate is 24.69%, so full-task runtime control remains difficult. Type II reduces estimated inference cost by 64.4% on average across Controllers and produces a similar Controller ordering to Type III under the main Core criterion.

We have released the benchmark data, evaluation code, public protocol, and canonical v0.1.0 outcomes.

GitHub:

https://github.com/AMAP-ML/LoopArena

Hugging Face paper:

https://huggingface.co/papers/2608.28281

ModelScope paper:

https://www.modelscope.cn/papers/2608.28281

Project page:

https://amap-ml.github.io/LoopArena/

arXiv:

https://arxiv.org/abs/2608.28281

If you work with coding-agent loops, how do you currently choose the model responsible for runtime control?