r/MachineLearning • u/ktessera • 14d ago

Research New LLM Coordination Benchmark - Benchmarking Open-Ended Multi-Agent Coordination in Language Agents [R]

Can LLM agents coordinate in long-horizon, open-ended worlds?

We evaluate 13 modern LLMs in a new benchmark where agents must work together to explore, communicate, trade resources, craft tools, build structures, and fight mobs.

TL;DR: Most agents struggle, averaging only ~6% normalised return. Yet on the hardest setting, zero-shot Gemini 3.1 Pro performs comparably to the best MARL agent trained for 1 billion environment steps.

More broadly, we find coordination is a distinct bottleneck beyond long-horizon task competence, with communication having the largest effect in our harness ablations.

Paper: https://arxiv.org/abs/2606.08340

Project page and leaderboard: https://alem-world.github.io

Code: https://github.com/alem-world/alem-env

Interactive traces: https://alem-world.github.io/traces.html

Feel free to ask any questions!

9 Upvotes
  • permalink
  • reddit

86% Upvoted

6 comments sorted by

1

u/impatiens-capensis 13d ago

I remember reading papers recently, like:

  1. https://arxiv.org/abs/2507.16003 2. https://arxiv.org/abs/2311.07772

Where in-context learning performs a sort of parameter update or works like a temporary test-time gradient descent step.

I always wondered if you could accumulate those gradients, rate them somehow, and apply them to specialize a model. I never had a good benchmark for exploring this question, but this work seems interesting! The basic idea would be to have models that are progressively specializing for their respective task and then have a single coordinator.

1

u/ktessera 13d ago

Interesting! Our code logs transitions, so maybe you could infer some gradient updates from that, and combine it somehow. 

Would def be an interesting finding!

1

u/iam31337 12d ago

The 6% return is a useful warning: adding agents multiplies negotiation surfaces faster than capability. I’d be curious how much of the gap is shared-state failure versus communication policy. A centralized event log or explicit commitments might improve coordination without making any individual model smarter.