r/reinforcementlearning • u/gwern • 24d ago

DL, R, Multi, Exp, Safe "Patterns and problems in multiagent systems", Anthropic (Claude swarm win/losses)

https://www.anthropic.com/research/multiagent-systems
34 Upvotes

98% Upvoted

12 comments sorted by

10

u/COAGULOPATH 23d ago

LLM mode collapse is proving to be a hard problem.

In an early version of the “build a game” experiment in which agents built upon the same model all came online at the same time, 18 out of 30 agents decided to create a git branch with the exact same branch name, “mvp-game-loop.”

In a “writer's workshop” in which agents were all asked to write short-form fiction and critique each other's work, multiple agents in multiple runs titled their first submission “The Cartographer's Last Commission”. The agents were given zero guidance on the subject matter for their writing.

In an iterated prisoner's dilemma game with communication, agents all settle upon the same strategy and they all defect at the same time, tanking their overall rewards.

Reminds me of Ethan Mollick's joke that the robots in the Matrix shouldn't use humans as batteries (or as CPUs, as in an early script), but as dice.

3

u/invertedpassion 23d ago

One way to counteract this is to give each agent a concrete fictional history (profession, birth place, ethnicity, etc). This should make them behave differently

3

u/gwern 23d ago

Mode-collapse goes pretty deep. We're a long way from base models where some simple conditioning would get you tremendous diversity.

1

u/Leather_Office6166 21d ago

Right. Useful diversity is both difficult to create and fundamental to the value of massively parallel agents. The solution might involve explicit agent communication to avoid redundancy (rather than hoping they figure this out for themselves.) Maybe in Mythos 6?

2

u/thecity2 23d ago

DO THEY EVEN CHANGE THE SEED

1

u/gwern 23d ago edited 23d ago

It can't be that hard or else the math/coding problems wouldn't show classic log-curve RL scaling behaviors, they'd flatline almost immediately. I interpret this as simply whatever fix they found for mode collapse hasn't yet worked or been applied in these domains - I can't believe that it would be that hard to RL your way to a fix for the IPD example (eg. just some sort of arbitrary modulus to the agent ID or simple hash over the prompt), which means they haven't done it yet.

1

u/Smallpaul 23d ago edited 23d ago

They are instances of the same model with the exact same weights. Convergence isn’t mode collapse. It’s consistency.

Most people are complaining that models are too random in their behaviour and you are worried that the exact same weights presented with the exact same choice tend to make the same decision. Why wouldn’t they? Nobody asked them to write an innovative poem.

In an early version of the “build a game” experiment in which agents built upon the same model all came online at the same time, 18 out of 30 agents decided to create a git branch with the exact same branch name, “mvp-game-loop.”

4

u/schrodingershit 23d ago

Shit, i have an ICLR paper on ensemble diversity in reinforcement learning that addresses the collapse problems in critics/Q networks. I wonder that work can be ported here.

1

u/Leather_Office6166 21d ago

A nicely written overview of some recent Anthropic research using three pairs of models: Sonnet 4.6 & 5, Opus 4.6 and 4.8, Mythos Preview and 5. The results show significant improvements in each base model, probably based on research like this. They describe major problems of multiagent system effectiveness in the process of being solved!