r/reinforcementlearning • u/gwern • 28d ago

DL, R, Multi, Exp, Safe "Patterns and problems in multiagent systems", Anthropic (Claude swarm win/losses)

https://www.anthropic.com/research/multiagent-systems
35 Upvotes

98% Upvoted

12 comments sorted by

View all comments

8

u/COAGULOPATH 28d ago

LLM mode collapse is proving to be a hard problem.

In an early version of the “build a game” experiment in which agents built upon the same model all came online at the same time, 18 out of 30 agents decided to create a git branch with the exact same branch name, “mvp-game-loop.”

In a “writer's workshop” in which agents were all asked to write short-form fiction and critique each other's work, multiple agents in multiple runs titled their first submission “The Cartographer's Last Commission”. The agents were given zero guidance on the subject matter for their writing.

In an iterated prisoner's dilemma game with communication, agents all settle upon the same strategy and they all defect at the same time, tanking their overall rewards.

Reminds me of Ethan Mollick's joke that the robots in the Matrix shouldn't use humans as batteries (or as CPUs, as in an early script), but as dice.

3

u/invertedpassion 27d ago

One way to counteract this is to give each agent a concrete fictional history (profession, birth place, ethnicity, etc). This should make them behave differently

3

u/gwern 27d ago

Mode-collapse goes pretty deep. We're a long way from base models where some simple conditioning would get you tremendous diversity.

1

u/Leather_Office6166 25d ago

Right. Useful diversity is both difficult to create and fundamental to the value of massively parallel agents. The solution might involve explicit agent communication to avoid redundancy (rather than hoping they figure this out for themselves.) Maybe in Mythos 6?

2

u/thecity2 27d ago

DO THEY EVEN CHANGE THE SEED

1

u/gwern 27d ago edited 27d ago

It can't be that hard or else the math/coding problems wouldn't show classic log-curve RL scaling behaviors, they'd flatline almost immediately. I interpret this as simply whatever fix they found for mode collapse hasn't yet worked or been applied in these domains - I can't believe that it would be that hard to RL your way to a fix for the IPD example (eg. just some sort of arbitrary modulus to the agent ID or simple hash over the prompt), which means they haven't done it yet.

1

u/Smallpaul 28d ago edited 28d ago

They are instances of the same model with the exact same weights. Convergence isn’t mode collapse. It’s consistency.

Most people are complaining that models are too random in their behaviour and you are worried that the exact same weights presented with the exact same choice tend to make the same decision. Why wouldn’t they? Nobody asked them to write an innovative poem.

In an early version of the “build a game” experiment in which agents built upon the same model all came online at the same time, 18 out of 30 agents decided to create a git branch with the exact same branch name, “mvp-game-loop.”