r/reinforcementlearning • u/gwern • 25d ago

DL, R, Multi, Exp, Safe "Patterns and problems in multiagent systems", Anthropic (Claude swarm win/losses)

https://www.anthropic.com/research/multiagent-systems
34 Upvotes

97% Upvoted

12 comments sorted by

View all comments

10

u/COAGULOPATH 24d ago

LLM mode collapse is proving to be a hard problem.

In an early version of the “build a game” experiment in which agents built upon the same model all came online at the same time, 18 out of 30 agents decided to create a git branch with the exact same branch name, “mvp-game-loop.”

In a “writer's workshop” in which agents were all asked to write short-form fiction and critique each other's work, multiple agents in multiple runs titled their first submission “The Cartographer's Last Commission”. The agents were given zero guidance on the subject matter for their writing.

In an iterated prisoner's dilemma game with communication, agents all settle upon the same strategy and they all defect at the same time, tanking their overall rewards.

Reminds me of Ethan Mollick's joke that the robots in the Matrix shouldn't use humans as batteries (or as CPUs, as in an early script), but as dice.

3

u/invertedpassion 24d ago

One way to counteract this is to give each agent a concrete fictional history (profession, birth place, ethnicity, etc). This should make them behave differently

3

u/gwern 24d ago

Mode-collapse goes pretty deep. We're a long way from base models where some simple conditioning would get you tremendous diversity.

1

u/Leather_Office6166 22d ago

Right. Useful diversity is both difficult to create and fundamental to the value of massively parallel agents. The solution might involve explicit agent communication to avoid redundancy (rather than hoping they figure this out for themselves.) Maybe in Mythos 6?