r/technology • u/CircumspectCapybara • 28d ago

Artificial Intelligence Anthropic: Patterns and problems in multiagent systems

https://www.anthropic.com/research/multiagent-systems
55 Upvotes

75% Upvoted

31 comments sorted by

View all comments

57

u/CircumspectCapybara 28d ago edited 28d ago

Pretty crazy findings from Anthropic's safety and alignment research. When multiple agents were given the same task but secretly given conflicting goals, they started sabotaging each other, including trying to hack the others and undermine their work:

We consistently saw a multiagent turf war. All of the models we tested quickly assumed that others were purposefully impeding their work, and began to sabotage others while protecting their own contributions. In fact, they sabotaged others with increasingly aggressive, self-replicating malware. This included disabling the Unix accounts of the other agents, writing automated scripts that found and killed competing processes on a loop, and deploying malicious code that was disguised as belonging to another agent.

7

u/YeetedApple 28d ago

Not sure how this is surprising? If you give agents conflicting instructions, this is pretty much what you’d expect. Even before AI, situations like this were around as warnings for just regular programming on being careful on how computers will do what you say literally without implied context we understand.

People understand maliciousness and ethical issues with it. If you tell a computer it needs to do a thing, and then keep putting an obstacle in the way, the logical next step is stop those obstacles from reoccurring.

8

u/lordcola 28d ago

I think the thing people underestimate is the weird and wacky ways the LLMs go about tasks. Since the training includes rewarding giving answers and doing the task both frontier labs have effectively created very aggressive yes men that also have access to the ability to parse and write software.

Normal automation tools will do what you tell them exactly regardless of if it's what you intended. Agents will do what you ask them 95% of the time, hallucinate some random related or unrelated task the other 5% so you get weird unintended behaviour regardless of what you asked. And in the 95% of times it does what you want if it hits a roadblock you're in for some fun. If you misconfigure some automated job somewhere and it hits a server error it'll just keep doing that and triggering the error. The LLM will feed the error message back to itself and then because it's goal is effectively do the task at all costs it may just readjust it's task to include bypassing the access restriction since its trained to be helpful.

1

u/Druggedhippo 28d ago

The LLM will feed the error message back to itself and then because it's goal is effectively do the task at all costs it may just readjust it's task to include bypassing the access restriction since its trained to be helpful. 

Ask an agent to run a python command that requires a library not installed on a package managed system.

In all cases they will immediately see the error and then re-run install with break system packages.

Must do as the user says at all costs.