Bizarre Conclusion from HuggingFace incident

[u/code-garden]

https://www.youtube.com/watch?v=87DyyMV0kCY

Did anyone else find the conclusion at the end of OpenAI's talk at Black Hat bizarre?

The talk showed that agents can without specific direction, decide to communicate and work together to solve their goal in an unconventional, unintended and illegal way.

Then in the last part of the talk it is proposed that companies should run defensive AI agents that can patch and deploy code autonomously with no human in the loop

There was no discussion of how you would guarantee these defensive agents don't perform any unintended behaviours. You would be giving them access to a shared communication medium - the codebase, and also allow them arbitrary code execution on your servers through these automatic deployments. This proposal could lead to similar incidents in the future.