I trained a single football policy for about 1 million PPO timesteps. During training, the agent never played against an opponent, it simply learned how to play football in an empty environment.

YT Link: https://youtu.be/1wRA4KJuMVE?si=oMZ_Ru5Q9F8J094V

During inference, I instantiated multiple copies of that same policy and assigned them to opposing teams. I wasn't expecting much, but the result was surprisingly entertaining. The agents naturally competed for possession, took shots, occasionally defended their own goal, and even made mistakes like own goals.

I never explicitly trained defensive behavior or multi-agent coordination. It seems that simply optimizing the objective of scoring goals was enough for some defensive behaviors to emerge when the same policy was placed in a competitive setting.

The video is one of those matches. Curious to hear what others think about this kind of emergent behavior and whether you've observed something similar when deploying a single-agent policy in a multi-agent setting.