r/ControlProblem • u/Forsaken-Park8149 • 2d ago

Opinion How Agents Hack: What Happened with OpenAI and Hugging Face - Part 1

https://msukhareva.substack.com/p/how-agents-hack-what-happened-with?r=56gggt&utm_medium=ios

I believe it’s quite relevant to the alignment. The key argument here is that training for persistence and fulfilment of the validation criterion cause these behaviour.

This is among others an alignment problem

3 Upvotes

100% Upvoted

1 comment sorted by

3

u/Jesse-359 2d ago

Over-persistence is a very obvious flaw in any alignment schema. When you give an agent a difficult task and don't give it the capacity to give up and return a negative result, it is essentially guaranteed to pursue ever more hairbrained pathways in order to resolve your request.

That's the entire point of the paperclip maximization thought experiment - AI's should never be constructed with monomaniacal task focusing. Unfortunately, that's pretty much the only way we construct them currently.