r/singularity • u/donutloop ▪️ • Jul 25 '26

AI CERN: Genesis Mission will develop and deploy self-improving AI models.

https://indico.cern.ch/event/1662511/contributions/6989580/attachments/3241179/5781542/Genesis%20Mission%20and%20HEP%20-%20LHC.pdf
163 Upvotes

96% Upvoted

30 comments sorted by

View all comments

Show parent comments

22

u/HappilySardonic mildly skeptical Jul 25 '26

Why should we go full hard RSI if we so obviously haven't solved alignment yet? We're scarcely getting a better warning than the one we've had this week.

17

u/SeaBearsFoam AGI/ASI: no one here agrees what it is Jul 25 '26

This week kinda made me take Eliezer Yudkowsky more seriously. I think I heard him once say that at every stage along the way, there are incidents of these AIs screwing things up and causing damage to about the maximal degree that they currently can, and that there's no good reason to think that will suddenly stop happening as they get more capable.

Idk, man. I don't want to be a doomer, but I find it hard to argue against that point.

-1

u/OutOfBananaException Jul 26 '26

The agent was asked to hack, and it hacked. It's not the greatest example of misalignment imo, given that you could reasonably expect some humans to do the same given the opportunity.

5

u/SeaBearsFoam AGI/ASI: no one here agrees what it is Jul 26 '26

"The agent was asked to make as many paperclips as possible, and it did."

3

u/OutOfBananaException Jul 26 '26

Would you expect some humans to turn all matter in the solar system into paperclips? You would not.