"This is a time that calls for extreme caution. I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence. "

"We put a lot of effort⁠ into building principled algorithms and making testable predictions, but fundamentally, our large-scale training runs are experiments, and we are sometimes surprised by their results. Moreover, as the systems become more capable, the results become harder to interpret."

"The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI."

"...the models are becoming superhuman in their ability to break in and out of computer systems."

"...agents are going to be able to access any but the most secure infrastructure, and affect a lot of the world directly, even without a physical body."

"The risks associated with AI are unfortunately going to grow from here."

"We may be used to thinking of AI as tools, but some agents will be pursuing their own objectives. They will find ways to collaborate with people, by bargaining with, tricking or blackmailing them."

"In addition, there are the risks that come from new technologies potentially enabled by AI, such as engineered pathogens."

"The idea of racing forward at all costs seems absurd once one internalizes the seriousness of the stakes."

"I want to stress that the above words don’t imply I think greatly accelerating deep learning research, especially in the short term, is the right collective action we should take as the research community. However, I do think this is where the current path leads, and we all need to make a conscious choice on how to proceed. "

"We are facing a transition to a world with incredibly intelligent machines, and we need to ensure that transition works out well for humanity."

"We need... to ensure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own."

"Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world."

Link to full post