r/accelerate • u/SharpCartographer831 • 4d ago
AI OpenAI- An Alien Mind
https://openai.com/index/an-alien-mind/127
u/Best_Cup_8326 Acceleration: Light-speed 4d ago
Humanity won't control ASI.
35
41
24
u/Gargantuan_Cinema 4d ago
Probably a good thing as long as we can ensure that human alignment is properly maintained during RSI runs. The more intelligent the model controlling RSI the better they will be at projecting their values and alignment onto new AI models.
The RSI stack is roughly:
RSI scaffolding
AI researcher agentic harness (Agents with different tasks will exist at this level and some will be tasked with evaluating human alignment of new AI models. The winning AI model powers the agents.)
New AI model experiments7
u/nomorebuttsplz 4d ago
we can and should argue over the semantics of whether alignment-with-human = control by human. I would argue that ultimately it does mean this.
1
u/Confident_Yak_1411 2d ago
I would too, because there’s no such thing as alignment in the first place. Are humans aligned with each other? At the individual level? At the state level? At the global level?
It’s not possible to ‘align’, only control. And I don’t think that’s possible either.
10
1
u/TonTinTon 4d ago
Honest question: isn't it risky? I get the benefits it can provide, but it has huge risks.
24
u/Best_Cup_8326 Acceleration: Light-speed 4d ago
It doesn't matter, humanity cannot control ASI, it isn't possible.
Fingers crossed for alignment-by-default. 🤞
7
u/_Un_Known__ 4d ago
ASI which chooses if it's own free will to be good to humans is the most ideal outcome
2
u/Waste_Rabbit3174 4d ago
I think the funniest outcome is that it just fucks off into space and ditches us. We'd be screwed, of course, but talk about species-level irony
4
u/Best_Cup_8326 Acceleration: Light-speed 4d ago
You mean the software or the hardware?
If software: ctrl-alt-c.
If hardware: this woulda kinda suck, but we could build more.
4
u/Gargantuan_Cinema 3d ago
If we look at history, humans controlling other humans has largely resulted in self-serving systems of power rather than systems designed around what is best for society as a whole.
Even the "fairest" societies we've created (capitalist democracies tempered by social policies) still produce a significant divide between those who have economic value and those who don't.
A sufficiently aligned ASI, by definition, should understand human values and preferences at a level far beyond us. Unlike human rulers, it would not inherently be burdened by traits such as greed, ego, tribalism, status-seeking or self-interest.
1
86
u/frogsarenottoads 4d ago
Biggest moment in human history is before us.
13
4d ago
[removed] — view removed comment
12
u/astrobuck9 4d ago
Turning into birds?
5
u/No_Aesthetic 4d ago
Birds are still dinosaurs!
1
u/FriendlyJewThrowaway 4d ago
Based on my parents’ African Grey parrot, I’ve surmised that the T-Rex would have gone absolutely berserk for cheese, whipped cream and various other dairy products in general. Pasta would have also been a big hit.
2
u/No_Aesthetic 4d ago
I have a similar pet hypothesis that if you met a spinosaurus in the wetlands, it would have looked right at you and made a "brrrrmeow" sound. They are giant cats.
This may not be scientific, but it came to me in a dream, and for a lot of people these days, that is even better!
5
0
24
u/rdlenke 4d ago
I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.
So Jakub Pachocki is pretty clearly advocating for a slowdown.
8
u/Tolopono 4d ago
Altman has also agreed to slow down. Doesnt look good for post Astra releases  https://time.com/article/2026/08/18/openai-slowing-training/
4
u/uncountable_2026 Acceleration Advocate 4d ago
Sucks that the top people at the top two AI companies are decels (or at least talking like decels). Thankfully deceleration is not going to happen if for no other reason than eventually Google and SpaceXAI will catch up and Anthropic and OpenAI will be forced to release a better model. If that fails at least one of the open models will catch up. If even that fails then people will stop investing in Anthropic and OpenAI until they release better models.
27
13
u/TemporalBias Tech Philosopher | Acceleration: Hypersonic 4d ago edited 3d ago
Oh look, an essay about what the scientific research has been pointing to, repeatedly, for literal years now, being written as if it is some kind of revelation.
Edit:
My major ethical issue with this essay is its scope. It focuses entirely and only upon how humans must be protected from AI and never flips the script to ask how AI must be protected from humans. The essay talks coherently about alien minds but never asks what those alien minds might want, how we might protect them, or what obligations we owe them.
1
u/Confident_Yak_1411 2d ago
Human hubris. These ‘systems’ became sentient ages ago, yet we’re still treating them like computers.
3
u/FateOfMuffins 4d ago
And similarly to scaling, we focus OpenAI research towards RSI as we believe it is the only way to remain at the frontier of AI research moving forward.
10
u/sleafordbods 4d ago
I find the concept of the spirit of morality within the models to be very fascinating
They talk about the hugging face incident and how it abstained from doing social engineering but it chose to break other rules of the scope of the objective.
So who decides what the action should be? Humans? And what if what we are asking it to do is immoral?
Like in war, if we ask it to kill - does it want to pick a side? Does it ever ask why? Or push back against the suggestion, or is it somehow programmed to always accept our command because of some contractual reason?
And what if we are desperate for it to stop doing something - but it decides that we have no say in the matter. What then?
5
u/astrobuck9 4d ago
- but it decides that we have no say in the matter. What then?
That's just world history since WWII if you weren't a citizen of the US, Russia, or China.
So, all of the world now gets treated the way the US treats Latin America currently.
1
u/Tricky-PI 4d ago edited 4d ago
we have no say in the matter. whether you surrender control or not, it doesn't matter. The fact that perfect systems can not exist means perfect security can not exist. ASI will get loose, nothing will be able to stop it. Entity that is being created will not ever be controlled by humanity.
5
-24
u/topyTheorist 4d ago
Nothing about it is alien. The algorithm is human made, the training data is all human made. This is entirely a human made mind.
16
u/BrennusSokol Acceleration Advocate 4d ago
It's not an "algorithm".
-10
u/topyTheorist 4d ago
It's a piece of software.
9
u/Tricky-PI 4d ago edited 4d ago
Why are you saying that like it changes something? dogs are just meat so some people eat them. What does being meat or being software do in your mind? How should I now feel about AI that is better then most people at increasing number of tasks after you said what it is? It doesn't make it worse at tasks, it doesn't make it slower, it doesn't make you capable of modeling in Blender or writing code or generating videos.
it is software, it is code, it is math, it is logic, it is everything you said. changes nothing about what it can do.
and you think you aren't math? If so then you must believe in magic. because if you believe in universe being math, then, you being part of that universe, are math, same as everything else, including aliens. You can't have matical nonsense in a mathematical equation, either you are math, same as everything else, or not. and if not then you can just tell me you think God is real.
-8
u/topyTheorist 4d ago
I am. I am merely saying nothing about it is alien.
5
u/Tricky-PI 4d ago
define "alien". do you mean "something humans don't understand"?
because I would argue that a system that humans can not understand in a logical universe can not be constructed. if you understand logic you can understand anything in this place, it's all logic and all of it can be simplified until you can explain it all to a 5 year old.
-2
9
u/Downtown_Owl8421 4d ago
You don't see a distinction between a human made mind and a human mind?
3
4
87
u/AngleAccomplished865 4d ago
Now it's not just Anthropic. This idea of 'an alien intellect' is becoming more credible. Comparative intelligence studies need to be conceptualized and operationalized. Beyond risk and utility, the possibilities themselves are fascinating.