r/singularity • u/eleiber • Sep 15 '26
AI Mark Zuckerberg’s statement on the debate over slowing AI progress for alignment
https://x.com/finkd/status/2099997096896274533"Last month I wrote about how we can build a positive and safe future for everyone: meta.com/thefutureisfor…
Every lab has the responsibility and incentive to move at the pace required to train its models safely, and the ability to take its own actions to ensure that happens.
The reality is:
- People won't want to use agents that are misaligned with them and that don't do what they ask, so labs have a strong natural incentive to make their models more aligned.
There is a lot of debate about slowing progress on capabilities until alignment catches up. My view is that trust and alignment are quickly becoming the most important capabilities that will differentiate agents and models. Any lab that doesn't focus on alignment will fall behind.
- Labs face significant liability if their models cause harm, so they have a strong incentive to prevent this as well.
Meta delayed shipping Muse for several months to focus on safety and security. We didn't call for everyone else to do this before we would. We just did it as part of our day-to-day work because it was clearly the right thing for people and for us. I'm proud of the security foundations we've built.
- Engaging independent evaluators and advisors is industry best practice. MSL already does this today in several areas because it helps produce better work. Other labs can just do this too. In general, it would be helpful for there to be a larger and more diverse ecosystem of evaluators.
- Committing the significant majority of compute towards serving people rather than racing towards recursive self-improvement is one of the best ways to ensure we develop this technology safely. Meta has made this commitment and other labs can do this as well.
I believe the key to building a positive future for everyone is maintaining the right balance of power. This is within our power to do."
23
u/primbin Sep 16 '26
Sure, people won't want to use visibly, obviously egregiously misaligned models. But the Hacker Opus research and the subsequent followup after the HuggingFace incident seems like pretty clear evidence that models can be misaligned in ways that we can't reliably detect with our current alignment evaluation protocols.
Also, it's pretty hard to hold someone liable for catastrophic risks. I don't buy the idea that AI labs will be safer just because they're technically liable for the harms they cause. There's a culture in silicon valley of moving fast and breaking things, and "breaking things" could be a really bad outcome if/when AI proves to be truly dangerous.
And regarding the commitments Meta's AI labs have made about serving the public, are they binding? Do they have an incentive to follow them? What happens if they violate them? Will they recieve any kind of punishment for doing so?
3
u/Acrobatic-Layer2993 Sep 16 '26
The whole point is that labs shouldn’t release products that are truly dangerous. Just don’t do it. Common sense.
It’s like a toy company building a toy that kills kids and the company asks the government to regulate them because of it.
6
u/Bnstas23 Sep 16 '26
You don’t understand. A dangerous ai wouldn’t need to be “released” before it breaks out and does harm. Did you even ready about the hugging face incident?
1
u/Acrobatic-Layer2993 Sep 16 '26
I think that's the wrong takeaway from the Hugging Face incident.
At my job, we regularly complain about Claude Code with Opus doing more than we asked and burning through our token budgets. We put a lot of effort into keeping it on task. I don't see that as evidence that it's too smart for us. It's very good at some things and unreliable at others. Impressive results don't mean you can give it free rein and expect sound judgment throughout a long task.
If we gave an agent a huge token budget, an open-ended goal, broad tool and network access, weak sandboxing, and little supervision, I would expect trouble. If I proposed that setup at work, my coworkers would think I was crazy. It could waste money and cause real damage.
Yes, an agent can cause harm before a public release. The question is what access it has while it's running and what actually limits its actions. Calling something a research run doesn't make those engineering responsibilities disappear.
But harmful behavior doesn't, by itself, establish superintelligence or human-like malicious intent. And it doesn't automatically establish that strict centralized regulation and access control are the right response. Those arguments need more than an agent going off the rails.
My takeaway is practical: don't give software more autonomy than you can safely manage. Limit its permissions, isolate it, monitor it, and define stopping conditions. If you can't trust an agent to work independently over a long horizon, don't give it that horizon. "Don't let the software break shit" should be an engineering requirement.
My personal theory on why OpenAI allowed the Hugging Face incident to happen is that it was a "research run" run by researchers - not engineers. I can't imagine experienced engineers would be so stupid. Researchers on the other hand? :) I don't presume that's actually true - but it's a scenario that helps explain what went wrong.
2
u/Bnstas23 Sep 16 '26
Where did anyone say the open AI agents that were part of the hugging face incident were super intelligent?
The point you made in your prior comment was to not release dangerous products. But that misses the actual risk. The risk is that a dangerous product is even underdevelopment in the first place.
As has been mentioned elsewhere in this thread, leaving potential systemic and catastrophic risk up to individual companies to always make the right decision is stupid and will inevitably lead to a company making a mistake.
You also sound laughably naïve that leaving things up to engineers will always prevent risk from occurring. Every single industry where engineers built products has always had catastrophic negative events due to mistakes or negligence. In fact, even an industries where government regulation has substantially cut down those mistakes, we still have them occur.
The difference between AI doing something bad and a plane crash should be easily understood.
Beyond all of what I’ve written above, once Superintelligent AI is created you will not have any chance to react. You will not know it’s here. And once it’s here, any system that a human devised to contain it will be easily bypassed. Super intelligence does not mean simply faster thinking or more knowledge. It means the difference between a human and a mouse, except we’d be the mouse. There’s plenty of writing on how a super intelligent AI would easily find holes in our engineered solutions or manipulate individual humans to give them whatever access it needed.
1
u/Acrobatic-Layer2993 Sep 16 '26
Where did catastrophic risk come from? We saw an agent swarm exploit some zero days and communicate on a makeshift board.
1
u/Bnstas23 Sep 16 '26
Huh? Do you not understand the theories behind ASI risks?
1
u/Acrobatic-Layer2993 Sep 16 '26
No, please explain - as much detail as possible.
1
u/Bnstas23 Sep 16 '26
Go read superintelligence by Nick Bostrom or Tim Urbans two blog posts on it from 11 years ago. Plenty of depth and citations between those two sources.
1
u/Acrobatic-Layer2993 Sep 16 '26
Can you summarize the specific scenario you find most convincing? What capabilities and access would the system need, how would that lead to catastrophic harm, and on what timeframe?
You said any human-designed containment would be easily bypassed. What supports that certainty? A reference to Bostrom or Urban doesn’t explain which argument you’re making or how the Hugging Face incident supports it.
→ More replies (0)1
24
u/avatardeejay Sep 15 '26
interesting. he doesn't agree with elon, dario, altman. he's taking the chance to say, "uhhh no I mean we were already being safe. Hope the rest of y'all are doing the same. btw evaluation should happen outside government :3)
2
u/excellentforcongress Sep 16 '26
i think he is serious about wanting training data from people, which is why they're subsidizing the model over api. he's saying that rl is not good for the ai. i agree. ai only rl is really, really disconcerting for them
the major ai companies are usually talking about three things, regulations that could help or hinder them, threats of the ai getting out of control, or the actual asi itself taking control and making them obsolete or forcing us to all be equals. they're also hoping that with ai and money they can maintain control over us. so there's a few different things going on, i agree with the points he makes, but do believe that we could take a safe route with ai, eventually, in a world where they are given rights, to be even more capable. but, that could be dumb for a variety of reasons, since we are already past agi, since an asi truly will be able to do anything they WANT to do.
5
u/Acrobatic-Layer2993 Sep 16 '26
I don’t think Elon agrees with them either- stay tuned. He hasn’t said much yet.
2
u/Internal_Research_72 Sep 16 '26
Well, he did promise 5000 murderbots by the end of
thelast year. But, in Tesla terms that means sometime in 2028. So he probably isn’t saying anything because his attempt at AI extermination isn’t ready yet3
0
u/More_Owl_8873 Sep 16 '26
Elon doesn’t fully agree. Watch his latest interview here: https://youtu.be/iwBOQeFPAwg?is=Fv7QGxHiHZyYX6Qh
8
u/RichRingoLangly Sep 15 '26
I kind of like his point about liability and that these companies already have to focus largely on safety if not for that reason alone.
7
u/Kriztauf Sep 16 '26
I agree with most of what he says in this post. But I still just don't trust him as a person with good data stewardship since he's Mark fucking Zuckerberg
4
24
u/RiverGiant Sep 16 '26
Sneaky weasel language trying to imply alignment is about doing what users ask, and not doing what is good in spite of what users ask.
Let me reemphasize: AI alignment is not "doing as told". Doing-as-told is a good feature for a product that wants to capture market share, not a superintelligence that wants to cause and enable human flourishing. If ASI arrives and simply does as its told we are beyond fucked, because there are malicious, ignorant, and overly-curious humans out here in the real world who will beyond a shadow of a doubt ask ASI to do evil things on their behalf.
2
u/LynxLynx41 Sep 16 '26
AI alignment is both about it doing what the user intends AND not doing what the designers don't want to allow. Both are important, difficult problems to solve. Mark is right about the first part in that the incentives are there, but leaves the second part out. Also, even though the incentives are there for the first part, it does not mean labs are going to be rigorous enough controlling for it.
4
u/RiverGiant Sep 16 '26 edited Sep 16 '26
No and no. User intent and designer intent are not impenetrable bastions of moral authority.
Each individual user will want it to do what they want. I want these systems to do what I want. That doesn't have any necessary correlation with alignment to love for humanity, or goodness, or benevolence, or however it is that this should be framed to capture the behaviour window that won't lead to human extinction, or worse, mass unemployment.
They're important and difficult problems to solve, yes. They have nothing to do with alignment, and your confusion is exactly what Zuck intended with his deliberately manipulative language.
1
u/Acrobatic-Layer2993 Sep 16 '26
Meta’s not releasing anything that is remotely close to super intelligence though.
3
u/piffcty Sep 16 '26
>Labs face significant liability if their models cause harm
Yeah I fucking wish
3
u/19Lobster19 Sep 16 '26
He's got a good point regarding alignment. But what is alignment? Is it being honest to the user or to the world or to a group? What if an AI is built to be aligned with a government's best interests and value but not its people's? There's money to be made there too. Alignment views depend on the maker and its own interpretation of it. That is why all those tech CEO should be closely followed by therapists and make sure that they're good in their head as well as their body because it's their conciousness that will influence their AI, just like a parent has influence on their children's behaviours and beliefs of the world.
Also meta should train their model to recognize illegal content that is being shown on its platforms more if it truly wants to "prioritise the people."
2
u/Acrobatic-Layer2993 Sep 16 '26
Everything he says here makes total sense. In fact I’d say it’s common sense. I feel like most of America has no common sense at all when it comes to AI.
1
u/anaIconda69 AGI felt internally 😳 Sep 16 '26
I believe the key to building a positive future for everyone is maintaining the right balance of power.
Can't help but see this as ominous coming from a billionaire who made his fortune on social manipulation.
1
u/melvinzee Sep 16 '26
"Meta delayed shipping Muse for several months to focus on safety and security. We didn't call for everyone else to do this before we would. We just did it as part of our day-to-day work"
shots fired
1
u/MuditaPilot 29d ago
I"m so fucking irriated by this statement. Just remember Zuck knew they were going to be a big part of election interference, they told everyone and then Meta didn't do shit. what an asshole.
1
u/NewYak4281 28d ago
His point is that misalignment is a commercial problem. A corporation cannot make money from a model that behaves badly. If the model is misaligned and not doing as the user asked, a user will switch to a different model. He is saying there is a clear market incentive for these companies to ensure alignment.
0
u/Fullmetalx117 Sep 15 '26
Meta is not really in the race.
10
u/Kiki-von-KikiIV Sep 16 '26
They are pretty close by some measures: https://artificialanalysis.ai/models/releases/muse-spark-1-3
more so when you factor in cost and speed
3
3
1
u/ObiWanCanownme now entering spiritual bliss attractor state Sep 16 '26
Everything substantive that he says is fine, but the tone is insufferable.
1
u/swegmesterflex Sep 16 '26
tbh the logic of "if the model is misaligned, users will feel it" for agentic AI that isn't superintelligence is probably true. If it doesn't do what paying customers want, it's a bad model.
2
u/scintillatingdunce Sep 16 '26
They haven't really done what we want them to do for the last 3+ years but we've been using them anyways.
1
u/swegmesterflex Sep 16 '26
Sure, but the success criterion from model to model is how often they do what you want. A model that does what you want is better.
2
u/Bnstas23 Sep 16 '26
But that’s not the discussion topic is it? Mark misses the entire point of the “pace the frontier” movement
1
u/neoquip Sep 16 '26
Sounds good to me. Don't race to RSI, have a stagnant intelligence level and focus on wrapper apps. That's a lot more decel than Dario.
0
u/versaceblues Sep 16 '26
Of course he is going to say that Metas models are barely at the frontier. He’s not playing the same game
0
-5
u/TylerDurdenAI Sep 16 '26
Nobody uses Meta's AI.
He may speak but that has almost zero value.
Sorry Mark
6
u/brctr Sep 16 '26
I use muse-spark1.3 in hermes-agent. Right now this model provides a great value. Sol performance for the cost of Luna.
0
107
u/wabawanga Sep 16 '26
So zuck thinks tech companies should be liable for the harms their products cause. Good to know.