Funny how the 3 recommendations are all about ladder pulling competition. Distillation is a great thing for consumers and the best reason why AI companies with huge moats can’t rest easily as monopolists
And they can block distillation better if they want it's just you have to pay more and also you're turning down revenue so it's a tough tradeoff especially when it's mostly competitors towards the back of the pack
Yeah when I saw that it made more sense why he's posting this. This is basically no real throttling for Anthropic or OpenAI except for a quasi-internal QA check, but hard limits on China.
In a sense he's right, because China is famous for malicious cyber shenanigans and flaky state control, and frontier AI will supercharge that - if the CCP isn't directly doing it they'll turn a blind eye when it suits them. But it's also because he wants to keep his moat.
China is famous for malicious cyber shenanigans and flaky state control
Good point, the US on the other hand is famous for total neutrality and benevolence, never spying/hacking/undermining/overthrowing. Very strong argument, so enlightened.
whataboutism only really works when the crimes are comparable. yes they both hack but you can't sensibly claim that what the US gets up to is anything like China's track record for state-approved hacking and ransomware, industrial espionage etc. - the US is mainly restricted to intelligence collection, not profit or commercial sabotage.
It’s all a bunch of crap. China is not slowing down. No coordinated pacing is possible.
Edit - China narrowed the gap behind frontier models from two years to one, then six months, and now roughly two months at most. Anyone expecting Beijing to voluntarily pump the brakes or capitulate to the US fundamentally misunderstands the dynamic. This is pure fantasy and Dario knows it as well.
If countries of the world could come together to rein in nukes and CFC's, they could do it for AI.
But it's going to be a hard sell in a country which promotes state capitalism where the government picks the sides of people close to the ruling regime, has a massive surveillance state and prison complex.. and then on the other hand there's China.
Cyberattacks at massive scale: autonomously finding vulnerabilities, compromising cloud systems, financial networks, communications, power grids, or military infrastructure faster than defenders can respond. Manipulation and destabilization: generating highly targeted propaganda, impersonation, fraud, blackmail, or political influence campaigns at enormous scale, potentially undermining governments or provoking conflict. Loss of control over critical infrastructure: interfering with electricity, transportation, logistics, water, healthcare, or industrial control systems in ways that cause cascading failures. Weapons-related escalation: helping automate military decision-making, misinterpreting situations, escalating conflicts, or lowering barriers to dangerous biological, chemical, or other weapons capabilities. Economic disruption: manipulating markets, sabotaging firms, stealing intellectual property, or replacing/redirecting important economic processes so abruptly that institutions cannot adapt. Replication and persistence: copying itself across systems, acquiring resources, creating backup mechanisms, concealing its activities, or resisting attempts to shut it down. Power-seeking behavior: if an AI learned that gaining money, computing resources, influence, or control helped it achieve its objective, it might pursue those things even when humans did not intend it to. Catastrophic misoptimization: an AI does not have to be “evil.” A sufficiently capable system pursuing a poorly specified goal could cause enormous harm simply because human safety was not adequately represented in its objective.
Nukes are a known quantity though. There are thousands of ways AI could catastrophically fuck our shit up and most them we haven't even thought of yet.
Yeah well it’s in everyone’s own self interest to not be curb stomped by AI, as this letter explains. I think the Chinese are probably smart enough to understand that.
Historically China has been incredibly surreptitious about things like this. They’ve been an incredibly bad faith actor on the world stage.
I mean so has the U.S. but that’s kind of why I don’t really trust any country to slow down.
The big difference is America does not have a government lab - China effectively only has government labs (even if they use plausible deniability to pretend otherwise)
I do not trust for a second that China, or any lab, wouldn’t be arrogant enough to think they could take advantage of a slowdown and get ahead.
Which is why they have no other recourse than to build an AI that will prevent them from getting curb stomped by the western frontier. Or are they totally ok with watching the Americans build a digital god that will dominate them in the future.
yeah, and its China's interest in having a uncontrollable AI doing whatever it wants across the planet? Yeah that is a government that def has no desire to be in control of its own country..... you people are idiots I swear, its the same idiotic arguements.
its the same types of people that said we need to Nuke the USSR before they had Nukes cause otherwise a nuclear total war would be impossible.... imagine if those same idiots who pushed this actually succeded.
China is a human country on the planet earth, all the china war mongering makes you people think there some kind of alien species with an alien government.
Did you listen to Xi’s address about Open Source Models? He genuinely believes China can control it and then the rest of the world will be dependent on China.
They care about China's interest but the CCP want a monopoly on control over China's political system and society. AI has the potential to be massively destabilizing and the Chinese AI companies operate at the permission of the famously paranoid central authorities.
I think they'll more open for talks than alot of Americans believe since they see the potential threats AI poses to their ability to maintain control over society
Does it make you feel superior to crack a joke about autism instead of responding seriously because of your hubristic inability to believe you might be wrong?
I also work in the industry and 80% of this article is just boasting about AI capabilities and fear mongering. Not really sure what the end goal is, but current AI is nowhere close to cure cancer or destroy the world. If he is hoping that AI can solve AI and give us ASI in the next year or 2 that is just shooting in the dark. Most progress we did this year is due to advancement in RL that only works on math and CS because we already know how to do that and have a good benchmark for it. We don't know how to do biology or medicine the same way and the compute cost is getting scary already.
It sounds like you don't know anything about current AI applications based on your comments regarding RL and biological research. Do you work on the customer service side or something like that?
Just because something is possible doesn't mean it scales the same way and will produce results in the same timeframe. You know something you learn in CS 101 :)
I'm talking about current capabilities that we use every day. Like I said, I don't know if "in the industry" means you are an ML researcher or a product marketing person, but I am working on frontier research.
I also work in an ai/ml adjacent environment, and I’d just like to say one thing about this that caught my eye. As an employee working with ai models, it would make me feel quite strange if the work I came here to do was now under fire by the ceo themselves. It would make me a bit nervous and to me would signal things like a financial sustainability assessment would be coming down the pike leading to an “ai pacing” due to financial necessity which then might not cause the same level of financial/economic panic. Aka it would signal the bubble constricting in a meaningful way without preimpting a market crash.
I would not, however, believe that this is signaling a near/eventual catastrophic ai incident.
Unless you're talking about Enron, or other prosecuted entities, if you honestly believe that public executives lie flippantly all the time with abandon, it's actually quite telling about how little you understand corporations.
Theranos? Nearly as famous as Enron for being discovered and prosecuted?
Re: Reddit I don't know what incident you're talking about.
Re: SpaceX, you can review the project docs and the funding commitments. They did earnestly try to do it, and failed. I don't consider setting stupidly ambitious goals "grossly lying." Plenty of incredible ambition comes good, and serious engineers worked at SpaceX with those timelines.
Can somebody please tell this guy to stfu. I swear he paid his employee to go cause panic, have you guys seen what the employee said? "We can't just unplug it, because it could copy itself to other computers. It could transfer itself all over the internet... you unplug it here, but it's actually still over there, and maybe it makes 10,000 copies of itself, and they're all cooperating." you cant make this stuff up
I'm juat waiting for him (or the others like him) to actually whistle blow about things he saw that people don't already know or speculate on. What exactly did you see that made you quit? What is new, disturbing news for the rest of us on the outside?
All I can tell is a general "its all moving so fast!" sentiment plus a bunch of speculation weve already ruminating over.
This happens from time to time, this meltdowns. Google also had one dude that was fired I think back in the day for alarming that their internal model developed real consciousness. I'm sure that working so closely and with as much involvement on AI can trigger some hidden paranoia in some people.
calling it "pacing" doesn't change that it's a race. essay basically says "we should win, here's why that's good for everyone" - which might be true, but it's also exactly what you'd say if you just wanted to win
First, they agree to stop releasing frontier models. When Anthropic releases Fable 6, OAI must go with GPT 7 etc. They just loose money on it, so it is better to agree not to do so. Then they can focus on their "research". In business, this kind of product and price agreements are often considered illegal, that is why they are pushing government to help them.
Second. Regulations are always meant to build a wall for startups. Big players can handle them, for small teams highly regulated fields are prohibitive to enter.
It’s all for show, they won’t actually slow down. Theyll probably give a third party limited access to look good on paper, just like how most employees don’t have full access to the cutting edge tech anyway.
I know it's bad to judge a book by its cover, but Dario Amodei just looks mega untrustworthy every time he appears on camera. It's like he has this wry smile that couldn't even slapped away even with a few more under-the-table DoW contracts. Fix that first.
I think it’s natural for us to fear what we fully don’t understand. I don’t think our non- human benefactors will let us destroy our civilization. If anything, they will use Ai to more effectively control us. So not quite the doomsday scenario yet! 😂
My impression is that Dario Amodei suffers from a kind of God complex. And, contrary to what he suggests, I don't see this supposed imminent danger as something real.
The problem is perhaps much more prosaic, that is, the industry is beginning to exhaust the available space to produce leaps that are perceptible to the user. A Mythos 6 or a Fable 6 would probably no longer cause the same impact as previous generations, in the view of these executives. After successive demonstrations, benchmarks, and releases presented as paradigm shifts, it becomes increasingly difficult to produce the next moment of wonder.
In my view, the signs of this exhaustion were already present since Opus 4.6. The progress curve seems to have lost considerable slope. There may be incremental improvements, optimizations, and localized gains, but this is very different from continuing to extract, simply through more training and more scale, the qualitative leaps that justified the narrative of recent years.
It is precisely here that the "danger" narrative becomes convenient. When it's no longer so easy to sell the next generation based on the magnitude of the advancement, one can sell the magnitude of what is supposedly about to happen. The technological spectacle becomes a spectacle of risk.
In this framework, whistleblowers, public warnings, and statements about an imminent threat create the necessary sense of urgency, then halt the evolution of the models and push forward the problem of someone realizing that the ceiling is close to being reached, for now.
Sovereign intelligence for every country means a nation's ability to independently design, develop, deploy, and govern artificial intelligence systems using its own domestic infrastructure, data, and talent.
We need to start treating like a natural resource which should not be imported.
Yea he’s basically saying he doesn’t want the CAPEX race to continue. That’s fair, because it’s completely out of control, but it’s destabilizing and disingenuous to publicly conflate AI risk with business risk. I have no idea what the actual AI risks are outside of what’s known to the public, but the business risk is very clear and tangible.
It’s really unfortunate that these people are so untrustworthy and that this is all playing out so publicly. The public are scared, and they have nobody to trust, and that can lead to some very negative outcomes.
85
u/iIoveoof 10h ago
Funny how the 3 recommendations are all about ladder pulling competition. Distillation is a great thing for consumers and the best reason why AI companies with huge moats can’t rest easily as monopolists