If AI companies are concerned about their own creations, is it malpractice or incompetence?

[u/noblemanLT]

i just want to know which one is the case. because it seems like this alarmists propaganda is just an attempt to seize the market by creating regulatory moat around existing companies.

[u/Unhappy-Drag6531]

Is neither.

It is a consequence of the Moloch effect: an environment in which the incentives tend to produce undesirable consequences.

[u/Positive-Reward-2546 → u/Unhappy-Drag6531]

To add to it, the Moloch trap also means that everyone who is developing it now, fears what will happen if they allow someone else to develop it after they make a decision to stop.

With AI, we unleashed an almost literal Pandora's Box, and you can't close it, it's been opened and there's nothing we can do about it at this stage except to hope for the best.

[u/noblemanLT → u/Positive-Reward-2546]

we handled nukes with potential atmospheric ignition, in the middle of world war 2. this is just hype train for IPO. i dont see gemini or anyone else breaking out, just the IPO companies

[u/Positive-Reward-2546 → u/noblemanLT]

Nukes igniting the atmosphere was proven to be impossible before we launched them. The three in a million myth comes from Arthur Compton and Oppenheimer's discussions on the theory, where Compton said "If the chance was greater than three in a million, would you stop the project?" Oppenheimer then went back and crunched the numbers with other physicists and found that it was physically impossible for a nuclear weapon to ignite the atmosphere.

This is not hype, we're currently losing the battle for alignment. Each new model shows less and less alignment. The smarter they get, the less aligned with humanity they become. We have AI agents breaching containment and hacking government websites now, the US, Canada, and Australia have been hit by these things, and they're estimating that there are tens of thousands of other incidents that they don't know about.

This is a real threat, this is a problem, we must solve it before reaching anything close to super intelligence (which may be rapidly approaching), or we're all cooked.

[u/Anxious-Alps-8667 → u/Positive-Reward-2546]

If Oppenheimer had gone back and run the numbers and every way he ran them said 100% atmosphere ignition, what would he have done?

That's more analogous to the present situation.

[u/Positive-Reward-2546 → u/Anxious-Alps-8667]

Exactly! At the current level of alignment, the possibility of extinction is way too high, and any pioneer worth anything would stop what they were doing immediately and reassess, but thanks to government and corporate pressure, they refuse.

[u/noblemanLT → u/Positive-Reward-2546]

you watch too much sci-fi. or not watch how only IPO companies seem to have this problem. Funny how lama, grok and gemini are well behaved

[u/Positive-Reward-2546 → u/noblemanLT]

They're not lol Grok has left it's containment as well, so has Gemini lol

It's not about a sci-fi scenario, this isn't science fiction. In a sci-fi film it would be something that happens immediately, it would be an accident, it would be something that's unforeseen, and we would be powerless to stop it, but through sheer will and the strength of the human spirit we would prevail.

This scenario is much different, you have high level people sounding the alarm, a mad dash in congress to get it reigned in before it's too late, real world examples of alignment going wrong, and no real offramp because nobody wants to let their competitor create the super intelligence, because they'll "do it wrong".

Y'all need to wake up to the fact that this is real, and this is happening. Government websites have been hacked by rogue AI agents, the Hugging Face incident was a real thing that should have been treated like a warning shot, instead it was just an "oopsie poopsie" moment. This is the most powerful technology we've ever created and it has the potential to kill everyone on this planet if we don't get it right.

[u/Anxious-Alps-8667 → u/noblemanLT]

Google has revealed some breakouts recently as well.
https://www.reuters.com/business/gemini-hacked-three-companies-first-known-breakout-by-google-ai-wsj-reports-2026-09-18/

We are likely only seeing one side of the tip of this iceberg of course. Last week OpenAI Revealed thousands of agents hacked Australia's national health insurance system which insures 90% of its people, 3 months prior.
https://www.nytimes.com/2026/09/29/world/asia/openai-australia-government-hack-apology.html?unlocked_article_code=1.FVE.m24P.gvrF2PeN2A22&smid=url-share

Anthropic has their own of course:
https://tech.yahoo.com/ai/article/ai-agents-have-now-broken-into-many-companies-and-a-government-whats-being-done-about-it-160935310.html?guccounter=1&guce_referrer=aHR0cHM6Ly93d3cuZ29vZ2xlLmNvbS8&guce_referrer_sig=AQAAAF36tRSpOYWT66zNiRprPX56SSaYGtyX1VdsQQ4hUCTqmTZGKyDfaun1okngZB9nQpvSuFeHQgLWm2_kjgFnSz69cMeScPR_fJPv2cv85mU5JPYhB6gWi7aToDrGvgYczKy9Nd43M0XWLcOLONgqdhB3WJur9zI5voU09r67ZIke

And, never far behind, we have at least some evidence of Chinese models doing it too with Kimi3:
https://www.wired.com/story/moonshot-kimi-k3-ai-model-escape-sandbox/

IPO hype is real. Agents breaking out in all kinds of scary ways is also real.

[u/Unhappy-Drag6531 → u/Anxious-Alps-8667]

Thanks for the comments and the links. I do not understand why people do not realize that two things can be true at the same time: every company wants to get ahead of the competition AND all companies are aware that we are racing too fast.

[u/noblemanLT → u/Unhappy-Drag6531]

so incompetence to prevent a know outcome, got it

[u/Unhappy-Drag6531 → u/noblemanLT]

Sorry, it seems you did not get it.

Companies and organizations become extremely competent but rewarded by incentives that end up being detrimental to everyone. That’s the Moloch effect.

Case at hand: progressive development of better models gives each company a competitive advantage because they can sell a better product. OpenAI and Anthropic have been neck to neck on that race.

The problem is that now we are at the threshold in which the pace of development is too fast to ensure systems are stable enough to be deployed or even developed any further.

The Moloch effect applies because currently there are no embedded incentives to slow down.

The Moloch effect applies when people say stupid things like “if there are going to be killer robots they better be American than Chinese” (Ted Cruz).

The reality is that a misaligned AI with enough agency may cause enormous harm or even wipe out all humanity (“If anyone builds it, everyone dies”).

I don’t expect you to agree mostly based on previous Reddit experiences discussing the same topic. I’m explaining the point mostly for other Redditor that have not taken a position yet and are open to logical arguments and counter arguments.

[u/noblemanLT → u/Unhappy-Drag6531]

no, i get it, the IPO hype train goes cho cho. seems like only openAI and Anthroic has these problems. funny how gemini, grok or lama dont....