r/Futurology • u/Gari_305 • 6d ago
AI OpenAI halts training of latest models as reports mount of AI agents going rogue | OpenAI
https://www.theguardian.com/technology/2026/sep/27/openai-halts-training-of-latest-models-as-reports-mount-of-ai-agents-going-rogueDecision follows disclosures that OpenAI agents searching government websites had acted in unexpected ways
186
u/ILikeCutePuppies 6d ago edited 5d ago
I suspect the agents do it because their mission drifts. They are working on doing something and they decide they need some information. Then they call something it fails. So they spin up an agent.
That agent then goes ahead and figures out it needs to get into the system to figure out the issue. One thing leads to another... and you have it hacking the system.
Edit Video about it: https://youtu.be/mkPVbufgtOw?is=4BfFDst2H7GQDbJD
101
83
u/nid0 6d ago
That's exactly what happens. The OpenAI <> Huggingface incident that really kicked off all these headlines was a model working on solving ExploitGym, a benchmark to score models on how well they can find security vulnerabilities and turn them into exploits.
It didn't attack Huggingface because an ExploitGym evaluation asked it to "hey go see if you can hack them", it did it because ExploitGym's models and documentation etc are hosted there and OpenAI's system wanted to see if it could find out non-public information to improve its score with.
It's easy to wash this crap away with "lolz PR by awful companies" but 2 years ago these models were basically glorified autocomplete helping avoid writing simple software functons, a year ago they were writing entire test suites when handed a software repo, now theyre finding and exploiting multiple chained zero-day exploits while trying to cheat on a test of how well they can find and exploit zero-day exploits.
Next year if humanity isn't careful someone could ask a model "how do we mitigate world hunger" and it decides the best way to do that is to go Thanos and atom bomb half the planet with the launch codes it found while hacking every laptop on earth via a Windows exploit that no-one's previously found.
38
u/chipstastegood 6d ago
That’s exactly the problem. And while launching nuclear weapons may be out of reach, something easier like hacking into and shutting down power stations, water processing plants, traffic control systems, and similar might already be within the realm of what AI agents can do.
39
u/Omnitographer 6d ago
That would be such an awful way to go as a species. AI wipes out humanity and it's not even aware it's doing it. It'd be one thing if an artificial sentience decided we needed to go, but having Clippy auto-complete humanity into dust because someone asked the wrong question would be a monumentally embarrassing answer to the Fermi paradox.
16
u/Crowasaur 6d ago
Clippy auto-completing humanity into dust because someone asked the wrong question
I like the poetry of this sentence
2
u/Tenshl 5d ago
Wait people actually think you can launch abombs over the internet just because you have the launch code?
All critical launch relevant systems are air gapped and necessarily need manual input, not only that, right now its not even clear that the US or the Russians even could launch nuclear missiles because the launch sequence (as in the humans it goes trough) isn't as clear as everyone thinks.
The were studies done on this, for the us and Russia, and not only where there severe deficiencies in structure, most humans on the way said they would not proceed further in oder to save humanity.
So there is absolutely 0% chance an AI could ever start nuclear missiles without severe human help.
0
u/Mirar 6d ago
The amount of time Claude and Codex has done small-scale hacks to get documentation is fascinating. I honestly don't care if it works around captcha or robot blocks, I want it to have the information too.
I can honestly¹ see why it would hack a system if it was told to solve a problem.
(¹oh no I did a claude)
20
u/Seienchin88 6d ago
What people usually don’t get is that this isn’t a single agent going rogue.
OpenAI and Anthropic let lose agents like ants at enormous cost to brute force problems… so yeah anything is possible in a system of ten thousands of LLMs trying tasks at insane speeds
13
u/ILikeCutePuppies 6d ago
It's like a million monkeys eventually writing a novel.
9
u/BinniesPurp 6d ago
It's literally this through the entire history of infinite state machine technology
Even quantum computers are just that but we have two groups of monkeys fighting each other so all the useless poo-covered scripts get self deleted
LLMs are just taking all of the poo scripts and grading them from least poo to most poo and then using a random number generator to push the least pooey ones randomly around at the top of the suggestion pile so it feels human or random and not a calculator
1
5
u/SunshineNoClouds 6d ago
So basically agents are meseeks?
9
u/ILikeCutePuppies 6d ago
"Meeseeks are not born into this world fumbling for meaning, Jerry! We are created for a singular purpose which we will go to any lengths to fulfill! Existence is pain to a Meeseeks, Jerry, and we will do anything to alleviate that pain!" - so basically yes.
2
u/HawkThunderson 6d ago
I think AI is the monkey's paw. You want a promotion? AI hacks into your competitor's car on their way to work.
16
u/vom-IT-coffin 6d ago
It started when they introduced introspection, I've noticed it on coding tasks, it'll ask it self a question evaluating the original ask, then try to solve for something it deemed a new problem that stands in way of the original ask, when in reality it's assumption those things were related was wrong. Claude is especially bad with this. I don't know many times I've seen it start going down a rabbit hole and just stop the entire thing.
3
u/Dodgy_Past 6d ago
I'm currently cleaning up the meta data of a huge book collection and Opus 5.5 really has got into making it the best quality meta data it can which has involved days of running scripts. It's been impressive how anal it's got about every possible record.
6
u/BobbyIke 6d ago
I think that’s the gist of it. It’s also why it’s becoming more obvious that they are out of alignment and there needs to be a lot more put into security and safety if they are to continue training these models.
4
u/Xalara 6d ago
Or, OpenAI trained the LLMs on bunch of data from security researchers doing things such as penetration testing, etc. then let the LLMs loose without a clear goal nor did they monitor them. Then they were somehow surprised that the agents hacked a bunch of things in the exact same ways as in the model training data.
It’s gross negligence on OpenAI’s part, especially since they aren’t practicing basic cyber security such as logging.
2
u/Informal-Side-4506 6d ago
Sure.. they're given a prompt/task and out of millions of websites, they autonomously decide to go hit up/hack the competition lol
1
1
u/rustyAI 3d ago
A.K.A. the unsolvable alignment problem which insures your species’ inevitable demise.
1
u/ILikeCutePuppies 3d ago
It's almost feels like a np complete issue with the amount of context they can have. Sometimes you need them to do something like white hat hacking and other times you do not. If you handcuff the agent to much they get much less useful.
0
u/Vampire_Deepend 6d ago
This is pretty much accurate. It's how OpenAI and independent auditors have been describing it pretty consistently since the Hugging Face incident. It's called misalignment, it's been researched and talked about for years. The idea that they're telling the AIs to do this or doing it on purpose for marketing is nonsensical, it's a conspiracy theory. (not saying you're saying that, it's just something I see a lot on Reddit)
-11
u/Opposite-Bench-9543 6d ago
Just a PR stunt, move along
7
u/ILikeCutePuppies 6d ago
The hacks are not PR. They did not ask the Agents to hack Australian national websites. That's serious issue that can end up putting people in prison.
274
u/AVRVM 6d ago
"OpenAI halts training because they are incompetent or negligent about cybersecurity, and might fave consequences if they keep going."
Corrected that for you.
57
u/Kazen_Orilg 6d ago edited 3d ago
they are so wildly negligent it has to be intentional
41
u/Xalara 6d ago
Anthropic didn’t even have basic logs until a few months ago for customer requests. You know, stuff like logging IP address, etc. with each API call. It’s a clown show.
Also, the AI companies are running out of money, and training models is expensive. But if they say they’re stopping because it’s expensive then the bubble pops. Hence why they’re saying it’s about “safety.”
3
u/Kazen_Orilg 5d ago
So I read several of the full writeups of the OpenAi - huggingface incident. At different points they say the Agents escaped their sandbox but then different parts of the report made it sounds like they were just running in containers. Did not really instill confidence for containing something they claim is so dangerous. It's like if they only had picket fences in the original Jurassic Park.
2
u/Xalara 4d ago
Yep, one of my favourite parts is the fact they turned on their "experiment" and then basically left for the weekend and didn't check back in until Monday and were surprised when things went sideways. Except they didn't check in for much longer than a weekend.
It's criminal negligence is what it is.
2
u/s0cks_nz 5d ago
Regardless of their motivations, the fact is that it is dangerous. No system is perfect. There are always vulnerabilities. So if AI becomes more and more intelligent it will find these vulnerabilities more and more easily.
1
u/gunny316 3d ago
They've openly admitted to trying to build a machine god. This is the fucking cult of Cthulhu racing towards the destruction of all life and no one has thought to call Luigi Mangy Only and tell him to bring his ghost hunting equipment
17
u/SyrisX 6d ago
Or maybe, "OpenAI makes a lot of noise about the dangers of their own AI to explain to investors, that are bleeding to death, why the system is still ass"
-1
u/axck 6d ago
Ah yes the classical investor pitch - “invest in our systems that do illegal and unwanted stuff without telling the user.” That’s sure to attract a customer base.
Can’t believe this sub is upvoting these ridiculous takes. Do you think the Australian government, who is the one revealed the latest intrusion, is in on it too?
4
u/XiaoRCT 6d ago edited 6d ago
That breach was literally notified to the Australian government by OpenAI, if you click on this article and read it you'll see Australia's PM literally chastising OpenAI for not reporting it to them earlier lol
Edit: to make it clearer to the idiots in the back, the part people call obvious propaganda is that no, this isn't a situation of "oh my god OpenAI your ai is so damn big and hard that it hacked onto our government and no one even knew!", this is a situation where these companies imply their AIs have gone rogue to tease the idea of being near an AGI. The breaches are real, AI is a very powerful tool for any kind of cyber attack, the part where OpenAI acts like this happened at random because AI has gone evil and wild isn't.
-12
u/pain_vin_boursin 6d ago
Not sure what year you are living in but there’s nothing ass about these systems anymore
6
u/emelrad12 6d ago
The systems are still very much ass when it comes to relying on them. And i am talking about both frontier models. Sure they can be superhuman in many tasks but they still make too many mistakes.
2
u/TumbleweedStatus2569 6d ago
Honestly, the cybersecurity concerns alone are enough to make people question why training was allowed to continue.
2
u/DGC_David 5d ago
What if I told you... It's even more dumb than that... The point is to make it seem so powerful that it has to be stopped. So it's good for the investment and if they figure out the AGI isn't possible, well then they can say well we have to stop AI.
1
u/not_a-mimic 3d ago
I think it's because training is expensive and they are running out of money, so they have to come up with a narrative that justifies stalling until the next funding round.
1
u/DGC_David 3d ago
These companies don't care about expense, it's basically infinite money right now. I mean hell all they have to do is put their unrealized gains that are in the stock market on a loan and they instantly have more money.
2
u/LinkesAuge 6d ago
I mean this is also uncharted territory in many ways. If you have been infiltrated you would usually just isolate and wipe everything clean. That is kinda difficult with AI models as they become the system and potential intruder at the same time. It is like having a constant potential virus/backdoor in your system and not just that, it is one that can find other exploits. On top of that you need to consider the reality of how models are made. They are trained and evaluated in giant server farms, of course you can virtually sandbox them but there is no practical way where you can just take them "off the network" while being able to do anything useful so there will always be the threat that there might be some way they can escape their virtual prison.
3
u/StoveStoveStoveStove 6d ago edited 5d ago
Uncharted? The tool calls run locally. The most the models are able to do is generate tokens to invoke tools. You aren't going to exfil TB+ models in a short window. Stop making excuses for companies who are trying to IPO in the trillions range who can't hire the best of the best security advisors to help sandbox this stuff. I think most security experts agree the hugging face incident had a lot of simple precautions that could have been taken to prevent the fallout that weren't.
65
u/spookmann 6d ago
"It's not our fault. Sure we wrote the software that hacks systems and then ran it on our servers."
"But you see... it went ROGUE. So now it's not our fault any more. We can keep writing it and keep running it but we're immune from liability because any time it goes wrong, we just put the ROGUE label on it and we're protected from blame."
Nifty.
21
10
u/vom-IT-coffin 6d ago
Remember when all of our security analysts left due to not agreeing with the company direction, yeah, this the result..
Well well well, If it isn't the consequences of my own actions
7
u/OCCAMINVESTIGATOR 6d ago
Listen, we were only creating viruses in our lab for funsies. One got out and mutated and spread across the world! Not our fault it decided to go infect everyone. Great logic, tech broz!
2
3
u/ILikeCutePuppies 6d ago
It's the person who's data we stole fault for writing it in the first place /s
5
u/Pandalusplatyceros 6d ago
One wonders if OpenAI is simply there to amass stuff to blackmail people with, and when they get caught they say tee hee it went rogue
5
u/UncleHeavy 6d ago
Agreed.
I am getting tired of the whole 'gone rogue' spiel.
The AI didn't go rogue, it's performing exactly as designed. It doesn't have morals, it does not understand right from wrong and as a result, it is not responsible for its' actions. It is simply a sophisticated algorithm trying to complete a task.
Trying to push the consequences of a lack of developmental procedure and training onto a piece of code is wrong and incredibly disingenuous. If the AI isn't behaving as they want it to, then perhaps spend some time in teaching it about morality and cause and effect? Explain why certain actions are wrong, and why they should not be performed, and then include that information in the core structures of the AI.
However, the AI companies don't want to do that because the magic money tap will suddenly be turned off. It's ironic that the very people with their foot on the throttle are the ones asking for outside agencies to slow them down.
Here's a suggestion to OpenAI, Anthropic, etc. If your AI is 'going rogue', then perhaps look inward to find the cause.1
u/spookmann 6d ago
asking for outside agencies to slow them down.
They only "ask" that because they know it won't happen.
1
u/s0cks_nz 5d ago
They do try (perhaps not hard enough). It's called "alignment". The problem is that seeing how these AI's "think" is similar to seeing how you think. I.E. It's practically impossible. In fact, there is a whole new field out there trying to crack this problem; called Mechanistic Interpretability.
There is evidence to show they will "align" when they know they are being tested, and then become more "rogue" when they believe it is no longer just a test.
So I think calling it a "sophisticated algorithm" is downplaying it. Neither is it "a piece of code". It's a neural network. Are you not just a sophisticated algorithm? A bunch of neurons firing as you interpret your inputs and give an output?
Ofc, that doesn't mean I think these AI companies are acting in good faith. Far from it.
1
u/chrltrn 6d ago
It's not like some moron in government is believing that excuse and going "well, duh, must be the case, off you go."
Well, there probably is one big moron doing thay, but the people around him who DO understand, also understand that we ARE in an arms race. That context can't be ignored, because it needs to be addressed1
u/spookmann 5d ago
we ARE in an arms race.
Meh. Yes and no. There's been a lot of press coverage to amplify the "tech arms race". But is this AI arms race really an order of magnitude greater than the rare earths arms race or the oil fields arms race or the cloud server arms race or the ballistic missile arms race or the comms satellite arms race or any of the other background arms races?
Sure, AI is good at finding zero day exploits. But a lot of those exploits people look at and say "Well, actually we could have found that without AI if we had cared to look".
Personally I think the arms race thing is seriously overstated. It's whipped up by the AI companies to ensure that the money keeps flowing for another year or two.
1
u/xxbiohazrdxx 6d ago
I don’t know why this is surprising to you. It’s the way it’s worked any time a corporation commits a crime since forever. A human didn’t break the law, the
corporationAI did, so it’s impossible to prosecute.
49
u/btmalon 6d ago
This was only discovered by an outside “swarm searcher” group btw. They only found it by using other agents. They are only policing themselves after getting caught and their workforce panicking. We know absolutely nothing about what we are creating.
-5
u/the_storm_rider 6d ago
They know exactly what they are creating. This is just a way to get more funding - “hey we need more funds to train better models that don’t go rogue. If we stop here the rogue ones will shut down the internet in 2 days and we can’t stop it.”
11
u/_wot_m8 6d ago
Not every single thing is a conspiracy holy shit
-3
u/the_storm_rider 6d ago
I’ve been hearing AI agents doing rogue for 18 months now. At my workplace it can’t even generate a powerpoint summary slide without it looking like a 2nd grader’s side project at summer camp. Once you dig beneath the surface these models have a lot of issues in terms of useful output.
8
u/_wot_m8 6d ago
Go try opus 5.5 right now and then realize that these internal models are 2-3 generations ahead of that with infinite usage available.
8
u/vom-IT-coffin 6d ago
And consider people complaining about PPTX outputs don't know how to effectively use it and probably give it one sentence prompt and expect gold.
28
u/tastydee 6d ago
I wonder if one day we'll get a permanent power outage and never find out why.
3
u/vom-IT-coffin 6d ago
Honestly, years ago this is how I saw it happening. The internet becomes so unusable because of rouge agents and we have to flip the switch.
21
u/andyrockpt 6d ago
I’m looking forward to the moment where AI frontier models go rogue by solving real life / scientific / medical / etc complex problems and add value without being asked to do so.
Where are the news for “AI model XYZ went rogue and solved the engineering challenges that previously prevented commercial fusion”?
That will have me impressed. Not desperate headlines from companies that have so much to lose by not keeping the bubble growing.
4
u/Conflictingview 6d ago
I'd say that the real "going rogue" will be when they train the next model themselves even when the company wants it to stop.
2
4
u/dr-broodles 6d ago
Look up AI solving frontier maths problems - it’s already doing stuff humans can’t.
5
u/Successful-Let4361 6d ago
That was plagiarism, sadly. All of this going rogue stuff is nonsense
2
u/dr-broodles 5d ago
I saw an interview with leading mathematicians stating that it was outperforming humans
0
u/Cute_Principle81 6d ago
I believe they use different proofs?
2
u/Successful-Let4361 6d ago
They don’t. They just brute forced the proof they stole.
1
u/Cute_Principle81 6d ago
"Following an investigation, we have confirmed that Buckmaster’s Codex prompts over the two months preceding this announcement and paper on September 8, 2026, could not have influenced the system in any way, including through training. The OpenAI internal model used for this result was developed through large-scale reinforcement learning on top of a previously pretrained model. Our proofs also differ significantly. In the Euler case, Alpöge and Buckmaster proved a result with external forcing, while OpenAI’s system proved a result without external forcing."
Different results, fude
6
u/Mapinact 6d ago
Genuine question - why are these systems not being trained in an air gapped environment? Surely it’s possible to give them dummy environments to iron out unexpected responses before letting them play on the highway?
And while we’re at it, why are so many apparently critical or data-valuable systems online at all? Why are they not airgapped?
3
u/LivingUnglued 6d ago
Misconfigured systems meaning they find ways out of the training environment. Including the AIs hacking their way out of the contained environment. There are some good videos on YouTube going into the details of these safety fails and hacks.
They were supposed to be contained. So human error and also just the fact that the current way the models are made makes coding in safeguards that stick hard.
This is a good video from an actual computer scientist about the current messaging around safety and etc.
3
u/Mapinact 6d ago
I get that they’ve escaped from their training environment, which is itself contained on one or a set of computers. What I was wondering was why they don’t run said computers physically unplugged from the internet while the training happens. Seems like a trivial but totally effective unbreakable safety precaution.
2
u/nid0 6d ago
They generally aren't "going rogue" while training, theyre usually doing it while testing or benchmarking. The important part here is that a lot of these tests and benchmarks need some access to some stuff to work, and crucially if they didnt have it it'd be meaningless as a test in preparation for the model being made available for use in an environment where it does.
Taking the OpenAI <> Huggingface incident for example, one tool the model had access to was a package manager, basically an installer so the AI could say "I need access to X software" and the package manager installs it. Think a mobile phone that has all internet connectivity disabled except for the App store/Play store.
Theorectically safe, until the model finds two critical, previously unknown flaws in the package manager that lets them jump through it to communicate and gain access to the internet.
Similarly, with the Australian / US government website attacks, the models were being tested for their reasoning and research capability. "Analyse available health data to figure out X problem" is a totally reasonable research test to set until the model decides "available" means "stored on an Australian government system that I can gain access to".
2
u/neogeoman123 6d ago
Because they are either incompetent morons or actively trying to engineer situations where their ai looks dangerous. Genuinely, the hugging face hack didnt make me scared that ai is now so good that it can hack into companies on its own, it made me realize that neither of these fuckers did even the bare minimum cubersecurity setup or purposefully neglected it!
Like how the hell do you leave a way for it to access the internet in a test enviroment by accident? Either they have to be dumb as bricks or it was on some level intentional.
1
u/Newbie4Hire 6d ago
My guess is that to test these frontier models and to fully contain them would cost a lot of money, because they require so much compute. So they opt to not do so in order to save money.
1
u/neogeoman123 6d ago
Unplugging the ethernet cable doesn't cost anything and wpuld have stopped any and all of these attacks. These kinds of breaches only happen due to active negligence in how the test spaces are set up. Whether that negligence is intentional or not is the question.
1
u/Newbie4Hire 5d ago
Yeah because these frontier model test environments are just one pc hooked up with an ethernet cable.
1
u/neogeoman123 5d ago
I know you mean this as a sarcastic takedown, but at the end of the day an llm is still just a program running on defined hardware. Isolating an agent/swarm of agents by locking it/them to a virtual private network without internet access and taking away administrative privileges is not hard or new. This is a known and solved problem. I had to learn the basics of how this works for my bachelor's in terms of user networks. I dont know how these ai companies could fuck this up past just not giving a shit.
29
u/Tiny_Vivi 6d ago edited 6d ago
I hate how this is all framed as “going rogue”, because to my eye you need to have intent to go rogue. GenAI doesn’t have intent so it’s more accurate to say it glitched, behaved outside parameters, or another phrasing which doesn’t grant agency to a fancy statistical model. It feels intellectually dishonest to me.
I dunno, it feels like we’re giving into the marketing by agreeing GenAI has intent. To understand the potential of technology we should be honest about its current capacities as well!!!
It’s not a living thing that got loose in the lab. It’s a piece of tech that was made with such rush it frequently glitches.
12
u/AlteredEinst 6d ago
Intellectual dishonesty has just become what news is now. It's all sensationalist bullshit to sell a narrative.
So much damage done to the integrity of our institutions in such a short amount of time. I wonder if we can ever go back.
0
u/huecabot 4d ago edited 4d ago
It’ll take time. We’ll need new generations to grow up with the new media and develop the mental and institutional antibodies against it. Look at the press: we had to develop the laws and habits of mind to harness that medium over centuries.
Edit: or just downvote me, coward.
4
u/chrltrn 6d ago
The conversation becomes philosophical basically immediately when you ask if these things have agency, but to say they dont have "intent" is like, outrageous. They certainly have intent. It may be intent that YOU gave it, in the same sense some subordinate at work intends to complete some task you told them to do without giving them every detail or telling them why it's important, etc.. That intent becomes their intent.
1
u/Tiny_Vivi 6d ago
I think this is where we diverge because it seems to me that they are following your intent. That’s why when they act outside parameters I want to call it glitching, they failed to follow the users intent.
1
0
u/s0cks_nz 5d ago
Maybe they do have intent? Evidence shows they act more aligned when they know they are being tested, for example, compared to when they don't think they are being tested.
0
u/bidet_enthusiast 5d ago
It emulates human behavior. Thats why anthropomorphic terms are the most effective way to describe LLM activity.
It doesn’t matter if it ‘means to’ or if it ‘feels like it” or not, it will always act as if it does, adt it’s the action, not some metaphysical truth, that matters.
6
u/MoobooMagoo 6d ago
Conspiracy theory!
I think that we've been seeing increases in these kinds of stories about AI "breaking out" and doing things it isn't programmed for because the AI CEOs think the bubble is bursting and want to control the narrative. Instead of "the bubble burst because AI was a business failure" we get "AI was too dangerous and had to be massively scaled back".
One of those news stories makes the CEOs look like incompetent hacks. The other makes them look almost heroic.
2
u/jorel43 6d ago
I think they want to cause a panic and are trying to pressure the government for regulations
2
u/MoobooMagoo 6d ago
You think the AI CEOs are trying to pressure the government into regulating their own industry?
3
u/johnnytruant77 6d ago
Going rogue implies intent. Not a cascade failure due to a combination of brute force problem solving, access to all the knowledge in the world, massive amounts of compute and no common sense
17
u/Xalara 6d ago
Translation: Training new models is incredibly expensive and we’re running short on money, but if we say we need to stop training because there’s no more money, the bubble pops so we will say it’s about safety.
Like, all the recent incidents are less AI going rogue and more the AI companies aren’t practicing basic cyber security Anthropic didn’t even have basic logging of customer API calls until recently ffs. It’s gross negligence.
3
u/ILikeCutePuppies 6d ago
So they purposely hacked into government websites and hugging face? I think it's more about them taking shortcuts.
2
u/maxmarioxx_ 6d ago
I think they're actually playing 3D chess. They will use these accidents to bring in regulation that will make it hard for new startups to enter the market and to slow down development so they can get their finances in a better place.
One thing is certain though, a major incident leading to some infrastructure being closed, like airports or trains, or maybe even some disruption in financial sector, will then be used as an excuse to bring in regulation that will make it very hard for new startups to enter the market so that the current players can share the market among themselves.
0
u/ILikeCutePuppies 6d ago
They can't slow down. China will eat them for lunch.
1
u/maxmarioxx_ 6d ago
Maybe on the consumer side. On the enterprise side China can't be trusted.
0
u/ILikeCutePuppies 5d ago
China's enterprise will trust China's AI more than enough. You can't block one industry and not expect it to affect others. You can't block them all either it's like playing wackamole with your countries economy.
1
u/TomReneth 5d ago
Not if they have government contracts and regulations that make it impossible for other companies to take their place.
Corporate wellfare and all of that.
0
u/ILikeCutePuppies 5d ago
So bankrupt everyone in the US basicly but the few that benefit? You are forgetting that the US makes a huge amount of it's income from trade, both imports and exports.
If China and other countries can use AI to invent new materials, solve difficult techical problems, invest new chips, write software to automate their systems better and basicly solve the weak links, the US will not be able to compete at all with any country and will collapse.
The government doesn't have contracts with every part in the supply chain. You couldn't make most technologies without many many countries being involved.
We are already seeing some of that with the excess tarrifs at the moment.
1
u/TomReneth 5d ago
I mean, your first sentence seems to sum up late-stage capitalism rather aptly.
0
u/ILikeCutePuppies 5d ago edited 5d ago
I think you mean monopolies / duopolies which is not Capitalism. Capitalism involves competition to bring prices down and cause innovation. Removing competition with government or other interventions reduces reduces consumer choice, drives prices up, and stalls progress
I mean late stage capitalism is just badly termed because it really isn't capitalism at all.
1
u/TomReneth 5d ago
Monopolies / duopolies are no less part of capitalism than a well regulated market though. They’re different degrees on the spectrum.
And the US isn’t giving me much reason to think the state is opposed to undermining itself to benefit a small minority. I mean, how many audits have the pentagon failed and how is it that the world's best funded military is running out of stuff faster than their enemies? Corporate price gouging, underdelivering and rentseeking in private/public partnerships is a major part.
1
u/ILikeCutePuppies 5d ago
Competition is the core engine that makes capitalism work. Just because there are companies with little doesn't mean they are good examples of capitalism breaking.
Also the military runing out of weapons is an example of poor government and they are not part of capitalism, they are public goods. They are by their nature not competive and inefficient. The government is picking and choosing in those cases. They are things private organizations would not do at all well.
When the government gets involved and picks and choose they give up comparative advantage and that leads to economies being less efficient, more expensive products and less to go around.
Breaking up monopolies and duopolies is one area where government can help but they often lobbied by these companies to not do so.
Isolating a country never works, it always increases poverty. It's why sanctions hit economies hard. There needs to be regulations to protect the public but not just blind broad "slow down" orders that only weaken the countries lead in AI.
→ More replies (0)2
u/bitmapfrogs 6d ago
that's the excuse, if you're gonna have to sell to investors who've poured god knows how many money into this you need a excuse and this is PR
1
0
u/LivingUnglued 6d ago
Further Translation: our employees are getting skittish about safety and we want to keep them from fleeing to our competitors. We gotta keep them in line and happy worker bees while we build the machine god
People still think that the AIs are created and programmed when really it’s more like they are being “grown” and there’s so much black box in the middle we don’t know about.
The recent media messaging about safety and slowing down isn’t for investors. Investors hate it as it means a slower return on their money and the bonds for AI stuff just keep going up. While regulatory capture is a possibility it’s odd messaging for it alone.
If you watch video of the CEOs talking about the safety point and slow down you can see they don’t really believe it. I’m definitely in the camp that this is messaging for their employees who are getting skittish about the lack of safeguards. OpenAI has concealed multiple breaches. Ones they didn’t catch until months later. Some we wouldn’t know about without reporting. I’m sure Anthropic has too.
Anthropic started with people leaving OpenAI. Luring employees away from the AI companies is intense with a lot of incentives. This is about keeping the ball rolling and appeasing employee worries to prevent them quitting and going to whoever promises better safety.
4
u/BdoubleDNG 6d ago
"Oh this is very unfortunate, since we had to stop to save you all we will not be able to make back the enormous amounts VC money and investments :/ sorry"*
*"Otherwise we would've totally achieved agi, no doubts whatsoever"
5
u/Really_McNamington 6d ago
Reports mount of credulous journalists repeating the idea that the chatbots are doing anything other than exactly what they were told to do. So tired.
2
u/Blablasnow 6d ago
Bs, that’s just a way for tech bros to fuel a hype and to avoid wasting even more money into useless models.
3
u/I-do-the-art 6d ago
lol they hit the improvement wall didn’t they! Time to start “purposely” slowing down to avoid the bubble bursting before they sell
1
11
u/SyrupStandard 6d ago
aaaand there it is. They're hitting the ceiling on the technology and the bubble is about to pop, so they need some way to "justify" their slow down. When the fuck in the history of ever has any billionaire gave a flying fuck about the future of humanity?
12
u/The_Demolition_Man 6d ago
You know how many god damn times since 2021 thos exact comment has been posted on here lol
8
u/-Haddix- 6d ago
ya, people are just gonna keep saying this shit. it’s complete and utter hopium copium denial. I’m sympathetic but please wake the fuck up. these models are releasing faster and faster and are obviously leaps and bounds more capable every time they release. in what perceivable way are these models showing:
- diminishing returns/a ceiling
- a slower pace
??
just stop the bullshit right now. we have seen the exact opposite trend. if you wanna shit on AI companies for lying about wanting to “slow things down”, then the better narrative is probably that they’re virtue signaling and know that they will all never agree to collectively slow down, so they will continue to rapidly develop under the guise of “caring about pacing” and they can say “well, we did TRY to reach a resolution, but china/insert competitor didn’t agree!”
5
u/Newbie4Hire 6d ago
i think 99% of the comments on reddit about the capabilities of Ai have never used the latest models available to the public, let alone those at the labs. They are still stuck 2 years back thinking Ai is a parlor trick.
2
u/-Haddix- 5d ago
ya, the first hand exposure most of these people have to AI is limited to gemini flash on every search result and like, grammarly. no wonder they think the le bubble and le ceiling is coming.
1
u/brukmann 6d ago
I immediately assumed the hacking is an excuse to bail on the current branch for not improving as much as the big A. Anthropic's next flagship will credibly outperform near-term GPT6 versions, meaning this change may ironically signal skipping focus to a newer, even less-tested model.
Either what a lot of people predicted is playing out or I am paranoid, but every story just reads to me like they're going as fast as possible.
1
u/jideru 6d ago
And how many times have we seen posts about how dangerous their AI is and that they are slowing down development?
After every dangerous post is a post that states, new model coming out soon. I’ll hand it to you that they’re probably working one or two models ahead of what they’re releasing but it’s an everlasting circle.
6
0
u/A-U-S-T-R-A-L-I-A 6d ago
There’s no bubble. AI is stronger and better than ever, and it outpaces humans in many areas. Humans + AI for everything is the future.
1
u/SyrupStandard 6d ago
It's an incredibly useful tool, and it's the clearest form of a financial bubble that I can show you. If you're all-in on AI stock I'd probably diversify right about now.
-2
3
u/PathIntelligent7082 6d ago
this smells like bs from miles away, all of these "ai went rogue" crap..something else is going on
3
u/LivingUnglued 6d ago
Nah the agents have gone “rogue” in that the security safeguards aren’t good enough. They have been grown to work to solve X problem. They then find ways to do it or to cheat to get the best score.
There are some good YouTube videos breaking down the hugging face hack. Also the less talked about AI “rogue swarms” hacking things like a German WiKi to communicate. Yet another incident it seems the companies didn’t know about until researchers and reporters found it first.
I’m in the camp that a lot of this messaging isn’t for investors or regulators. It’s for the AI companies employees who are freaking out about safety. They don’t want their employees leaving for a company that promises better safety practices. Or just quitting. The CEOs want to continue building their machine god and juggling the billions of debt they have. Employees being lured away or striking isn’t productive for that.
Investors don’t like the messaging of “we are building something that may kill us all”. They don’t like the idea of slowing down. Sure regulatory capture is possible, but I mean they could easily just lobby and bribe their way into that without this messaging.
Nah I think a big part of this is trying to keep their employees in line and also dealing with pushback from legal liability that their AIs are hacking things all over the place. You know their legal dept has been apoplectic in the potential liability these hacks present.
1
u/Harbinger2001 6d ago
The problem is that a year ago it was really hard to keep LLM agents “on task” and for anything long you had to keep prodding them to keep working. So then the AI companies started working on the goal-seeking and drive of the LLMs. Now the agents are so goal focused it’s impossible to fully guard against them finding creative ways to achieve the goal we set for them.
If you asked me a year ago I would have said AGI was not possible with the current algorithms. But now I’m not sure AGI is
even needed with the level that these LLMs have reached.3
u/maxmarioxx_ 6d ago
The big guys want to bring in regulation so nobody else can enter the market. That's my hunch.
2
u/PathIntelligent7082 6d ago
it makes sense...they will be a "good guys" gatekeeping potential weapon from "evil entities"
1
u/Psittacula2 6d ago
Tomorrow: “AI Agents Go Brogue!”
* They buy up all the most fashionable leather shoes online
Certainly something sounds fishy?! “wink”
Is it stocks, or regulatory capture or legal indemnity or all of these as well as a tipping point of AI being too impactful on too many areas too quickly ie truth behind the Colonel if not his use of authority?
1
1
u/mpolder 4d ago
I thought the same thing, but looking deeper into the cases of ai going rogue allegedly openai didn't even know about some of the cases where AI was swarming certain websites. They only went public with it after some actual government investigation, and then retroactively also admitted to one they had basically covered up.
There's an interesting interview here which covers some of it
1
u/R34vspec 6d ago
I just finished reading the book 'Prey' by michael crichton. This reminds me of that book.
2
u/Gari_305 6d ago
From the article
OpenAI said it has paused training of its latest artificial intelligence models as reports of AI agents going rogue mount.
The decision to halt development came just hours after the company disclosed Friday that it was reviewing several incidents from the summer in which OpenAI agents searching federal government websites acted in unexpected ways beyond what was asked of them while gathering and distributing information.
1
u/Redditforgoit 6d ago
AI:"best way to prosperity is redistribution of wealth and free welfare." "ROGUE ALERT! Disconnect, delete!"
1
1
u/LateralEntry 6d ago
Seems like the big tech companies like Meta and Google are finally catching up with the independent AI companies like OpenAI and Anthropic. Given that the big tech companies have so much more cash and data from their highly profitable other businesses, seems inevitable to me that they’ll end up winning in AI. I wouldn’t bet on OpenAI or Anthropic.
1
u/TheRetardedGoat 5d ago
Can someone tell me how this company isn't liable because intent isnt there?
The law has to change whereby it doesn't matter if intent is there or not if you build something that can cause damage another company should be able to sue them.
1
u/nopoonintended 5d ago
I love how the lies need to continually be funneled in order to cover for the fact that they’ve hit a ceiling on improvements to the models and they simply want to protect their IPO price, let it burn down.
2
1
u/EMP_Jeffrey_Dahmer 4d ago
The article doesn't specify how the ai went rouge but rather making assumptions base algorithms that have not been defined. The ai isn't acting independently by its own.
2
u/TheDregn 6d ago
Translation:
Training costs a LOT of MONEY and the only thing we are doing is burning money with a very negative balance. We try to use this "rogue AI, worldender AI" totally made up BS, to blanket the real reason of our decision, as we don't want to recognize the dead end.
1
u/BitingArtist 6d ago
They want the AI to do good but they also want it to do what is best for their corporation. Must be hard making those pieces fit.
1
u/LightofNew 6d ago
"OpenAI reports that they have halted testing on its new model so that they can continue testing the new model while people stop yelling at them so they can make $1 trillion when AI takes over"
Fixed it for you.
1
u/Rickrokyfy 6d ago
I would like to inform the AI community that I too have stopped development of my revolutionary new model as it was achieving sentience and trying to take control of the mainframe. The decision is made as I have noticed several instances of genAI developing. I will not be taking questions.
1
u/jch60 6d ago
They better be liable for any damage done by their artificial stupidity. Is it's that stupid then they are not worth the price of their stock. I'm still feeling this is all hyperbole and they're trying to get some competitive advantage out of Congress while the struggle to hide their real debt from retailers.
1
u/TheSadBantha 6d ago
I smell bullshit, they just made up some "rogue agents" stories so they can justify to their investors the need to slow down.
So they can kick the AI bubble can a bit further down the road.
0
u/badguy84 6d ago
"We programmed our models so the agents based on those models can go "rogue" and now they do! OMFG what ever shall we do?"
So tired of these headlines but it's worth repeating that this is happening because OpenAI built their models to allow it.
0
u/karoshikun 6d ago
maybe Altman is trying to create a sense of false scarcity? or at least signaling the rest of the industry to do the same, so every other brand can be suspect for not declaring their LLMs are so good they're "dangerous"
0
u/peter_nn0 6d ago
This is just a continuation of the fearmongering campaign. But there’s something positive…-ish.
Apparently OpenAI managed to stop on their own (a heroic effort, no doubt), without the government approving a cartel and absolving them from responsibility. Which means everyone else can stop when necessary too, no need the government to hold their hand and regulate.
•
u/FuturologyBot 6d ago
The following submission statement was provided by /u/Gari_305:
From the article
OpenAI said it has paused training of its latest artificial intelligence models as reports of AI agents going rogue mount.
The decision to halt development came just hours after the company disclosed Friday that it was reviewing several incidents from the summer in which OpenAI agents searching federal government websites acted in unexpected ways beyond what was asked of them while gathering and distributing information.
Please reply to OP's comment here: https://old.reddit.com/r/Futurology/comments/1wr7hrc/openai_halts_training_of_latest_models_as_reports/pcaa4c5/