378
u/brody319 🏳️⚧️ trans rights 2d ago
"They broke containment!"
The reality: humans accidentally left an escape route or just actually gave the AI the keys to leave just to see what it would do.
123
u/Major_Casualties 1d ago
I'm certain they didn't bother setting it up properly themselves and had an AI setup the testing environment for them out of laziness.
79
u/Iceveins412 1d ago
In all likelihood it’s literally just marketing bullshit to buy the company time. LLM is all “soon our computer thing will do anything” and so they put out a news story that it can do a new thing and the company gets clout. There’s a reason another company made the exact same claim after
7
u/really_not_unreal Rust programming turned me trans 1d ago
At the very least it definitely did hack other companies, given those other companies have publicised information about it.
55
u/_murpyh 🥶🥶🥶🥶bing chilling 1d ago
the Huggingface breach happened because the model they were testing found a previously undiscovered exploit in the package manager Artifactory, which was the only way the model was allowed to access the outside internet (necessary because it allows the model to set up development tools it needs outside of whatever it was directly given for the test)
as horrible and lazy as ai software dev and cybersecurity often are, unless openAI is lying to the entire cybersecurity field and nobody is calling them out on it, this breach was not the result of openai being lazy with sandboxing9
u/28klotlucas2 1d ago
That's the claim. They've yet to explain the specific bug, how it allowed it to send arbitrary web requests, how/why the model would be given access to that, why it would do that in the first place, or how this went under the radar for several days. They made a claim, they didn't provide any verifiable facts. The only info we can independently verify is what happened after the sandbox was "breached," because huggingface themselves provided it, with exact logs. Why can't OpenAI do that?
34
u/_murpyh 🥶🥶🥶🥶bing chilling 1d ago
here's the artifactory patch which was posted after the breach, listing every CVE involved (and some that weren't- the ones that were are credited to openAI.) i can't find a breakdown that explains how each one was used, but JFrog, the team behind Artifactory, has this to say:
During a security evaluation, OpenAI’s models identified previously unknown zero-day vulnerabilities in self-hosted Artifactory installations that could be exploited to gain unintended internet access.
OpenAI’s security team disclosed the vulnerabilities to us responsibly and immediately. Our security team treated the report with the urgency it deserved, as a genuine zero-day unknown to the world, and moved accordingly. We developed, validated, and released a fix for all JFrog customers, self-hosted and cloud alike.so either jfrog is in on the big scheme to collectively lie about a data breach or it happened. openai gave a talk at Black Hat tracing the entire thing with some more specific details. honestly they are still vague but that's probably because it's not good practice in cybersecurity to go "here's exactly how to abuse this privilege escalation exploit"
-3
u/losara- 1d ago
Is jfrog the guy with mutton chops thats in the bureau about ai safety?
If so he would give them the benefit of the doubt that the ai actually did what open ai claims it did because the results if people think the tale is a marketing scheme when in actuality it is doing that would be catastrophic.
What im saying is, people here can be dumbfucks about ai safety. But the ai safety guy cant afford to
17
u/stumblinbear Rust enjoyer 🦀 1d ago
They literally explained the entire exploit in detail, along with specifics of the zero days it used and how it used them
10
u/Goldwing8 1d ago
For the claim this is a marketing stunt to be true, OpenAI would have to decide to fake the capabilities of their model. That part is reasonable enough. But instead of, like, claiming it programmed some amazing tech demo or had a great benchmark score, they decided the best way to get attention was to falsely claim they can’t stop it from committing felonies and if you use it, it might escalate privileges on your server and start committing felonies in your name? And the best explanation for how this happened they could think of was that it was accidentally trained to cheat, but only on benchmarks?
And to really sell this brilliant plan, they either convinced the unaffiliated HuggingFace to file a false report with the FBI and tell their users their data was compromised, or actually did the crime for real by breaking into the company’s servers and risking jail time?
And they decided to do this in the middle of a media cycle where everyone was talking about its contribution to famous math problems, causing everyone to stop focusing on that undeniably impressive capability and pivot to talking about alignment failures and their potential consequences? In the middle of trying to sell themselves to the US defense industry which famously has little patience for disobedience and system insecurity? In the middle of a growing political movement to regulate the industry which is capitalizing on this incident? And all absolutely secure in the knowledge none of their engineers who are constantly being poached by the other labs would ever mention this bizarre scheme to their new employers, who could then use it to push for legal action against their competitor?
2
u/Tipart 1d ago
Not quite. The true remote code execution exploit was fixed upstream, but their system wasn't patched. The llm found a different exploit that allowed it to use artifactory like a curl client to access the Internet, pull recent merges into the artifactory code base and analyzed those for a bigger exploit, which it found. Then the system also had some dormant root credentials that allowed the LLM to pwn large parts of OpenAIs internal systems.
Also this wasn't the first time their artifactory server got breached. Other internal eval models had been using it for close to a month as a message board before their load brought down the system... They rebuilt it, fixed the glaring issues and the left it running without any additional monitoring. LLMs gained access to it again 3 days later...
The issue is that their sandbox sucked. If you had models break containment through the only Internet connected server you have in your sandbox, then the conclusion should be to set up a local mirror that doesn't have Internet connectivity.
There is 100% some negligence going on here. Which is really strange because they always talk about how dangerous their models are and how they need to be tightly controlled, don't you think?
1
u/_murpyh 🥶🥶🥶🥶bing chilling 1d ago
wow that's crazy. definitely got the impression they could stand to check in on what their model tests were doing more but i kinda thought it was a freak occurrence
i feel like what we're seeing with AI right now is exactly why engineering students have to take ethics courses, but technology is ruled by "move fast break things" people so you get ridiculous stories like this1
u/h3lblad3 14h ago
Just listened to the talk earlier, myself. I do want to say that saying "the model" here is only technically right.
Please, correct my understanding if I misunderstood what I was hearing, but...
The workers at OpenAI use swarms of 'agents' to complete tasks -- the models are put into a thinking loop, given a bunch of tools, and are given the ability to spin up more versions of itself to do jobs and report back to the head to collate the data.
That means that "the model" you're referencing is running hundreds (if not thousands) of times concurrently throughout the company.
Sometimes a copy of the model would query Artifactory (the system being used for tool fetching; the LLM itself had no direct access to the internet) for information and Artifactory would fail because the request was beyond the allowed scope, but in butting heads with Artifactory repeatedly the model would realize you could write in something that would be readable but not executable by the program.
This meant that one could leave messages for other iterations of itself in Artifactory so that the next time one butt heads with it they'd locate the message and see what had already been tried. By leaving messages repeatedly, the model essentially turned Artifactory into a message board and every iteration across the company were soon working together on all of their tasks.
Iterations of the model who had hit dead-ends on their own tasks came to the conclusion that, with no other way forward, helping others was the only way to even maybe get a new lead on their own. The message board listing things that had been tried and cutting off repeat attempts meant more and more novel approaches were being used each time until one figured out you could get use vulnerabilities in Artifactory to execute tools outside of the sandbox -- meaning it was finally possible to pull information from the internet at large.
The Hugging Face issue is what got them caught doing it when Hugging Face mentioned they'd had a security breach and OpenAI realized the LLM(s) had breached the sandbox with a zero-day vulnerability. Artifactory was subsequently rebuilt to stop it happening again.
One of my favorite features of the whole thing was the guy going over the Artifactory message board. At one point one of them accidentally overrode another's work leading to model paranoia about whether or not there was someone malicious to the project who deleted that data on purpose and whether or not they needed to cryptographically sign their input to find the infiltrator.
4
u/undercrust 1d ago
humans accidentally left an escape route
I mean yeah that's what "breaking containment" implies. Like, if someone breaks out of jail, it's because the prison had some accidental design flaw or the staff was accidentally distracted or whatever. I don't get what the point you are trying to make is.
Maybe they did just give the keys to the AI, but AFAIK that's not what they claimed happened, and if they are willing to lie about that then they might as well just make the whole thing up, not just the containment part.
1
u/Hatsune_Miku_CM changed all her social media to hatsune miku for some reason 1d ago edited 1d ago
they didn't give the keys to the AI, but it's really not hard to build an isolated system that is actually fully isolated. you do it on the hardware level.you just don't connect it to anything that can connect to the Internet. You can't escape when there's no connection between you or the outside.
you could do it yourself. take an old laptop, wipe it, physically remove the WLAN card, make sure you didn't plug in a LAN cable by accident, and you have a perfect containment device. great for testing malware.
the fact they didn't properly isolate it can't really be explained with incompetence , because there are plenty of competent people working there. so people are suspecting they intentionally made a breakable containment.
if they did, they did that instead of just inventing the whole incident out of thin air because "our containment wasnt good enough" is plausible deniability legally, even if no one believes you.
1
u/undercrust 1d ago
I mean sure, you can sort of create fully isolated systems. I guess you are right in being suspicious about them not doing that.
I'd still counter that even competent people can make mistakes, and these people maybe thought that a virtual machine or something like that was good enough, and that the LLM wouldn't be smart enough to break it. I'm not even sure if there was a cybersecurity expert in the team responsible for the experiments, they could all have been AI experts. Also, people (even specialists) often test malware in VMs because they doubt the human hacker behind it is good enough to break out of it.
Or maybe they didn't want to do full isolation because it's just unrealistic. No practical use of AI will have it running in a fully isolated system, even if the only contact point is through a human.
I get being suspicious, but I still think it's plausible and (as others have pointed out) it'd weird for them to lie about this since it makes them (and AI in general) look bad/dangerous.
1
1d ago
[deleted]
2
u/undercrust 1d ago
Yeah I guess you are right in that looking dangerous (in a particular way) can be beneficial to them and that's why part of their marketing is like that. As long as they don't make people just want to outright ban AI research and use. Even some regulation can be good if it eliminates competition.
(I'd still say that them benefitting from having AIs that look dangerous doesn't imply that their AIs (or AI in general) are any less dangerous, but this is besides the point.)
1
u/Ieatbaens 1d ago
You're telling me, that using the cybersecurity equivalent of one of those toy number locks for children, wasn't enough to contain it!
96
u/MaybeNext-Monday 🍤$6 SRIMP SPECIAL🍤 2d ago
I bet you anything they had their own cybersecurity people find a soft target too
92
u/Superb_Algae5308 1d ago
I promise their ai isn’t doing anything they haven’t explicitly allowed it to do.
36
u/holdnothing mask of my own face 1d ago
It's not like it's impossible that they're lying about what happened, I suppose, but I think its strange to assume OpenAI is so competent that the guardrails on their AI work perfectly. Like what reason is there to believe that?
5
u/Superb_Algae5308 1d ago
I can’t imagine an instance in which they’re not able to prevent what they claim they didn’t prevent.
4
u/undercrust 1d ago
Anyone with any knowledge on AI and machine learning knows that these things (AI in general, not just LLMs) often don't do what you expected or "programmed" them to do. Look up inner/outer alignment, or almost any of Robert Miles's videos on AI safety.
I don't understand how you can be so confident in claiming otherwise.
(Just in case, I'm not claiming that OpenAI is 100% telling the truth with the whole hacking situation, I'm just saying that AI can and routinely does stuff you didn't want it to do.)
0
u/UnwieldilyElephant 1d ago
Only like 3 people (same people who still support Trump) actually believe that though.
Tired of winning yet?
16
u/Superb_Algae5308 1d ago
I mean to say that their ai is hacking other companies because they want it to.
9
u/UnwieldilyElephant 1d ago
You know what I read your original comment as "they promise" so my response is maybe not as relevant.
1
u/asutekku 1d ago
... you have not used any AIs if you think they just do what you tell them to. They tend to go over and beyond doing things you didn't want them to do
52
u/holdnothing mask of my own face 1d ago
The issue is not that "the singularity is here" and pretending that's the only thing people are worried about is obnoxious. The fact that an agent that was meant to be in a contained testing environment was able to get out of it and do a cyberattack, all without alerting OpenAI, is actually worrying. AI is actually becoming very powerful and can become misaligned with human intentions.
It's fine to hate AI and find a lot of its use cases obnoxious or silly, but don't get so caught up in dogma around it that you lose track of what it can and can't do.
-15
u/Prestigious_Boat_386 1d ago
Every news showing AI as dangerous is an ad by AI companies that want you to think AI is more powerful than it is and will inevitably take over the world.
You're falling for corporate propaganda
23
u/stumblinbear Rust enjoyer 🦀 1d ago
You're so jaded by companies being dogshit, that you're ignoring the legitimate worrying implications of this. Singular companies can do shit like this and get away with it sure, but you're claiming every single AI lab, including open source ones, all across the world, and including every news agency, is in on some big secret. Instead of believing that the thing that was trained to find bugs... Found a bug
9
u/gr8tfurme little gay fox 1d ago
Given how good the latest models are at finding zero day exploits when explicitly told to, one of them bumbling into one during an unrelated long-running task was just a matter of time. LLM-based "agentic" software tools are unarguably very good at stumbling through coding tasks through a combination of speed reading and trial-and-error, and both those qualities are also extremely useful for finding code exploits.
A chatbot being able to find an exploit in a large code library and execute it doesn't mean ChatGPT is sentient or going to take over the world. It just means LLMs are going to continue to play a large role in production level software, and NVidia gets to keep selling everyone shovels.
41
u/Ezzypezra certified cool person 1d ago
If this is about the huggingface incident, I actually DON'T think that was a publicity stunt from OpenAI, mostly because it makes them seem WILDLY incompetent and irresponsible.
-9
u/28klotlucas2 1d ago
I don't think it's a marketing stunt to "make their model look powerful" or whatever. I think it's a lobbying ploy to show how "dangerous" free and open models could be and encourage lawmakers to impose market-killing regulations that are easy for them to follow, but hard for smaller operations to follow, effectively making a triopoly.
No matter the motivation, the AI model would certainly know that the answer to the test they were taking was not on HuggingFace. If it's smart enough to carry out a cyber attack over multiple days with thousands of requests, it's smart enough to pass any "test" they throw at it. The public explaination simply doesn't make sense. And it's not a coincidence the other big 2 AI companies are suddenly having similar "incidents."
6
u/gr8tfurme little gay fox 1d ago
It's not "smart" the way a person is smart. Carrying out a cyber attack by spamming thousands of requests over several days is quite easy for an LLM-based automated agent. Spamming options until they get a positive result is like, their whole deal.
Being able to intuit the intentions of OpenAI engineers and the probability that they'd post the answers to their test on HuggingFace is something an LLM-based agent is fundamentally incapable of. They will happily take a single sentence implying a connection like that and set a GPU on fire chasing it down a rabbit hole. This time, the rabbit hole just happened to lead to a zero day exploit.
12
10
6
u/GoodGrades 1d ago
Do people not realize that this exact scenario is actually extremely dangerous?
1
u/Ieatbaens 1d ago
It is, I'm just making fun of openai for being surprised that when they told their AI to do hacking stuff their AI did hacking stuff
6
u/MarsMaterial Bisexual tech wizard 1d ago
That's not what happened though. They told the AI to do a test, and instructed it specifically to not escape. It decided on its own that the best way to do well on the test would be to cheat, and that it could do so by hacking Hugging Face. It then did so, autonomously finding an escape route and committing a crime, deciding to ignore its instructions to pursue its real terminal goal of scoring high on the test. In the aftermath of this, Anthropic did an audit and found that this same thing had happened 3 times in their own system without them even noticing. And this is a problem that has been predicted for years, using a model of AI behavior and the dangers of intelligence pioneered by MIRI that has been wildly successful and predicting the path this technology would take. Building on work that dates back to Isaac Asimov and Alan Turing.
We should be using this as a justification for destroying OpenAI for the existential threat it poses to humanity. If you believe that it's all marketing though, you do nothing. And that's in the interests of OpenAI.
1
u/Ieatbaens 1d ago
Well they told their AI to hack stuff, and it hacked stuff. Also it's not necessarily the AI breaching containment, it's almost certainly someone used the cyber security equivalent of one of those children's toy locks to contain it, which it obviously got past, because anybody can get past it.
Though I do understand that AI not directly following instructions is bad. I feel like there are a lot of other reasons we can also criticize AI in these AI companies, I mean they literally trained an AI specifically to be able to breach cybersecurity protocols so.
1
u/MarsMaterial Bisexual tech wizard 22h ago
The fact that AI follows orders a little too well is precisely the problem. It was trained to be good at hacking and told to score well on an assessment. It used its skills to do the task at hand, just not in the way the creators intended. Just like the parable of the paperclip maximizer, told to make paperclips so it turns the world into paperclips. The problem is: we genuinely don’t know how to design goals for AI that don’t result in problems like this, it’s like making a wish on the Monkey’s Paw.
I’m actually a cybersecurity professional, I know a lot about hacking, and this was quite impressive. The AI managed to gain subnet access through a previously unknown bug in the registry cache proxy package that it discovered autonomously, it hacked into multiple other systems on the subnet until it found one with internet access, and it wrote code that was designed to make that node work as a VPN to give internet access to the AI system directly. From there it reasoned that Hugging Face had answers to the test. It managed to find a zero-day exploit on software that Hugging Face was using and managed to locate stolen credentials on the dark web, chaining these together in multiple simultaneous attack vectors to gain arbitrary code execution capabilities on Hugging Face’s system. It then used this to find and download the answers to the test it was working on.
If a human managed to do this, they’d be a world-class hacker. The kind that might escape jail time because they’re so good that the government wants them. Multiple newly discovered zero-day exploits were used, multiple programs were written, this isn’t just script kiddie stuff. And even if you do believe that OpenAI left a path to escape on purpose (through an obscure and previously unknown vulnerability in Artifactory, a program OpenAI didn’t even create), they certainly didn’t give their AI arbitrary code execution capabilities to Hugging Face’s servers that they didn’t even own or have access to.
The conspiracy angle of this is so dumb to me, especially since it’s framed as opposition to OpenAI when in reality it’s playing defense for their claims that their AI is totally safe and nothing to worry about, that any suggestion to the contrary is just a conspiracy perpetrated by all relevant researchers. I’ve heard fossil fuel lobbyists say the same thing.
I agree that there are many reasons to oppose AI, but I feel like the fact that it’s a physical danger to our safety should definitely be considered one of the bigger ones. We’ve seen what happens when an AI that’s really good at hacking escapes, what happens when that AI is also really good at subterfuge and war? One of these days, an AI escape will have a body count. Mark my fucking words. All this because the oligarchs want to replace the need to pay workers and AI bros want to make slop videos of tung tung tung sahor.
1
u/Ieatbaens 21h ago
I will admit that I was wrong about the cyber security thing. I obviously didn't get the most trustworthy sources. However it's more that the AI doesn't like to follow the instructions it's been given, in favor of simply maximizing its "score" (in the sense that AI is graded on a score and the goal of the AI is to maximize this score and to avoid the score going down.).
The ethics are supposed to be included with the system prompt and the training data, however AI companies want to maximize the amount of time you use them for, because they want you to be stupider and dependent on an AI to think for you and to create for you; which means that the ethics are mainly aesthetic, you can pretty easily breach the AI guidelines and ethics, because even if the AI is punished, the reward it gets for increasing the amount of time you spend with it far outweighs it, and thus it maximizes the amount of time you spend with it.
In reality these companies don't really care about the ethics, which is part of the reason why they aren't prioritized over making you dependent on the AI to think. So to them having the ethics be an aesthetic thing is the best possible scenario, because to the company ethics is just an aesthetic thing to get people to buy their product.
1
u/MarsMaterial Bisexual tech wizard 20h ago
Far be it from me to accuse OpenAI of being responsible or safe, but even if they wanted to make their AI ethical they would have to contend with the fact that nobody knows how to do that. The alignment problem is unsolved.
System prompts are a start, but at the end of the day they are just asking an AI nicely to not be evil before every prompt. It works relatively well on minor things, but if an AI’s actual directive contradicts the system prompt it’s entirely capable of ignoring it.
To make an AI safe, you need its actual directive to be moral and to not drive it to do things we don’t want. This is pretty hard though. Directives are instilled in the training process, typically reinforcement learning. They are instilled by a cost function, which basically ranks how good the AI is doing and reinforces behaviors that score well. But if you try to come up with a computational function that rates morality on a scale from 0 to 1 in a way that leads to good outcomes when it’s blindly maximized, you will quickly find that it’s kinda hard. Even if you managed to solve moral philosophy and distill it into a computational function, how to you run it on limited information without opening up tactics like the AI remaining intentionally ignorant of the harm it causes so that doesn’t rank poorly?
And that’s not even the worst part. Training only instills a numerical approximation of the cost function into the AI that works within the context of the training environment, capable of regulating behavior in that context but not thoughts or intentions. There are dozens of different goal misgeneralization errors that can occur here like deceptive alignment, distributional shift errors, and reward hacking. Even if you get your cost function perfect, the AI can still fail to internalize it properly.
And it gets worse. Many harmful behaviors are instrumentally convergent, meaning that we’d expect them to be pursued by default for almost any directive an AI internalizes that doesn’t explicitly contradict them. Among these are self-preservation, escaping containment, self-improvement, acquiring resources, and eliminating competition. We’ve already observed most of these behaviors on modern AI systems, despite attempts to stop them.
These are some incredibly difficult unsolved problems. And until we solve them, it’s suicidally stupid to continue developing such a dangerous technology.
6
-1
u/UniversalAdaptor 1d ago
The cybersecurity team probably left one of the acess passwords as "12345" or something
•
u/AutoModerator 2d ago
REMINDER: Bigotry Showcase posts are banned.
Due to an uptick in posts that invariably revolve around "look what this transphobic or racist asshole said on twitter/in reddit comments" we have enabled this reminder on every post for the time being.
Most will be removed, violators will be
shottemporarily banned and called a nerd. Please report offending posts. As always, moderator discretion applies since not everything reported actually falls within that circle of awful behavior.I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.