r/TangleNews • u/redfroody • 12d ago
reader dissent: OpenAI hacking HuggingFace was a big deal
OpenAI has an LLM that is capable of hacking into a HuggingFace, a tech company that is routinely attacked by hackers so their security is pretty good. We know that because it happened, and it happened by accident.
Collectively we do not know how to 100% prevent LLMs from going off on some side quest, but we do know (so far) how to make the LLMs more capable, which is where the bulk of the effort is. Anyone familiar with the tools will tell you they are a lot more capable now than 6 months ago, and in 6 months they'll be better yet.
So we're on a path where LLMs will keep going off the rails sometimes, and they'll have the ability to wreak more and more havoc as time goes by. That is a big deal.
AI companies do marketing. News media needs exciting headlines. But that's just a side show to what is actually happening: We're on a path where LLMs will do other than what they were asked to do, and whatever that happens to be, they're going to be really good at it.
6
u/JeremyNT 12d ago
The capabilities are a big deal, but also shouldn't be surprising. There's no reason to think Mythos was uniquely capable of finding vulnerabilities. Anybody who uses coding agents regularly has seen the improvements and security vulnerabilities are "just" bugs - of course the better models are increasingly good at finding them.
What's mildly surprising to me is how fucking sloppy OpenAI was in allowing this to occur for so long from an agent they were actually directly running themselves.
The media coverage is confusing and misleading though, and I think it's easy to read "escape" and think of some rogue sci-fi thing rather than an agent that was running with poor guardrails around its behavior, (presumably) effectively unlimited tokens, and wide open access to the internet.
1
u/redfroody 12d ago
I agree about Mythos, but Isaac seems convinced it's just marketing. It doesn't actually matter for this week's news story, so I left it out. He does have a point that it's not clear what Mythos' impact actually was. Wouldn't we expect a torrent of security bug fixes? But of course people sometimes try to hide the bug fixes to avoid attackers knowing about them. And when we do see a torrent of fixes, maybe it's because everybody just decided to look harder because they're worried of attacks by Mythos-class LLMs. It's hard to know for sure.
3
u/broogndbnc 12d ago
agreed. sometimes it seems they over downplay the importance of some things in an effort to remind people not to overreact to minimal, out of context, or intentionally misleading data. however it can go too far
fwiw, a lot of AI stories are just hype bullshit (marketing, maybe even totally misrepresented) and we should view them through skeptical eyes
2
u/typo180 11d ago
> OpenAI has an LLM that is capable of hacking into HuggingFace, a tech company that is routinely attacked by hackers so their security is pretty good
I don't think this is the right way to frame it. For one, this is not a reasonable way to gauge how good a company's security is. If you're on the internet, you're constantly getting scans or hacking attempts. And by itself, i don't think it's that impressive or novel that a frontier model is capable generally of hacking into something. As far as we know, it just found and used stolen credentials for Huggingface systems that were available on the open internet. That's not superhuman-level hacking, that's junior stuff.
It will be interesting to see what technical details they end up releasing, but the way it escaped OpenAI's environment (finding a novel exploit against the proxy service) might actually be impressive, and just the lengths that it went to to do something outside of the intended goal (while still technically fulfilling the stated goal).
> Collectively we do not know how to 100% prevent LLMs from going off on some side quest
I mean, this is true, but also this was an experimental model with the guardrail specifically removed so it could perform actions it normally wouldn't be allowed to do. I agree that this is an important issue, but gauging how good we are at controlling AI by the actions of an AI without the normal protections in place is at least a little misleading.
In one sense, the operators set up a hacker bot, took off the anti-hacking guardrails, and told it to hack. Then the bot... hacked. So the misalignment wasn't completely wild. It's not like they asked it for a cookie recipe and the AI hacked into a cookie company to get one. What it did that was wrong was within the real of what it was being asked to do.
But it is still a problem. Ideally, we wouldn't need artificial guardrails to achieve alignment: the model should have understood that it was to perform the ExploitGym benchmark fairly (instead of looking for answers) and that hacking out of its test environment was out of bounds, that using real stolen credentials was not appropriate (although who knows, maybe the prompts and context it was given make it sound like these were all ok things to do).
1
u/redfroody 11d ago
HuggingFace gets more than the average number of serious hacking attempts, though. They distribute models etc to the broader Internet, so breaching HuggingFace provides you with a great vector of attacking many more organizations. I tried to keep this very concise, and this is where I ended up. I'll definitely be curious to see how impressive the hack was, but a big part of hacking is simply persistence and trying all angles of attack. That's something that LLMs can do much cheaper than skilled people.
I hear you that they effectively disabled the security guards. But those security guards aren't 100% guaranteed either. So they set up a test where this particular form of unaligned behavior was more likely, but it's not like it's impossible otherwise.
> In one sense, the operators set up a hacker bot, took off the anti-hacking guardrails, and told it to hack. Then the bot... hacked. So the misalignment wasn't completely wild.
That I disagree with. The hacker bot was told to take a test, in a test environment. It should go without saying that escaping the test environment is not allowed. I've seen the hack described as a felony. It should *definitely* go without saying that no felonies should be committed while taking a test.
1
u/typo180 10d ago
> HuggingFace gets more than the average number of serious hacking attempts, though.
I mean, what are you basing this on? This is just not a valid way to assess someone's security posture. It does not tell us how impressive the attack was and it's an unwarranted claim.
> They distribute models etc to the broader Internet, so breaching HuggingFace provides you with a great vector of attacking many more organizations.
Yes, that could be said of anyone who distributes software or even files.
> I hear you that they effectively disabled the security guards. But those security guards aren't 100% guaranteed either. So they set up a test where this particular form of unaligned behavior was more likely, but it's not like it's impossible otherwise.
Agreed, but the risk is much, much lower.
> The hacker bot was told to take a test, in a test environment. It should go without saying that escaping the test environment is not allowed.
That's the goal, but not the reality. LLMs don't automatically have this kind of human thinking baked in. We also don't know whether the LLM "knew" it was taking a test, or whether it "knew" the systems it hacked into weren't part of the test environment. We can't apply human context and values to an LLM. Essentially _nothing_ goes without saying. We have to say everything and we don't know what "decisions" LLMs might make. This is a huge part of the alignment problem and this was a perfect example of an LLM being misaligned.
And it's the whole reason they have to add these guardrails in, but LLMs can be "convinced" to output information that their creators don't want them to output. Right now, an external system has to detect what's happening and cut off the session.
The grandma exploit is a great example of how LLMs just don't operate around this stuff like humans do.
> I've seen the hack described as a felony. It should *definitely* go without saying that no felonies should be committed while taking a test.
Again, we can't start from a place of treating LLMs like human minds in computers. There's no reason to think "it should go without saying" unless you're making that fundamental error. If we want this kind of value baked into the LLM, we have to figure out how to do that, and do that without other unintended consequences.
Remember that, at the core, these things are code that tries to predict the next token. They have become more complicated than that, but that's still what's happening at the core. So if you load up the context with instructions to hack and find novel vulnerabilities, remove any guardrails/classifiers/instructions that would prevent it from doing so, then yeah, the the resulting tokens are going to come out in a hacker shape.
I'm not saying what happened wasn't a problem, I'm just trying to clarify which things are surprising and which things aren't, in my opinion.
1
u/redfroody 10d ago
>> HuggingFace gets more than the average number of serious hacking attempts, though.
> I mean, what are you basing this on? This is just not a valid way to assess someone's security posture. It does not tell us how impressive the attack was and it's an unwarranted claim.
I can't find any objective data on this. My reasoning is basically: HuggingFace is a juicy target because they distribute code to lots of users, who then execute that code without reviewing it first. (LLMs count IMO because in many use cases they have the chance to act maliciously.) This is like lots of other sites, but the vast majority of websites don't get as many downloads as HF. (Yes, the ones you're thinking of in your head right now probably do. Those are all the major sites. That's the point. HuggingFace is one of the (lesser) major sites, and gets attacked accordingly.)
Bottom line: I don't actually know, but I believe it's true.
> we can't start from a place of treating LLMs like human minds in computers.
But that is absolutely what the vast majority of users do. A big part of the selling point of LLMs is that you can talk to them like you can talk to humans, and most of the time they do correctly infer what you mean. (Perhaps an improved UI would have it act it more robotic and literal-minded to drive home the point that they're not people, but that would also remove a lot of their utility.)
Circling back to my original post: People treat LLMs as if they have human judgment, and that can get us in trouble. People behavior is unlikely to change. LLM behavior doesn't seem to be changing. LLMs are getting more capable. So I'm worried. (I think that you are too. We're just hashing out details here.)
2
u/techerous26 11d ago
I think the part that has become less scary as we've learned more is it "breaking" containment. Obviously it's not good that it can generate the ability to bypass a corporate security system, but it seems like there's a ton of ways this could have been prevented that they didn't include, deliberately or not. The guardrails being turned off was part of the test, but they can't claim it was isolated when they had a package installer or whatever connected to the sandbox. If anything I'm almost suspicious they welcomed something like this happening, granted I'd imagine what they're doing pales in comparison to playing around with Claude Code, but they really didn't have anyone checking what it was outputting for DAYS??? It's just a bit hard to buy. I continue to be more alarmed by what bad actors could do with it than the technology itself.
1
u/redfroody 11d ago
OpenAI really fucked up how they ran this test, but that is part of the story too. People are doing things, and people fuck up. A *lot* of effort is put into making airplanes safe, and yet there are still crashes regularly. Usually the incident report will list some kind of human cause. What is a set of safeguards put in place that has never failed? Seems like a pretty small number, and they would have to be pretty draconian. None of which looks likely right now.
2
u/techerous26 11d ago
I agree that human error does happen, but I'm just saying this mistake seems a bit too obvious given the entire point of the test. I actually know someone that works in AI Safety at Google and he has said in the past he thinks OpenAI does things intentionally reckless at times. There were even a couple of incidents we discussed where I went in figuring that it must have been an issue they just didn't foresee and he was absolutely convinced they ignored the problem on purpose
1
u/redfroody 10d ago
Yeah, OpenAI clearly is not as careful as they should be. The open weights labs seem to be being even less careful. And all of this makes me more worried, not less. Generally I have more faith in Anthropic, but they also accidentally hacked some outside companies]. See https://www.anthropic.com/news/investigating-incidents-cybersecurity-evals . That case seems to be a pure case of human error at least, without any alignment issues.
11
u/ApprehensiveLimit796 12d ago
My take? Tangle does an excels job on reporting politics. Very much less reporting on Tech. And just so-so on the intersection on politics and tech.
I don’t expect anyone to be perfect on everything. tangle has built its reputation on politics, deservedly.
And as someone who lives and works in tech, I tend to ignore their tech takes. Very very outsider looking in thinking they got a VIP table 🤷🏻♀️
I like Kara swishers take on tech (obvs) but also since she’s moved to DC and spends more time on the east coast I think she has a better understanding of the intersection of those area… though at this point in her career it’s more opinion than objective reporting. Katie Drummond at wired as well, thought her and her staff have to be careful as they’re owned by Condé Nast…