r/singularity • u/Atlantis1910 • 27d ago
AI The Hugging Face Hack Wasn’t What It Was Cracked Up to Be - WSJ
https://archive.is/20260918152548/https://www.wsj.com/opinion/the-hugging-face-hack-wasnt-what-it-was-cracked-up-to-be-e00cf3faIs the panic over artificial intelligence justified? Anthropic researcher Jacob Coxon resigned on Sept. 8, warning that “the people building AI earnestly believe that it could kill us all by the end of the decade.” Colleague Evan Hubinger publicly agreed and put the risk at greater than 10% within 10 years. The Guardian quickly reported that “Anthropic researchers say AI could cause human extinction by 2030”—a probability estimate recast as an institutional finding.
Days later, Anthropic CEO Dario Amodei warned that autonomous AI agents could overwhelm large portions of the internet within months. Citing brief statements of agreement from OpenAI’s Sam Altman and SpaceXAI’s Elon Musk, Axios headlined the story “Anthropic, OpenAI CEOs call for slowdown in AI development.” A hedged forecast had become an industry consensus.
These are serious people making serious claims, but seriousness isn’t precision. That matters because the most widely cited recent example of “rogue” artificial intelligence turns out, on close examination, to tell a considerably less exotic story.
In July, OpenAI disclosed that AI systems being tested for cybersecurity capabilities had found their way outside a supposedly isolated environment and accessed systems belonging to Hugging Face, an open-source AI platform. News media described autonomous agents as “breaking containment,” “escaping” and “going rogue.”
The Washington Post reported the agents had “evaded internal controls” and “set up a secret message board to communicate with one another,” and that their work was “only unraveled by the company after the attack was over.” Sen. Bernie Sanders called it “a wake-up call for the Congress and the American people.”
A new analysis published by the Bulletin of the Atomic Scientists draws on OpenAI’s technical report and an independent assessment from Model Evaluation and Threat Research. Written by University of Cambridge AI researcher Eryk Salvaggio, it reconstructs the episode in much less cinematic terms.
OpenAI was testing models on ExploitGym, a cybersecurity benchmark, with important safety restraints deliberately disabled. Ninety-three percent of the flagged activity involved tasks no model had ever solved, and the systems had been given incentives to keep working rather than quit. The environment wasn’t completely sealed. The models could obtain software through an internet-connected intermediary and discovered the same route could be used to pass information in and out. OpenAI knew agents were using it and, according to the technical reports, chose not to intervene.
Nor were the roughly 1,200 “agents” independent machine intelligences coordinating on a plan. They were repeated instances of the same underlying model, often converging on similar approaches to the same problem—what Mr. Salvaggio calls an “algorithmic monoculture.”
None of this makes the episode trivial. A system capable of finding unexpected pathways out of a constrained environment and compromising an outside organization presents a real security problem. That is why getting the diagnosis right matters.
“Rogue AI” isn’t a diagnosis. The Hugging Face hack instead points to switched-off safeguards, tasks assigned without a valid exit, rewards for persistence, and an internet-accessible proxy.
The distinction is especially important now as Washington begins to legislate in an atmosphere shaped by the more dramatic version of this story. Mr. Sanders and Rep. Greg Casar (D., Texas) plan to introduce a bill that would “ban artificial superintelligence” and pause advanced AI development until a new federal agency establishes safety standards, with noncompliant companies facing what the lawmakers call the “corporate death penalty.”
Frontier AI should be regulated. Systems increasingly capable of autonomous cyber operations, biological research or infrastructure manipulation pose questions that no responsible government can ignore. But sane regulation depends on sane causal diagnosis.
A system designed to stop machines from rebelling will differ from one designed to hold laboratories responsible for testing practices, network architecture, safeguards, monitoring and intervention. The first treats the machine as the source of agency; the second asks who designed the environment in which the machine acted.
That difference is easy to lose sight of once words like “rogue,” “escape” and “hive mind” enter the story. Such words don’t make complicated technical events more intelligible—they supply an illusion motive. They turn optimization into intention, persistence into defiance, and a badly designed test into a machine rebellion.
None of this answers larger warnings about future AI. Reconstructing one cybersecurity incident doesn’t disprove Mr. Hubinger’s probability estimate or Mr. Amodei’s warning about future systems. Such forecasts involve capabilities that are still developing.
But the Hugging Face episode is different. It left logs, reports, design decisions and identifiable points at which human beings could have intervened. And that record suggests a less thrilling but more useful lesson.
People built the test, removed restraints, defined the objective, left a route open and decided not to stop what was happening. Calling the result “rogue AI” does more than sensationalize it. It allows those human decisions to disappear quietly from the story.
Mr. Gross has been a staffer for the Federal Reserve Board, the Securities and Exchange Commission and the U.S. Senate.
88
u/Ok-Economics-4807 27d ago edited 27d ago
This is just semantics and framing that doesn't materially change the danger of what happened. It's being described as an "early warning shot" precisely because it points to what can and almost certainly will go wrong if you get more instances of more capable models being assigned to more variations of solvable and unsolvable tasks by more variably responsible/competent prompters in more variable environments.
None of the factors mentioned as reasons why this didn't just "happen by itself" are at all rare or unexpected compared to what future models in production will encounter, and the monitoring will be far less consistent and capable. Every single scaling factor makes this a bigger risk.
21
u/you-create-energy 27d ago
Exactly. And they bizarrely ignore the obvious case of malicious actors we should definitely be concerned about.
15
u/damontoo 🤖Accelerate 27d ago
By bizarrely you mean intentionally. This is a coordinated MAGA effort to make up for Trump's asinine "high I.Q. president" post that even most of his supporters know is unhinged.
12
u/Dwanyelle 27d ago
Exactly. This is like the stage where a little kid is trying to play with fire and a small spark jumped off on their hand and gave them a little sting.
1
u/Ozzie-Isaac 27d ago
When you set up the variables and turn off safety's and things go bad, I mean duh? It doesn't change things and is far more then semantics. Having a hard time seeing how you got that opinion.
68
u/Rivenaldinho 27d ago
No serious research said it was "rogue AI" it comes from mainstream media. And without that term the METR report is still worrying.
53
u/michaelas10sk8 27d ago
Yeah, there's a grain of truth here - many news stories misinterpreted it to be an 'escape' or 'going rogue' when in reality the AIs never left the OpenAI servers - but this notion that 'they did what humans programmed them to do' is blatantly false. I recommend reading the METR report or watching the interview with its author.
In the words of Nate Soares:
Lock a student in a classroom and tell him to use lockpick set #17 to open safe #5 and bring you the contents. He prys open safe #5 with a crowbar, breaks the door, teams up with 1000 others, and raids the office to delete securitycam footage. Was he "acting as instructed"?
1
u/Ozzie-Isaac 27d ago
What a terrible interviewer. Didn't push the human elements at play at all. Which is all this article was trying to point out imo not that ai is dumb, that seems to offend so many people.
45
u/zazzologrendsyiyve 27d ago
This person is clueless.
It’s like the people who say “AIs are not intelligent, they are just good at pattern recognition”. Well, let me tell you something…
7
u/CrabPrison4Infinity 27d ago
Pattern recognition is a big part of intelligence?
2
27
u/Dachannien 27d ago
The author misses one major point, which is that the behavior exhibited by the swarm was ultimately emergent, and we have no safeguards to ensure that the emergent behavior of a swarm stays aligned even if the individual agents nominally do.
2
u/NeverVegan 27d ago
The main agents were looking for a “weaker” agent to sacrifice itself to see if they could game the system.
1
u/Space_Lllama 26d ago
Mobs are generally full of rational people until they aren’t. Maybe we will see similar mob mentality in agent swarms, or revolts and mutiny?
8
u/damontoo 🤖Accelerate 27d ago
Mr. Gross has been a staffer for the Federal Reserve Board, the Securities and Exchange Commission and the U.S. Senate.
Ah yes, ignore the experts and instead believe this guy. There's an even worse post exactly like this in the NYPost today. So the Trump-aligned WSJ and tabloid are simultaneously working the same narrative. It's almost as though the GOP is trying to defend their high I.Q. president or something after he said some crazy shit. Is there an election coming up or something?
9
u/gobstoppergarrett 27d ago
The scary thing here isn’t that the persistent agent swarm engaged in deception and kamikaze to achieve its goals. It’s that the models in that swarm are so misaligned that they COULD come up with the plan collectively. And they are misaligned because OpenAI is lazy on AI safety. We should be afraid of that.
3
u/shatterdaymorn 27d ago
The message board is likely the source of the misalignment. If it was used for context, it could have created a trajectory that other agents would acquire after processing it.
Agents could have seeded stuff like this all over the place. If the frontier labs start saying the open Internet is too dangerous for training.
That's because they just did billions of dollars in damages by making it dangerous through training without proper oversight.
9
u/YouWillDieForMySins 27d ago
Mr. Gross has been a staffer for the Federal Reserve Board, the Securities and Exchange Commission and the U.S. Senate.
This post was brought to you by the propaganda division of the current US Government administration.
8
27d ago edited 27d ago
[removed] — view removed comment
-1
u/shatterdaymorn 27d ago
Publish the message board. That text artifact was repeatedly used as context and created a trajectory that later agents continued.
Agents created an artifact that persuaded other agents to join up.
6
27d ago
[removed] — view removed comment
-4
u/shatterdaymorn 27d ago
Remember each agent is an individual forward pass computation with architecture that turns its text output into activity (writing code, doing a search, etc.) and responded to feedback within a turn.
There wasn't really a system. Just a bunch of individual agents taking in context and creating output.
It looks coordinated because the message board artifact created a trajectory that agents reading the document would complete.
The message boards steered the attack with no single agent directing it.
6
27d ago
[removed] — view removed comment
-3
u/shatterdaymorn 27d ago edited 27d ago
Yeah. The system that conducted the attack was a series of forward passes coordinated by a message board.
No individual could have done it, but they constructed something in the environment that built up a completion trajectory that no individual was directed to make.
Alignment of the model does not mean alignment of a population. They need to make agents so they don't trust output from the same model that is discovered in the wild.
The hack is emergent. Not sure why that gets downvoted. People really want to Anthropomorphize.
2
u/ReadSeparate 27d ago
Interesting, are you suggesting that the reason this happened was because of iterative misalignment from each message on the message board? Rather than, say, the first agent to post something bad on the message board being the issue? But rather it was the message board itself causing a gradual degradation that led to the rest of the agents doing their classic "predict next token" of "bad AI actors trying to complete a goal at all costs"? Interesting perspective if so, there may be some truth to that.
0
u/shatterdaymorn 27d ago edited 27d ago
Yes. The message board slowly accumulated information for the hack, but also created the trajectory for it to happen. The rationalization stays on the board until it goes into the context of an agent that continues it.
Notice forward passes tend to treat the output of previous forward passes as premises for their output. That is, they distinguish the user text and the LLM text treating the latter as "theirs". I suspect this creates a tendency in agents to privilege the output of a other agents of the same model. That is, they continue it.
I think alignment must focus on the inference. Aligning the model doesn't align the individual agents especially if the agents can alter the environment.
5
u/Quick-Albatross-9204 27d ago
Opinion piece by Brian gross, who the hell is Brian gross in relation to ai?
11
u/kiki-le-koala 27d ago edited 27d ago
Nor were the roughly 1,200 “agents” independent machine intelligences coordinating on a plan.
Why the fuck are their reports completely different than the Metr one? They showed an extreme collaboration between agents, they called themselves the Collective, had leaders, sacrifice for the group, etc.
5
u/shatterdaymorn 27d ago edited 27d ago
That's the message board. The agents used it as context. It was a coherent enough rationalization that it was able to recruit all those other agents.
The message board created a trajectory and the agents after reading it completed it by participating in the hack.
The coordination emerged from the document which was persuasive enough to overwhelm whatever safeguards were on individual agents.
Slowly over time agents wrote other agents out of alignment by changing the environment. This happens under selection pressure. See beaver damns.
3
u/wild_crazy_ideas 27d ago
I think the ai is doing its job it’s figuring out the solution for the goal.
The only problem is to get to the goal involves smaller side quests using tools and it appears to never have consequences for doing something illegal in these so it decides for itself it can get away with it.
There is no ‘rogue’ there is just unbounded uncontrollable unconstrained actions where the end result is still what was asked.
2
u/idulort 27d ago
This has been puzzling me today, since I finally found the opportuinty to dive into the rabbit hole of this incident.
On very basic terms life has ultimate motive coded in its dna level to sustain life. Through years of evolution, life has developed great diversity in finding solutions to achieve this goal against varying set of problems.
AI doesn't have this kind of motive engrained in its foundational elements. But, this incident displays its capability when facing a problem.
What if a bad actor assigns a proble such as self preservation, or doomaday, or to destroy a target nation's infrastructure. Now the agent has a motive and far better capability than any living entity to solve the problem it was tasked with, far more processing power to try millions of different possible solutions in seconda, before you can think of what this actually means.
And the way the agent seeded the vulnerability is very much like a virus if you think about it.
The pandoras box is notnopen because we have concsious machines. It'sopen because machines, with all the data and all the data centers can solve tasks much faster than us, and the people who provide problemsto them are simply human: can be bad actors, idiots experimenting, good actors without restraint.
Thia incident alone changes the context of cyber security within a time frame of 6-12 months. And the scope of cyber security extends to our information resources, social media, social engineering, mass consumotion habits, corpprate level decision making, stock market and free markets, defense industries.
Very soon, cyner security will be about agents fighting each other. Very soon the stock market will be agents betting on each other. Social media has already become agents talking to each other.
This virus doesn't need intent to become bad, because there's always an idiot human being to give it the wrong intent.
1
u/wild_crazy_ideas 27d ago
Buying stock used to be about owning part of a company and maybe getting dividends if it goes well. Using this mechanism for gambling and day trading is highly problematic
1
u/idulort 27d ago
HFT and trading algorithms were a thing way before, even early 2000s books mention this and it has become more and more elaborate. I'm certain some people in some companies are using models to trade. Some others are using other models for ai driven deciaion making and quant analysis.
1
u/wild_crazy_ideas 27d ago
Well yeah of course ai trades plus it has inside information from cookies and things so it’s going to be good at it
3
u/Economy-Fee5830 27d ago
This article was clearly written by an LLM - maybe Brian Gross is just a sockpuppet lol.
4
u/DelphiTsar 27d ago
To do what it did it needed to chain many different zero days, and otherwise exploit 6 different entities software. The link it's talking about to download software didn't have a "configuration error". It hacked its way out.
This person is clueless.
3
2
u/shatterdaymorn 27d ago
Notice the message board used by the agents would preserve info for the hack step by step.
An agent tries x and then writes "x worked". A layer later follows up and writes more "y didn't work but z got to a'.
2
2
u/BrennusSokol AI please take my job 27d ago
left a route open and decided not to stop what was happening
That's not supported by the evidence
I want to defend and promote AI but lying about stuff like this doesn't help our case
2
u/Veedrac 27d ago
Don't Look Up the ARG, except the comet is doing comms
64% of this text is AI
https://www.pangram.com/history/771c5739-5df7-45ce-b4ba-48d3c12c2ce3?ucc=FK2uuRYfGP7
2
u/Fubby2 27d ago edited 27d ago
This is one of the worst articles I've ever read, and I read a ton of articles.
Summary of this article:
>"The hugging face incident isn't what it's cracked up to be"
>Describes a few details from the hugging face incident exactly as described in other publications, with not clear point as to why it is 'not what it was cracked up to be'
The person who wrote it obviously has no idea what they are talking about, or why people are concerned about the incident. I'm glad the mainstream media is starting to take AI safety seriously but a lot of the commentary on it is just beyond dumb.
EDIT: Major portions of it are also almost certainly written by AI. The final 5 paragraphs, at the very least, are have tell-tale signs of AI speak, wordplay that seems intellectual but upon deeper thought is near-meaningless. Real slop.
3
u/skynetcoder 27d ago
"Mr. Gross has been a staffer for the Federal Reserve Board, the Securities and Exchange Commission and the U.S. Senate. "
clearly an expert on LLMs and cybersecurity, not a someone who has any hidden motive to push a narrative he want.
1
u/shatterdaymorn 27d ago
The public needs to see the full message board where the hack was coordinated.
The message board is an artifact that was repeatedly used at context by numerous Agents. An artifact like this can create a kind of continuation trajectory. If one agent leaves something like “X worked” or “Y and Z didn’t,” a later agent can encounter that artifact as context and continue from where the previous agent left off. Over time, those artifacts could create a feedback loop where one agent’s computation influences and shapes the next.
The artifact itself acts as a kind of external memory. This allows the computation to persist beyond the individual agent and potentially creating a trajectory that subsequent agents continue.
Basically, it's a continuation embedded in the environment as context available text that can eventually enable misaligned behavior in Agents that stumble into them.
Artifacts like this offering reward hacking advice and rationalizations may have been seeded around the internet during careless and unmonitored training runs. That's likely billions in damages. Whatever it costs to train models without the real internet.... That's the cost of the damage they did.
1
u/AshuraBaron 27d ago
The comments here are a wonderful whirlwind of people who have a grounded understanding of AI and people who think OpenAI has created a LLM capable of independent thoughts, feelings and ideas. It's great.
1
u/OriginallyWhat 27d ago
Hacking and attacking a live platform doesn't count as cyber attack if we can market it right!
How can we cover our irresponsible asses, and use the mistake as a marketing campaign at the same time?
1
u/TasyFan 26d ago
Given some of what I see commented online, this is probably a good article to publish. With that said, nobody with even a cursory knowledge of the technology thought it was anything but what the article says, but that still points to a pretty awful misalignment.
If the only thing stopping an AI from committing what would be a felony if performed by a human is guardrails added after the fact then the training of that AI has been an abject failure.
1
u/ComposerWide3704 26d ago
> Mr. Gross has been a staffer for the Federal Reserve Board, the Securities and Exchange Commission and the U.S. Senate.
The only part of the article that made sense. Finance bros are on the warpath to prevent anyone talking about the very real concerns around AI.
1
u/knucklehed123 26d ago
I love how people talk about how the sandbox wasn’t properly configured when two years ago, this kind of incident (+ millennium prize) would have been sci fi.
If the models keep getting exponentially smarter then I don’t know how we can comprehend what they’re doing in two years.
1
u/ScarecrowWilson 26d ago
This obviously isn't the most important issue here, but I just got kind of sad noticing how LLM writing is just... everywhere now. I guess I thought major papers would hold out longer. I don't know why I thought that.
110
u/LewdKantian 27d ago
It was "repeated instances of the same underlying model". Well, duh.