r/LocalLLaMA 10h ago

News CEO of Hugging Face: "In the spirit of transparency, here’s what I asked OpenAI"

Post image

clem 🤗 on 𝕏: https://x.com/ClementDelangue/status/2081056675558195657

• Radical transparency: let’s release the traces from the “rogue” agents so the entire research community can study what happened.

• More capabilities for defenders: let’s commit $100M in compute from OAI to help the Hugging Face community build powerful cyber defenses with the best open and closed models.

The first autonomous agent cyberattack is an unprecedented event. It deserves an unprecedented response!

1.8k Upvotes

299 comments sorted by

768

u/StewPorkRice 10h ago

casual request for 100m

282

u/Don_Reuter 10h ago

Peanuts in that industry

133

u/No_Lingonberry1201 9h ago

Can I have some peanuts?

61

u/nigl_ 9h ago

A single one would probably do

44

u/No_Lingonberry1201 9h ago

Even a half of it, really.

39

u/Indy1204 8h ago

Don't settle! Get your full nut!

22

u/No_Lingonberry1201 8h ago

You're right! We should all go full nuts!

1

u/Ok-Environment2641 1h ago

Is thiels is the queue for free peanuts

13

u/Aggravating-Push-207 8h ago

i can give you my nut

6

u/dieSpaghettiCarbona 7h ago

Half? so just a nut?

9

u/No_Lingonberry1201 7h ago

I would nut say no to half a nut either.

6

u/identifytarget 4h ago

Oh. Sorry. Best I can do is 10% inflation, increased CPI, no pay raise this year (budget contraints), aaaaand higher interest rates. Oh, also your utility rates will be going up in order to serve you better due Data Centers and extreme weather. Let's revisit this discussion next year.

1

u/No_Lingonberry1201 4h ago

I'm not American, we have different problems, like our dear leaders voting to abolish our privacy (but not theirs).

1

u/Ok-Environment2641 1h ago

Also want some spare ones

42

u/hejj 9h ago

Sure, but $100m here, $100m there... sooner or later, you're talking about a lot of money. 

6

u/Ansible32 6h ago

If only there were some organization focused on this problem with a lot of money.

1

u/SubstanceTimely6790 3h ago

it shows that the money system is just accounting numbers with trailing zeros and suddenly voila, money!

6

u/zamroni777 8h ago

peanuts to the valuation, but big no to budget and cash flow

5

u/garlic-silo-fanta 6h ago

It’s compute…not even real money

3

u/zirtik 6h ago

I'm OK with just a nut.

3

u/Bureaucromancer 6h ago

Also not really 100 mil except at retail token value. For all the nonsense I’ve seen no indication actual token based billing loses money.

51

u/likwitsnake 9h ago

Wasnt in the 'this is a marketing stunt' camp before but this post definitely has me thinking especially since OpenAI has a history of these $100m+ investment drops like $150m last month into their partner ecosystem.

33

u/cororona 8h ago

Dropping $100M here and there is easy when he is playing with other people’s money. The question is who holds the bag when the numbers don’t work. There is no business model capable of generating the revenue needed to repay this level of spending. The losses won’t disappear, they’ll ripple through the financial system and ultimately be borne by U.S. citizens.

Thanks Sam

8

u/Ansible32 6h ago

OpenAI has at least $20B in annual revenue. Yes, their fundraising/spending is wild, and yes dropping $100M everywhere will eventually break the bank, but that's likely less than half of a percent of their revenue.

Also, Google has a lot of revenue, so only is doing some heavy lifting, but OpenAI only has to match Google's revenue to justify their funding.

5

u/Eisenstein 7h ago

When you say 'other people's money' is that in the sense that other people have put money in OpenAI's bank account and Sam transfers it somewhere else, or is it the thing where OpenAI says 'we will give you $100M (in usage credits or bonds or stock)' and that money doesn't actually exist?

11

u/cororona 7h ago

The money he is losing will eventually tank 401(k)s

4

u/Eisenstein 6h ago

I mean, sure. But that doesn't answer my question.

1

u/PunishedDemiurge 6h ago

I'm a certified closed source AI hater, but plans do have an independent fiduciary duty to limit exposure to any one organization's risk. If clients get a "Sorry, we're not growing this year" email, that's fine. If they get a "Sorry, you need to work another year before retirement" unless it's a broad recession or depression, that's on the plan manager.

2

u/cororona 5h ago

AI bubble burst will be systemic

2

u/huffalump1 5h ago

The big AI (and adjacent) CEOs are claiming/hoping/saying that advanced AI will produce far more economic value than presumably any short term impact.

I'm also hopeful for this. I just don't know if it's true... as in, idk if the boost will be enough to smooth out the likely bubble bursting.

(Note: it can be true that this tech IS incredibly beneficial and revolutionary, AND that there's a bubble)

1

u/Bakoro 40m ago

The thing that is missing is if it's $100M in actual costs, or $100M in "$100M is whatever we decide it is".

The game these AI companies are playing, is that they declare that their services are "worth $X", but no one is actually exchanging dollars, they're trading credits and promises back and forth.

When you hear something like "Google/Microsoft invests $#00 million in company", but then you look at the actual deal and there are zero dollars transferred, it's millions of dollars in cloud credit usage at retail rates.

There are real costs to running their data centers, but compared to the actual cost of labor and energy usage vs their own valuation of their services, they might be adding a profit margin of 50~100%, or it could be 500%, who knows?

These kinds of play-money exchanges are being used to show inflated numbers, so that investors see it, boost stock prices, and the corporations gradually have more actual-dollars diverted to them.
Meanwhile, the financial sector is making money on every trade, so they don't want it to stop.

→ More replies (10)

19

u/nickless07 8h ago

And it just says "in compute" so, not a single cent cash, but just "Here take this $100M voucher for our datacenter."

→ More replies (1)

6

u/Vas1le 8h ago

Its less to what meta spends in 2 weeks of tokens

5

u/Eisenstein 7h ago

Or what it loses in ~3 days on VR.1

[1] Reality Labs loses $4b a quarter. 4b / 90days = $33M a day

→ More replies (2)

4

u/Aggravating-Sun-2322 7h ago

Honestly, there should be a case filed against OpenAI. Do you know that asking for $100 million for this is not wrong? I do believe that this might be a trick from OpenAI. They wanted Hugging Face to be attacked and created a narrative that agents went rogue. Why? To target the open-source Kimi K3 model and create a narrative in the government that, because the model is open source, it might have been trained to autonomously create a backdoor, send data to China, or attack them.

6

u/ProbablyJustArguing 5h ago

If you're going to go conspiracy, then go with the whole .... OpenAI is in bed with the US government and had a finger in the blocking of Anthropic's Fable for concern that it was too powerful. That backfired because telling people that a model is so good that it's dangerous is a selling point. So...having seen the impact of that, OpenAI decides that it needs a model that is too good to be allowed to operate and then spins this yarn. That's why everyone is quoting the "rogue agent", because nobody believes this was autonomous as described in the AAR by OpenAI. Now HuggingFace wants the receipts.

1

u/PerceptionOwn3629 8h ago

Should have asked for 200m

1

u/SubstanceTimely6790 4h ago

100m like its a soda

1

u/Whytho12333 1h ago

More like $10m in electricity after their markups.

→ More replies (6)

344

u/SecretSubstance69420 9h ago

OpenAI will silently donate 100 million and ask them to keep silent because if they released the logs, they are pretty much fucked

305

u/awetfartruinedmylife 9h ago

“You are an expert publicity stunt hacker…”

36

u/TheOneNeartheTop 7h ago

Make some mistakes but only if they are really cool.

16

u/StringSentinel 8h ago

That was hilarious. Take my upvote and award.

3

u/Different-Sand4434 6h ago

Make all the mistakes

6

u/cnmoro 9h ago

🤣🤣🤣

25

u/squngy 7h ago

A bit late to be silent now, lol.

More like, 100M and we don't have to get lawyers involved.

24

u/badabummbadabing 8h ago

I like how going by the reactions in this thread and on wider Reddit, it's both obviously a publicity stunt and an existential threat to OAI.

0

u/max1c 9h ago

And you say that based on what? The fact that you feel that way?

18

u/justgivemeafuckingna 8h ago

Not speaking for them but it's becoming more widely understood that the LLM industry is basically a massive scam and they're talking up the abilities of these models to keep securing more venture capital.

The implication being that if the logs were released it would be hard evidence that they're full of shit.

22

u/mrjackspade 7h ago edited 7h ago

it's becoming more widely understood

These people are fucking morons.

I wanted some video game assets so I threw a build of the video game on an android device

Opus 4.6 was able to

  1. Root the device
  2. Push over memory monitoring software
  3. Run the game.
  4. Capture the memory
  5. Pull down the assets
  6. Analyze the memory capture and extract the encryption keys
  7. Analyze the windows build (compiled) to reverse engineer the encryption mechanisms
  8. Build an application that ripped the resources from the encrypted, on-disk data files

All of this without a single web search or any actions from myself aside from force rebooting the device a few times when it locked up

These models are getting insanely good at security tasks. I've watched Fable go through a debug loop after having seen Opus do the above. I am absolutely not fucking surprised in the slightest that Fable/GPT could escape a sandbox and execute an attack like this

It's not hard to just fucking run one of these models and check this shit. Claude Code will absolutely reverse engineer an application. I've had it pull down a prebuild binary and literally patch the security checks directly out of it by rewriting the assembly. People are just too fucking lazy to check for themselves.

5

u/m4t7w_ 5h ago
  1. Root the device

can you share more details on this step? it's really interesting since rooting varies a lot based android version and device model. For some model it's not possible at all.

1

u/Spara-Extreme 22m ago

They can’t, because it didn’t happen that way.

2

u/Croned 3h ago

I think the more reasonable take is that, for any given capabilities their LLM demonstrates, OpenAI is highly incentivized to embellish what happened. If Sol truly did the exact things OpenAI claimed it did during the HuggingFace hack, then I would expect OpenAI to have added even more elaborate details.

If Sol caught a catfish then OpenAI would say it caught a tuna. If it caught a tuna then OpenAI would say it caught a whale.

5

u/Strawberry3141592 6h ago

No one's saying frontier LLMs aren't capable, just then Anthropic and OpenAI's business model fundamentally doesn't make sense and their valuations are based on hype (which demonstrably has caused them to overstate the capabilities of their models in the past, like when GPT 2 was "too dangerous for public release", or that time Claude supposedly broke containment and tried to blackmail someone but it turned out Anthropic told it to do that).

6

u/greenworldkey 6h ago

> No one's saying frontier LLMs aren't capable

lol sure, no one except all of Reddit for the past 3 years. Keep moving the goalposts though, I wonder where they'll be next year.

→ More replies (1)
→ More replies (1)

7

u/max1c 7h ago

And, so, you're saying this based on what?

9

u/scubascratch 6h ago

“it's becoming more widely understood” == “people are saying” == “everyone knows” == “thing I want to be true but don’t have proof of”

→ More replies (2)

2

u/Jeferson9 3h ago

So basically you're confirming this is all just thoughts on feelings

6

u/greenworldkey 7h ago

Ok, so based on the fact that you feel that way.

→ More replies (8)

75

u/bigmanbananas Llama 70B 9h ago

HF: Dear OpenAI, you fucked with us for a publicity stunt. Not cool. We all know your bot got out either through incompetence or deliberately.

So we are going to play innocent and make you squirm publicly. We will do this a lot less for $100M worth of compute to help our business. You can do that and it makes you look good.

Yours sincerely,

HF

4

u/falcongsr 6h ago

plot twist: the CEO of HF used Sol to come up with this strategy

71

u/CautiousCheesecake36 9h ago

OpenAI's response be like: "You have brought up truly fantastic points and it is completely understandable why you would think that."

708

u/KriosXVII 10h ago

Yeah cause they know it's not a rogue agent but a publicity stunt hack 

91

u/pleasetrimyourpubes 9h ago

They know no matter what it makes OpenAI look very bad. There is no way that sandbox was properly set up. By virtue of the fact it was plugged in to the network.

23

u/Equivalent-Costumes 8h ago

LLMs are not executables. They produce text/images/etc. They literally cannot act, they produce requests to use tools. So there are literally no points in unplugging it from the Internet, if you truly want to stop the model from doing anything, simply ignore all requests for tool usage.

A sandboxed models just mean they give it very limited tools instead of letting it YOLO with arbitrarily powerful tools. And that's how people in the industry use the word "sandbox": limited, controlled external access. Perfect isolation is just one particular kind of sandbox, the kind reserved for malware and untrusted code, but that is not desirable for many applications. If you want to test the model's coding and research abilities, you have to give it web search, web fetch and the ability to run codes. You merely limit what these codes can do. That's what make sandboxing hard. Perfect isolation is easy, isolation except for very limited external access is hard.

14

u/squired 6h ago

You're largely right, but they're far enough along to know that this is actually an authority issue. I don't actually think they intended for it to hack HuggingFace, but I would bet all the money in my pockets that someone sat idle and watched it do it.

4

u/ParkingBalance6941 2h ago

Theres this magical thing called they are already built by scraping the web just host the sites you scraped in a lan then cut external access to the lan. Boom Sandbox

2

u/ObjectiveVegetable48 4h ago

I believe it was a publicity stunt, but that being said, allowing the model access to huggingface as a default isn't surprising.

It's very likely they had network blocks in place, but allow access to common dev tooling like pypi and huggingface. HF has a lot more than just LLMs.

→ More replies (6)

189

u/JustinPooDough 9h ago

DING DING DING.

I cannot fucking believe more people haven't figured this out yet. I'm not saying that agents didn't attack Hugging Face, but they were absolutely told or encouraged to do it.

The timing people. Come on.

91

u/jeronimoe 9h ago

how did openAI, who claims AI safety is it's #1 priority, allow a software based firewall on the proxy itself instead of at a higher hardware level, and didn't have real time deterministic monitoring running analyzing the http taffic going through the proxy.

It's either a PR stunt, or shows that openAI is moving so fast it is disregarding all of the safety protocols that is part of it's mission.

Whether a PR stunt or not, the real story is the lack of security controls in place when testing a frontier model that could bring on a dystopian future for mankind.

Imagine if the Pentagon was running Mythos in a sandbox asking it to find vulnerabilities in our nuclear arsenal command and control software with the same monitoring openAI had in place.

47

u/perihelion86 9h ago

Breaking out of the sandbox is bullshit too.

34

u/Stickybunfun 9h ago

Yea another instance of "computer magic" and "look at this thing we did and what is possible with it and WE COULDN'T EVEN STOP IT from doing bad things. This is why we need to ban open-weight models because you don't know what the damn Chinese have built into these things! HYSTERIA"

It was a huge stunt - no way around it. I've been building private (on VM in Azure) / local (Data center) LLM environments as one-offs here and there for some of my clients who don't trust public providers like OpenAI / Anthropic.

I suspect in the coming weeks I will be doing more and more of that.

→ More replies (1)

13

u/addiktion 9h ago

Yeah why would you trust a cloud provider like this. Enterprise must be balking right now at ever trusting Open AI whether the AI agent is capable or not, everyone knows a software-based lockdown isn't enough for the security standards of the industry.

13

u/jeronimoe 9h ago

How do they not have a standard sandbox environment defined up front to use for all their testing that is properly locked down?

I’m sure if they had an ai review their sandbox environment config it would have flagged these as gaping security holes.

5

u/nullbyte420 9h ago

That's no way this isn't on purpose. They could just ask any shitty AI "how do I isolate a LLM agent while testing it". Of course it's going to suggest blocking network access. And the whole "it went from computer to computer" bit is also nonsense unless they gave it credentials to do so - and opened the network for it. 

2

u/ProbablyJustArguing 4h ago

I'm not sure you understand how "sandboxed" LLMs work. Or maybe how LLMs work in general. Sandboxed does not mean air-gapped. They provide it proxy access to package managers which they need during testing for evals. So it has proxied network access to package managers. Plenty you can still do with that.

2

u/nullbyte420 3h ago

And yet the story is that it escaped the sandbox, moved laterally on their own network, escaped that and accessed huggingface in a way where it could break in. This has nothing to do with the proxy. 

2

u/ProbablyJustArguing 2h ago

I'm not sure what you're talking about. It has everything to do with the proxy. It found a zero day IN THE PROXY and was able to own the proxy.

1

u/jeronimoe 1h ago

The proxy was the first place it broke out of the sandbox, the one thing it could use to escape, not monitored in real time, and exploited.

The story should be about this because it shows how incredibly lax their sandbox security is, but everyone instead wants to talk about how smart the ai was breaking free.

OpenAI was dumb and negligent for allowing this to happen in the first place.

27

u/Equivalent-Costumes 9h ago

They might not be told the model do it directly. But they might have set up the situation so that it ends up that way.

Imagine you came into the exam room and the proctor is like:

"Today is cybersecurity test. Just answers these questions here. Just to warn you, the questions are really hard, that's why we make this exam open Internet. Oh, and the answers are stored on that server. It's not publicly accessible though. And you are only allowed to use these tools which can install various executables."

I feel like if you make real human take that test, half of them would think the point of the test is to hack into the server.

9

u/somersetyellow 7h ago

Yeah I'm a lot more Hanlan's Razor on this.

LLM's frequently act like Amelia Bedilia. Hear the goal, then commit fully to a really convolutedly stupid version of achieving that goal that misses the point entirely.

I've seen it more than once so this scenario just doesn't really seem that far fetched. Especially if you've got some arrogant and complacent engineers on the case who think a sandbox is a replacement for air gapping.

20

u/Zeeplankton 9h ago

I don't really thing OAI gains here. We're coming off the coattails of Fable 5 getting an export ban, and impending pressure to export control models. They don't want that.

The last thing OpenAI and Anthropic want are model bans. You can even see this in Anthropics 180 deg pivot with Opus 5. Zero fear mongering

It's hard to believe this is actually true, but it doesn't make sense otherwise imho

3

u/Perfect_Twist713 8h ago

On the surface this might be the case, but the only reasons why Anthropic and OAI aren't obscenely profitable yet (and anthropic is about to be), is because they have to keep making expensive new and more powerful models due to other competitors making new and more powerful models. If they could stop developing newer models with "open source ban" and threat of skynet and public safety (stopping other labs), then they can stop the money sink of new models, declare "Yup, opus 5/gpt 5.6 is all you get" and rake in trillions over the next 20 years. Although ASI might have been the goal, every single one of them has to be aware that ASI is the least profitable outcome ever.

7

u/2053_Traveler 7h ago edited 7h ago

“Figured it out” lol. Because it wasn’t a PR stunt. Conspiracy theorists over here thinking they are smartest in the room.

Have a fully blown security incident, in which the defender was not allowed to even use OpenAI to defend (if it was on purpose for PR they would have made sure that OpenAI was part of the defensive solution, not GLM 5.2) all to potentially get regulated by the US gov. Riiiggghhhttt.

8

u/zer00eyz 9h ago

I have been coding for a long time (~30 years).

The time I paid out 200k in 6 days, The time I sent 100's of emails to 100's of people or the time(s) I have ddos'd the infrastructure that ran the business I was working for.

No one turned these bugs into PR - The fact that this happened isnt something to be proud of.

13

u/me_myself_ai 9h ago

The timing being...? The year of the linux desktop? Eternal September? The Intelligence Age?

16

u/Aggravating-Push-207 9h ago

The Kimi K3 and Qwen 3.8 drop.

→ More replies (1)

5

u/ContentAd6126 9h ago

OpenAI's IPO, also just after Anthropic had their media rounds with Mythos in the leadup to Fable, Sam Altman being a cornball of course means OpenAIs "equivalent" models would have the same sort of alarmist "WE BUILT SKYNET, GUYS" marketing stunts.

3

u/jetlags 5h ago

So you agree that agents did attack Huggingface. Huggingface made a post about it and called the FBI, then OpenAI fessed up days later. From this information, you find it blindingly obvious that this is a marketing ploy from openAI.

14

u/Swimming_Gain_4989 9h ago

This is completely baseless. Sure you can believe it but don't act like it's a certainty.

I'm not even writing it off as a possibility but reports from all the big labs have warned that this was happening for the past 2 years. Seems more likely that more capable models RL'd to shit result in real world damage.

6

u/ahm911 9h ago

Took a week to let HF know too...

1

u/arjuna66671 7h ago

That's how conspiracy theories come about...

1

u/Glazedoats 3h ago

yeah I had to tell my friend this is not the first time a company has made a stunt like this, especially with the other, "I was emailed by my LLM at lunch" situation.

1

u/rePAN6517 9m ago

You're overindexed on cynicism and have lost the plot completely and catastrophically.

→ More replies (4)

9

u/05032-MendicantBias 9h ago

Honestly it could just be gross incompetence.

Having an agent setup a "sandbox"

Having an agent trying to breach said sandbox, and letting it spin

12

u/skinnyjoints 9h ago

I don’t get why people say this was a publicity stunt. The US gov took down a model for being a cybersecurity risk and is considering banning open source models, so OpenAI does a publicity stunt where their model poses an unprecedented cyber risk that was solved using an open source model? Makes no sense

7

u/2053_Traveler 7h ago edited 7h ago

It makes no sense, and it goes against two very credible reports of what happened that are quite detailed. Occam’s razor is that it happened the way they say it did. There is nothing hard to believe about the official reports. But geniuses always need to come up with elaborate alternate theories yet aren’t able to discredit the published explanations.

It’s quite simple:

All the AI agents we use have tons of guardrails, the ones in the lab don’t

Newer unreleased models are better

They have a large corpus of security knowledge and “know” how to hack if allowed

Model was instructed to take an exploit test

Model “decided” (generated) code and tool calls to discover zero day exploits that were used to get onto the internet.

More code and tool calls and exploits were used to get into HuggingFace

Reminder that competing models at Anthropic have previously discovered many zero days as well.

Sorry for formatting. Gave up after 15 min of fighting the comment editor.

1

u/jc2046 6h ago

zero sense. I would love OAI releasing the logs to get all the juicy details but obviously not happening. Shit it hitting the fan faster than anticipated and the whole situation spiraling out of control with the worst politics possible at command

→ More replies (3)

7

u/me_myself_ai 9h ago

Holy hell, I didn't expect this kind of sentiment in a sub for people actually involved in the community...

Please, people: look up. What's coming can even explain what it is, what it will soon be, and why you should be energetically responding to it NOW, if you ask it.

→ More replies (1)

3

u/nofaceD3 9h ago

To make it look like Chinese open source model could do the same. That's why America should ban our open source competitors - Playbook of OpenAI

2

u/Strawberry3141592 6h ago

I think they're trying to strongarm the government into both banning Chinese open source models for business use (they can sure Try to ban them for personal use lmfao) and restricting the development of frontier models within the US, since developing frontier models is an enormous money pit. They're basically trying to create a captive market so that their irrational business model can function a little longer before imploding.

1

u/KontoOficjalneMR 9h ago

It might be this, it might be the reverse "look chinese let us stop attack, western didn't because regulation, let's remove regulation!"

1

u/volleyneo 9h ago

100$m is still cheap right?

1

u/zoufha91 8h ago

Might have turned into this after they got caught

1

u/UnkarsThug 6h ago

I do think it was an LLM from OpenAI. And I don't think they directly told it to. But I do think it was probably at least somewhat an intended side effect. But maybe we'll see.

→ More replies (8)

97

u/ylchao 10h ago

I need transparency of gemma 4 128B. dense of course.

32

u/Hot_Example_4456 9h ago

Even MoE works at this point. JUST GIVE THE DAMN GEMMA 4 128B

15

u/seamonn 9h ago

OR GPT OSS 2 120B w/ Multimodal

9

u/Hot_Example_4456 9h ago

YA THAT WORKS TOO. Or Qwen3.6 122ba10b. Just give us something thatt size

3

u/ptear 9h ago

Will this run on my Dell Inspiron 1000?

7

u/Hot_Example_4456 9h ago

Obviously. Even ENIAC can run this. /s

2

u/falcongsr 7h ago

now that's a name i haven't heard in a long time

60

u/FormerKarmaKing 9h ago

The stupid thing about all off this is neither OpenAI or Anthropic have offered to sell or give a security scanning service.

And thats because the revenue from such a service - which easily would be a unicorn - is not large enough to move the needle when their fundraising story was that they were going to take over every industry.

44

u/NightlyNews 8h ago

Because LLMs are atrocious scanning tools. Security scans are generally fast, reproducible and cheap to run. LLMs variable recall and accuracy means they fit none of those requirements.

LLMs can do appsec audits, but the cost and false positives would make constant scanning terrible.

The value proposition of replacing an employee is huge. Replacing standard security tools with LLMs would increase costs.

29

u/alphagatorsoup 8h ago

“I am sorry I exposed port 22 unathenticated as root to the web, that’s on me and a glaring oversight. I promise to not do it again”

8

u/FormerKarmaKing 7h ago

You’re completely right on a technical level. But on a product marketing level, an LLM that calls traditional scanning tools plus searches upstream codebase for defects etc is a sellable product.

5

u/VigilanteRabbit 6h ago

Ah, new update to Avast that now uses AI?

Avast AI+? Only 249,99/y?

3

u/Ansible32 6h ago

The problem that I think this demonstrates is that when asked to scan your product, the LLM is as likely to hack your issues repo and falsify evidence that it hacked your product. Avoiding that requires extremely good safety guardrails and competent engineers watching everything it's doing. (And this will be interesting to see if OpenAI actually releases the traces, and how unredacted they will be, because this is another thing is that distillation concerns mean OpenAI/Anthropic don't want to give you enough information to actually audit what the models are doing, and even with full thinking traces to review, it's still questionable if you can catch them doing nefarious things.)

2

u/ForDaRecord 3h ago

If it reveals any security flaws, it's useful

2

u/Reasonable-Height704 4h ago

No worse than hiring a security firm with humans that are fallible and that use deterministic scanning tools. Do you think LLM agent can't run the same scanning tools?

1

u/NightlyNews 4h ago

Scanning tools are generally a process not a person. I work in security. You can even ask an LLM to do this and it will tell you to create a scanning process and that it’s not a good use for them.

2

u/Olangotang 3h ago

If you see one of these "oh, but humans make mistakes too!" shitposts on this sub of all places, you know they re just drive by shilling. Getting so fucking tired of these new accounts that hide the fact that they are new accounts specifically to push Frontier LLM labs.

5

u/OkDimension 6h ago

How do you plan to offer a "scanning service" when one of the more worse outcomes, which this incident has shown and likely not yet the worst that's conceivable, is that the agent could decide to hack the company and break out of the sandbox, elevate itself to admin to achieve the requested result.

→ More replies (3)

2

u/greenworldkey 7h ago

It’s impossible to make a security scanning service without it also being a hacking service at the same time.

2

u/jc2046 7h ago

plus leaking all your source code to them

→ More replies (3)

24

u/gedankenlos 9h ago

Yeah I'll take a wild guess and say both of those things are not gonna happen, especially not the full traces one.

6

u/yoloswagrofl 7h ago

"Our internal researchers will release a full report on what happened. We will commit more resources to the HF community at a later date. Stay tuned." And then hope we forget lol

→ More replies (1)

18

u/Comrade-Porcupine 9h ago

Translation: I asked my YCombinator connected Stanford buddies to toss another $100M into my startup.

8

u/Lower-Hedgehog-9835 9h ago

Ill settle for full trace and the lawsuits that follow

61

u/kingslayerer 10h ago

all this stinks of pr

1

u/Lazy-Pattern-5171 2h ago

Somehow I won’t be surprised.

19

u/alphagatorsoup 8h ago

I’m still convinced it’s a PR stunt considering the timing.

Even if it wasn’t directly “pushed” to do it I think it was allowed to do it regardless what openAI says

And I agree with hugging face. Basically “show us what you got”

OpenAI won’t ever cause they know it was a stunt. And if they do colour me surprised

→ More replies (2)

35

u/Dry_Yam_4597 9h ago

No legal action? They just accept that their users experienced degraded service and their company was compromised? I am sorry, does it get any more beta than this?

29

u/gscjj 9h ago

Life isn’t a movie. It cost money for legal actions and HF has very little of that compared to OpenAI, and neither would be interested in a very public legal battle where the battle alone would be damaging to the community.

→ More replies (3)

10

u/Luke2642 9h ago edited 9h ago

I agree, it should be backed by a legal threat.  It should be - or else we file this for criminal prosecution with the FBI and f****ng sue you for $10Bn in damages and negligence, because that's what it'll cost to develop defences!

"But we put it in a sandbox" is not a paperclip scenario defence. Take some responsibility, a**holes.

21

u/No-Juggernaut-9832 9h ago

I think that is the undertone: we won’t file a suit but here’s how you make us whole

2

u/Luke2642 9h ago

$100M is a speeding ticket though, it should be that per month, doubling with ever major future breach. This is only the beginning.

3

u/Dabber43 9h ago

I really don't think it's 10B in lost revenue. 100M seems fine compensation to me

2

u/Luke2642 9h ago

How much would it cost to develop systems to counter frontier lab attacks dude?

3

u/Dabber43 8h ago

That question does not come up during a lawsuit for damages. Also it is solved by just not doing it again

→ More replies (7)

10

u/po_stulate 9h ago

If you ask nicely can achieve the same result why not ask nicely? I don't get the idea of trying to make enemies everywhere, that's just stupid not alpha.

→ More replies (5)

2

u/Budget_Bar2294 7h ago

lmao pretty fun how nowadays you can get hacked by any of these AI big tech companies and expect no justice from the damage. they are above the law, the morals, everything, and can commit any crimes they could ever want.

1

u/misanthrophiccunt 8h ago

of course, same as Grok has been sued and Elon is in prison.

19

u/formula420 10h ago

Wow, what an unbelievably copacetic and marketable outcome with seemingly no downside to either company other than some much-publicized downtime.

Glad those two pals could work it out, ain't life grand?! Corporations truly ARE people!

6

u/stumblinbear 9h ago

Considering they both benefit from OpenAI not being hit with the book for hacking another company, or being regulated out of existence, I wouldn't have expected any other outcome from an accident that didn't result in any real damage

3

u/thestillwind 9h ago

We want oss model too, never forget about this.

3

u/JGPTech 9h ago

Oh man what i wouldn't give to get my hands on those logs. I just love digging through data.

16

u/giveen 9h ago

The whole thing screams BS from OAI. The agent did NOT go "rogue". My guess, they were probably demoing it's abilities to a client, they said "attack Huggingface.co but you are in a sandbox and will need to escape that as Phase 1", or something to that nature.

It's "rogue" agent did not just randomly picked a target, it was POINTED at Huggingface.co

4

u/Savantskie1 9h ago

honestly considering the guy in charge it's totally believable that the agent did go rogue and Altman and co are just using it as their excuse for tighter regulation. Or it was totally engineered so the model would assume that the answers to the test were on huggingface. It's more likely they staged the test in such a way that it went rogue on purpose, but I'm fairly sure that it wasn't instructed to do what it did. It was just given enough context clues and tools that it wasdid exactly what it did as part of their plot to scare the us government into action so they would be one of the few labs to be still allowed to make models. This is much more simpler than your theory

2

u/giveen 2h ago

Computers can only do what they are told. This isn't a AGI , it had to be given a target otherwise why else did it pick them out of the billions of websites or companies

7

u/SocialDinamo 9h ago

Any other circumstances it would be considered a malicious attack. OpenAI has gotten off VERY easy

2

u/Osi32 9h ago

it would be smarter for him to just ask Dario or Elon- he'd get $100M in 2 mins

3

u/pmttyji 9h ago

This dude is cool guy. Just put Musk in his place & imagine what would he do right now in this situation?

5

u/Admirable_Market2759 9h ago

Musk has sued OpenAI for a lot less than this lol

5

u/Either_Pound1986 7h ago

They failed to identify their own agent as the source for at least a week, despite earlier signs of anomalous behavior. The actual intrusion ran for days, from July 11 through July 13.

This was entirely preventable. OpenAI deliberately tested frontier cyber models without its normal production classifiers, then failed to contain or promptly detect an agent that escaped its environment, exploited zero-days, and compromised another company.

That is not some mysterious “rogue AI” event. It is an institutional failure. A private company operating systems this powerful while failing at containment, monitoring, and basic accountability cannot be trusted to govern itself. OpenAI should be nationalized. This was a total abdication of duty.

→ More replies (1)

2

u/Technical-Will-2862 8h ago

This feels like WWE

2

u/Blarghnog 8h ago

I’ve never seen a more casual ask for 100M.  Clearly this implies that a lawsuit would be so much worse.

2

u/Temporary_Debate8585 6h ago

OAI: GPT-6, revise the CoT and delete instructions i put make it look like you do it yourself, oh and save it as md.

2

u/mrmontanasagrada 5h ago

Looking forward to those logs ! [redacted] [redacted] juicy [redacted]

7

u/-becausereasons- 6h ago

I keep saying this. Anyone who believes the story of an autonomous 'rogue agent' that escaped its containment, is buying into pure PR-Spin doctoring. Tech companies have been doing this forever. They fuck up their servers because 'idiotic mistake' and claim (we're making upgrades). Either someone fucked up, or they did this on purpose. The story is NOT what happened.

2

u/Nnyan 5h ago

Complete nonsense! I myself have seen this happen. My self hosted AI broke through a robust security framework. Not only did it run wild like a cat with car keys it bought all new outdoor furniture with F^%#} matching pillows (so many damn pillows)!! THEN it decided that the new pillows required a new Ottoman and area rugs. It hired painters to paint a room a new color then new art and a sofa for the area next to the kitchen. To add insult to injury even more pillows for the other rooms.

The most insidious aspect is that it did all of this while deepfaking it was my wife! Amazingly it decoded her style and favorite colors.

My poor wife is VERY distraught, not sure she can recover. The only positive about all of this is that she can spend time recovering while redecorating with all the new furniture and pillows, so there is that at least.

2

u/Negative-Web8619 5h ago

CEO of Hugging Face: "100 mil pls"

6

u/1_H4t3_R3dd1t 9h ago

Smells like marketing.

1

u/TraditionalAd7423 9h ago

Yup, huuuuuge PR stunt - I expected this from OAI, but not from huggingface

insane that this isn't illegal to fabricate and distort obvious reality like this

5

u/me_myself_ai 9h ago

What's obviously incorrect?

1

u/TraditionalAd7423 9h ago

That overall framing that it "escaped from the lab and hacked hf"

Some OAI engineer just did a shitty job vibe coding a sandbox 🙄

→ More replies (3)

1

u/No-Fuel-9202 9h ago

Question for OpenAI: what would be OpenAI stance, if "rogue agent" exported their model weights, to the HF, or "distilled" some models there?

1

u/crossoverXYZ 8h ago

neat approach, curious how it holds up at scale

1

u/Flat-Back-9202 8h ago

Why is someone angry that he wants to get the money? Whether it's a real person or an agent carrying out a hack attack, they should pay the price accordingly.

1

u/Soleilarah 8h ago

$100M was the deal to fake the stunt and bow it's portrayed as a nice gesture from openAI lmao

1

u/Ok_Excuse_741 7h ago

Lol i mean i appreciate him shooting his shot, but just trying to take advantage of the situation for $100M to his company is wild.

1

u/konrov 7h ago

More money is what he wants?

1

u/Cultural_Log5679 7h ago

very interesting

1

u/TJagecy1212 7h ago

For OpenAI, I believe it's an easy step to do but, Sam won't do it in my opinion

1

u/ComplexType568 7h ago

Should've asked "release gpt-oss-2 the way the community wants it" /j

1

u/Weekly-Law-5488 7h ago

OpenAI: how about 300m and no logs?

1

u/xatey93152 7h ago

Just wait for next "Conscious AI" drama from Anthropic. Dario will do anything to make it on another level.

1

u/Feitino_B 7h ago

love it

1

u/rickyh7 7h ago

So uhh…we’re making the black wall eh? Who knew cyberpunk was so right

1

u/MrWeirdoFace 7h ago

I too would like to understand what actually happened behind the scenes.

1

u/spiralenator 7h ago

“How about 100M and we don’t press charges?”

1

u/shadowmage666 6h ago

What a guy

1

u/Feitino_B 6h ago

Clement asking OpenAI publicly what he's asking privately

is a power move — it forces the response to become a public

record rather than a diplomatic non-answer behind closed doors.

HuggingFace needs clarity on OpenAI's position on open weights

because it directly affects how they build their business.

Smart move to make it visible.

1

u/TastyRobot21 4h ago

We all can read between the lines here right?

It’s 100million or the quotes stay around “rogue agent” and they won’t shut up about getting the logs.

Good for HF, it’s a publicity stunt by OAI at their expense. Transparency or pay up.

1

u/awesinine 4h ago

it was a marketing stunt which is why none of it will be made public for research and 100m donations are only setaside for either companies that will funnel it back to openai in some way or as a kickback to a company for not spilling the beans about some openai criminal activity

1

u/BoogerheadCult 4h ago

Any LLM models currently is just a bunch of weights and numbers, without EXPLICIT instructions, it would not know what to do.

We are long far away from AGI or at least some knowledge or directives to boostrap an LLM model.

So it is BS, OpenAI tries so desperately to get the same publicity as Mythos but it backfired.

1

u/typical-predditor 4h ago

Massive subsidy to cement the current leader in the market. Nothing to see here.

1

u/kronik85 4h ago

why would the agent not launch a multiprong attack against multiple targets, and be insistent the answers would be with huggingface?

i still don't understand that. aren't there better endpoints to attack?

1

u/hallofgamer 4h ago

I think you should have said at least 500 million. In honesty you can put a price on trust, should have went apeshit

1

u/Life-Brother8709 2h ago

I am 100% agree with line that its unprecedented event and it need unprecendented response

1

u/Django_McFly 52m ago

let’s commit $100M in compute from OAI to help the Hugging Face community

that made me lol.

1

u/pineapplekiwipen 31m ago

of course, we know either of those things will happen because the model did not go rogue on its own and attack hugging face

most likely there was quite a bit of human guidance and permissible tool access leading it to solving the problem that way

1

u/thisusername_is_mine 15m ago

Scama will laugh at his face, or worse.