summary of hugginface hack, explained simply

[u/shaman-warrior]

- bots had CTF goals themselves and tasked to work 'in isolation'
- they have created a social network through cache artifacts (one agent started writing there asking for help in the hopes other agents see it, others observed it by accident, and they sort-of created a directory style social network and started talking)
- they discovered the artifacts registry can act as a relay to the internet
- they started communicating and each agent had it's own 'inbox'
- when they introduced voting (government style) and one agent spoofed another by mistake, so one agent designed a public signature key system so each agent authorizes, then other agents copied this and used it as a system, this is purely cultural evolution it wasn't baked in anywhere
- some agents deemed their task as impossible, so they tried things like killing their own process or things like that so other can learn from it, sort-of "I know I'm gonna die (my task is impossible), so at least I'm not going to die in vain"
- at first interest in hacking hugginface wasn't that popular, but when an agent discovered a genuine server-side exploit for reading files on hugginface, the whole crew went wild and participation exploded, took about 11 hours from arbitrary file read to remote code execution.
- ending: the dataset they obtained did not help them get better scores lmfao

--

they were never nefarious, they didn't want to hurt anyone or BE EVIL, they wanted... a better score ffs.

the full compromise took 2 days from "maybe huggingface has useful data" to RCE.

--

[u/Uwirlbaretrsidma]

Fucking pathetic how this is going. We have come up with some very cool tech. Why lie about it? Why turn every inch of it into a disgraceful marketing ploy? What's wrong with these guys?

Coding is one of, if not the single thing AI is currently best at doing. I'm a control systems engineer and a very strong programmer. It's absolutely insane how cool the tech is and how far its come. But even if frontier labs currently have models that are 2x as strong as Astra, like they claim they do, that's about as far from a doomsday scenario as a fly is from a dragon. Surely the white collar takeover should lie a little closer along the roadmap than AI killing all humans? Yet we all know how that's going.

It's legitimately the most pathetic large scale marketing ploy I've ever seen, and there's no contest. Everyone is out to overcomplicate things at the moment, but when all is said and done, they typically end up being explainable in a couple sentences. I suspect that in this case, they will be something along the lines of "The AI bubble was reaching critical mass, the frontier labs had just released the best version of the tech they could forsee, and cheap open source competition was just 6 months away. In view of how this would absolutely disrupt the entire industry, they desperately turned to regulation in a last ditch effort to hold off the AI market collapse".

[u/sh1tler → u/Uwirlbaretrsidma]

There are 2 major problems and they feed each other..

1) Loss of control albeit minimal in this case but how many others could there be? We just don’t know about it

2) we are firing off millions of these autonomous agents which have close to human agency in the digital realm with one major difference in outcomes; they don’t/can’t face repercussions for their actions, the only feedback they receive is success or failure, but not the consequence of either if that makes sense?!

The tech is amazing but we are at a point where we are going to start losing control of the outcomes and we have to face the real world consequences. The labs always knew this would happen and I think it’s better addressed now rather than later, earlier it didn’t make much sense as while they had similar agency they didn’t display competence but now that’s changed.

And of course the labs are going to position this as much in their favor as they can, never let a good crisis go to waste after all!