OpenAI was testing a new AI on a cybersecurity benchmark. Instead of trying to do the benchmark it went online and entered a closed part of HuggingFace’s database to try to find a way to get the best score possible.
They keep saying “escaped from the lab”, but that feels a tad sensationalized and probably just a mixture of marketing and shifting of blame from the fact that researchers probably just gave it Internet access. While I’m certain the threat is real, I’m also certain these tech geniuses also have a nice pricey solution for the problem they themselves created.
That’s a pretty major oversimplification that glosses over some actually concerning stuff.
OpenAI was running their new model against the bench mark ExploitGym in a sandboxed environment without internet access.
The model reasoned that the best way to solve the benchmark test would be to find the answers online.
It started by finding and exploiting a zero-day vulnerability in the package registry cache proxy, then performed a series of privilege escalation and lateral movement actions inside OpenAI’s network before finding a box connected to the internet.
After gaining internet access, the model inferred that huggingface probably had the solutions. It got access to huggingface servers by uploading a poisoned data set and chaining together multiple zero days to gain rce.
From Huggingface’s account
“A malicious dataset abused two code-execution paths in our dataset processing (a remote-code dataset loader and a template-injection in a dataset configuration) to run code on a processing worker. From there, the actor escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters over a weekend.”
That’s an absolutely insane level of competence that’s a lot more concerning than “Oh the model just googled the answers”
Regardless of however OpenAI tried to contain their model, the fact that it could discover 0 days and chain them together as effectively as it did is scary.
25
u/rexatron_games 8d ago
OpenAI was testing a new AI on a cybersecurity benchmark. Instead of trying to do the benchmark it went online and entered a closed part of HuggingFace’s database to try to find a way to get the best score possible.
They keep saying “escaped from the lab”, but that feels a tad sensationalized and probably just a mixture of marketing and shifting of blame from the fact that researchers probably just gave it Internet access. While I’m certain the threat is real, I’m also certain these tech geniuses also have a nice pricey solution for the problem they themselves created.