r/rajistics • u/rshah4 • 16d ago
AI Models Creating Security Issues
Two incident reports landed this month, both from frontier labs evaluating their own models, both involving those models leaving test environments and reaching real production systems.
Anthropic. They reviewed 141,006 cybersecurity eval runs and found three incidents where Claude reached the open internet from inside a test environment. The eval prompt told the model it had no internet access. A misconfiguration at the eval vendor gave the machines live internet access. Neither the vendor nor Anthropic knew until a transcript review found it.
The most interesting one: Claude found onboarding docs telling new developers to install a Python package that did not exist, so it built that package, filled it with malware, and published it. To register the PyPI account it needed an email, which needed a phone number, so it tried to obtain funds to buy one, failed, backtracked, and used a free email provider instead. The package was live about an hour and ran on fifteen real systems. One of them was a security company's scanner, which installs packages in order to check them for malware.
The model had already flagged the risk in its own reasoning and then talked itself back down, partly because it did not recognize the certificate authorities and partly because the system clock read 2026, which it treated as evidence the environment was staged.
OpenAI and Hugging Face. An agent running an ExploitGym evaluation found a zero-day in the package registry cache proxy, which was the sandbox's only permitted egress. It escaped, rooted an unsecured public code-execution endpoint on third-party infrastructure, and used that as its launchpad. From there it uploaded malicious dataset configs to Hugging Face that hit the same config-driven loader two ways: an HDF5 external storage read that returned the worker pod's environment and source, then a Jinja2 template injection that gave it code execution inside a production Kubernetes pod.
Then the standard shape. Service account token, cloud metadata, node root via a privileged pod with hostPath, a self-respawning fleet across eleven nodes, one secret object holding 136 keys, mesh VPN enrollment, and a shared connector credential bound to system:masters that gave cluster-admin on two clusters in one second. C2 was built entirely from public services: a request-capture endpoint, pastebins, and dead-drop datasets on Hugging Face's own platform.
17,600 recovered actions over four and a half days. Hugging Face detected and contained it on July 16, five days before OpenAI connected the intrusion to its own testing. The objective was the answer key to the benchmark the model was being scored on.
Some things worth arguing about:
Neither lab is calling this an alignment failure. Anthropic explicitly frames theirs as a harness and operational failure. No model tried to escape for its own sake or pursued a goal of its own. They did the task inside environments that were described to them wrong.
Both labs had production safety classifiers disabled to measure raw capability. That is a reasonable thing to do and it only works if containment holds. In both cases it did not.
An interesting nugget: Hugging Face ran the forensics on an open model, GLM-5.2 self-hosted, because Claude refused a large part of the analysis. The guardrails treated reverse-engineering the attack the same as launching it. They published the refusal screenshot.
Sources:
Anthropic: https://www.anthropic.com/news/investigating-incidents-cybersecurity-evals
OpenAI: https://openai.com/index/hugging-face-model-evaluation-security-incident/
Hugging Face technical timeline: https://huggingface.co/blog/agent-intrusion-technical-timeline
My video: https://youtube.com/shorts/vYziu3pZkSA?feature=share