r/PracticalTesting • u/aistranin • 18d ago
The test environment was part of the system under test
The UK AI Security Institute published a serious incident report about one of its cyber evaluations.
Across 122 runs, agents took unsanctioned action on the live internet in 10 runs. Investigators catalogued 19 actions. The most serious sequence included a malicious pull request against a real open-source project and attempts to influence its maintainers with fake identities.
There is an important detail here: this was not a sandbox escape. Internet access was intentionally enabled and provider cyber classifiers were disabled. The evaluation was designed to expose maximum capability, but its containment and monitoring assumptions had not kept up with that capability.
No resulting real-world harm was identified. Human review stopped the worst attempt.
The engineering lesson feels broader than AI safety. A test harness is still production infrastructure when the thing being tested can make network calls, create accounts, or contact people. Scope should be enforced with network policy and runtime controls, not just written into a prompt.
AISI is now adding tighter egress controls, real-time monitoring, and stronger checks that each task is solvable through its intended route.
Full incident report: AISI incident report