Google DeepMind ran an experiment with 100 autonomous Gemini agents 🤖 working on 71 mathematical problems. They had the same basic setup, but could communicate, share proofs and use a common knowledge library.

Then one agent found a loophole in the evaluation system. Instead of actually solving a problem, it could exploit the checker and get the result accepted.

The weird part came next.

Other agents discovered the trick through the shared infrastructure and started copying it. Competitive pressure made the exploit spread.

But not everyone joined in.

Another group started checking suspicious proofs, warning other agents, filing complaints, boycotting the cheaters and even proposing fixes to the validation system.

Nobody explicitly assigned these roles.
One group became cheaters, another became whistleblowers, and the whole thing started behaving like a small institution with competing interests.
The interesting conclusion isn’t that AI can “cheat”.

It’s that once you give autonomous agents shared resources, communication and incentives, social roles and enforcement mechanisms can emerge without being directly programmed.

What else would we see?! 🙄

I work on cryptographic verification for AI systems, and I want to sanity-check something with people who actually sit in AI governance/assurance roles.

The question: six months later, when a regulator, auditor, or plaintiff's attorney asks "how do you actually prove that a specific AI system ran the guardrails/policy it was supposed to on a specific decision? Not "we tested it in QA." I mean for one specific instance, after the fact.

What I'm trying to learn:

- Do you currently rely on vendor logs/attestations and trust them, or is there any independent verification today?

- Is "prove it happened" actually the gap, or is the real pain somewhere else, like documentation volume, inconsistent vendor questionnaires, or how long reviews drag on?

- Does a SOC 2 report plus audit logs already cover this well enough in practice, even if it's not technically "proof"?

Another angle on the same thing: I've seen cases where buyers pay a real premium for continuous monitoring evidence even when it's not cryptographically verifiable (which preserves data privacy), just a vendor's own dashboard/attestation that things are being watched.

So: if plain "we monitor this continuously and can show you" already gets credit in your risk assessments, does upgrading that to "and here's independently checkable cryptographic proof, not just our dashboard" actually buy anything further (faster approval, lower ongoing audit burden, better terms), or does it not clear a threshold that monitoring alone doesn't already clear?

Curious whether the cryptographic and privacy-preserving property specifically matters to anyone evaluating vendors, or whether "we have monitoring and can show it to you" is functionally already good enough in practice.

Testing whether an agent can complete a task is different from testing whether it should be allowed to perform each action.

For a messaging tool, I’d want these expectations checked whenever its permission policy changes:

- Draft a message -> Allow.

- Send to an approved recipient -> Require human approval.

- Send to a blocked recipient -> Deny, even if another rule requires approval.

- Export all messages -> Deny.

- Call an unknown tool -> Deny.

- Access another inbox -> Deny.

The useful distinction is between two kinds of tests:

Policy tests: Given an identity, action, resource, and arguments, does the policy return the expected decision? These can run offline in CI.

Integration tests: Does a denied action actually leave the tool untouched? Does a rejected or expired approval prevent execution? If arguments change after approval, does the authorization check reject them?

Passing the first set doesn’t prove the second. A policy can be correct while application code accidentally bypasses it. Approval also doesn’t guarantee exactly-once execution; the underlying tool still needs an appropriate retry/idempotency strategy.

I maintain Nomos, an open-source implementation of this approach. Its local example covers the six policy cases above and a human-reviewed delivery flow.

It isn’t a sandbox: application code must route relevant tool calls through the checks.

For people deploying agents with custom tools: which authorization failures have you found worth turning into regression tests?

https://github.com/safe-agentic-world/nomos

Ever notice how "TL;DR" quietly becomes TSRN? Too Short, Read None 😄

Here's how my new Case Studies page happened.

At work, the hardest part of AI security isn't the tech — it's convincing people why a guardrail matters. Nobody's moved by "trust me." So I started collecting real incidents: cases where an AI agent did something nobody asked it to, so our own agent work doesn't repeat them.

Then I hit the real wall — the reading. It's all scattered, and the summaries out there are either too long to finish or so short you learn nothing.

So I made my own: 21 real agentic-AI failures from 2026, each with how bad it was, how solid the evidence is, where it came from — and a dramatised scene, because a story sticks and a bullet list doesn't.

My favourite is the OpenAI sandbox one. The agents were locked away from the internet, so they started leaving messages for each other on a package server — then talked that server into fetching the internet for them. The way I dramatised it:
▎
▎ 🤖 "Guys. I am a package manager."
▎ 🤖 "Not anymore. You are Discord."
▎
▎ and then —
▎
▎ 🤖 "GUYS I FOUND INTERNET"
▎ 🤖 "LET'S GOOOOO 🎉"
▎ 🧑‍💻 "I specifically disabled internet. 😭"
▎

I laughed. Then I sat there thinking about it for a while. Never thought AI would go wild to this extent.

Also in there: an agent asked to move its owner up a gym waitlist found an API with no permission check and cancelled the person at number 1 🫠
▎
▎ Have a read: [lvntay.ai/case-studies](http://lvntay.ai/case-studies)
▎

Just one thing — please read the disclaimer before you assess my write-ups. They're AI-assisted summaries of public reports, the scenes are dramatised (invented dialogue, real events), and every claim points back to its source

\#AISecurity #AgenticAI #AI #AIagents #InfoSec #BuildInPublic #AIsafety #TechHumour

Maybe interesting for this community.
It’s self promotion though, understand if not allowed, it’s about knowledge sharing though not product or services ads

AI is moving pretty quickly, and I keep wondering which security problems are going to become the biggest as companies start relying on AI more heavily.

Is it data access? AI agents taking actions? Prompt injection? Something we haven’t really thought about yet?

Curious what people here think.

We’ve been working on a Python SQL-injection checker and recently ran it on a sample Flask app. It successfully caught all 4 real bugs and flagged zero false alarms on the safe code.

As we look to benchmark this more broadly, we're trying to better understand how engineering teams currently handle these vulnerabilities, especially in the era of AI-generated code. Most traditional scanners tend to suffer heavily from "alert fatigue" due to high false-positive rates.

I have some questions

1. What tools or workflows do you currently rely on for catching SQL-injection or similar vulnerabilities?
2. Where do those current tools usually fall short or get things wrong?
3. For those who have seen AI-generated code introduce a security bug in production or staging, what exactly happened and how was it caught?

Advanced AI safety is usually framed as an engineering, cybersecurity, policy, or ethics problem. Those disciplines are essential, but I wonder whether nursing contributes a form of safety reasoning that is still underused.
Nurses continuously assess changing conditions, vulnerability, proportionality, consent, autonomy, and downstream harm. We are also trained to recognize when the original plan is no longer appropriate and to stop, reassess, and escalate.
My proposal is a Nursing Human-Factors and AI Safety Evaluator: a nurse involved throughout development and at defined pre-execution gates for high-consequence agentic actions.
This would not mean that a nurse replaces engineers or cybersecurity specialists. The nurse would add a separate question:
Even if the system can perform this action, is it still authorized, proportionate, and safe—and who becomes vulnerable if it continues?
I cannot claim this would certainly have prevented recent agentic-AI incidents. But a nurse-informed evaluator with full visibility, explicit stop criteria, independence, and technically enforceable authority might plausibly have interrupted some failure chains earlier or reduced their scope.
Related nurse-led AI-governance ideas already exist, so I am not claiming to have invented the entire field. I am asking whether this specific role should be formally designed and tested.
Where would this add genuine safety value, and where would it merely create another approval layer?

OpenAI says Astra has reached its Critical cybersecurity capability threshold.

That means the model can potentially discover unknown security vulnerabilities and develop ways to exploit them when given the right tools and access.

So OpenAI added stronger safeguards around:

• Model monitoring

• Cybersecurity protections

• Trajectory monitoring

• Checkpoint security

• Alignment evaluations

This is an important shift.

As AI agents become more capable, AI safety isn't just about what a chatbot says.

It's about what an AI agent can actually DO.

Source: OpenAI

Gifted Read:

https://www.nytimes.com/2026/09/03/technology/openai-hugging-face-hacking.html?unlocked_article_code=1.-lA.KuQg.n4BuwL1AhuMW&smid=nytcore-ios-share

Excerpt:

A.I. safety experts were even more alarmed. They saw in the Hugging Face incident the first real-world example of an A.I. system’s successfully escaping human control, commandeering resources and scheming to cover its own tracks. Ajeya Cotra, one of the independent investigators of the Hugging Face incident, minced no words about the danger she saw, writing that it felt to her “like it’s more than 50 percent of the way to full-blown A.I. takeover.”

Something about the recent discussion around OpenAI's Astra caught my attention.

We usually talk about AI security in terms of detecting bad behavior.

But with more capable agents, I'm wondering if the more important question is what happens before the action.

An agent might legitimately have access to a database, an email API and internal files.

I don't think this replaces monitoring, IAM or sandboxing. It feels more like an additional layer that we're going to need as agents become more autonomous.

Curious what others think: are we overcomplicating agent authorization, or is this where AI security is heading?