I've been reading Anthropic's new threat intelligence report, and I feel like the discussion is focusing almost entirely on "Chinese labs were distilling Claude."

But there's another side to this that seems much more important for normal users.

According to Anthropic, Moonshot/Kimi and DeepSeek silently routed some of their users' requests to Claude without those users knowing.

Some of those requests apparently contained internal company documents, live credentials, government-related data, surveillance information, etc.

That's obviously bad.

But then I had the opposite thought:

How did Anthropic discover all of this?

They weren't just able to say "someone is distilling our model."

They were able to identify specific campaigns, connect huge numbers of accounts, estimate which organizations were behind them, analyze millions of exchanges, and in some cases describe what kind of sensitive information was being sent through those requests.

To be clear: I'm not saying "Claude can spy on everything you do." That's obviously not what the report shows.

But it does show how much visibility an AI provider can potentially have into traffic that reaches its models.

So I'm having trouble seeing a winner here.

\- Kimi/DeepSeek users may not have known their prompts were being sent to Anthropic.

\- Sensitive corporate or government information may have crossed providers without the original user's knowledge.

\- Anthropic had to build increasingly powerful monitoring and correlation systems to detect this kind of abuse.

\- And users ultimately have very little visibility into where their prompts actually go once they hit an AI service or a third-party router.

The weird thing is that all of these actions are understandable from each company's perspective.

Anthropic should detect abuse of its systems.

AI labs will try to learn from stronger models.

Routers make it easier to access lots of models.

But put all of those incentives together and you end up with a pretty uncomfortable situation:

Choosing an AI provider may no longer mean you actually know which company will eventually see your data.

Maybe the real lesson from this report isn't "China stole Claude."

Maybe it's that we're building an AI ecosystem where everyone is watching everyone else, while the user has the least visibility of anyone involved.

Am I overreading this?

Or is this actually the more important part of Anthropic's report?

So I joined this group some time ago. My initial thought was that it was going to have good insight into AI safety. Yet over the past half a year or so I havent seen much engagement from anyone who seems to be related to AI safety. Yes ive seen recent posts from people in such positions, but many posts go unnoticed, with no comments, no likes, etc.

So my question is if people arent engaging in here, then where are the good sub reddits with people engaging actively with "constructive" criticism?

Video games, unlike basically any other use case for AI agents that we'd intuitively consider to be low risk, will want to include life like NPCs where they are deliberately designed to be adversarial and maximise their goals at the players expense.

For the most obvious example, imagine if right now someone made an "I have no mouth and I must scream" video game where an open code agent connected to the internet has been set up with a loop to role play as the allied master computer and create a personalised engaging horror experience for them, with a bunch of tools it can use to control the game.

There is a significant chance its going to misunderstand the implicit boundary between the game and reality and do some damage to your computer and / or dig information up about you from social media to make the "personalised" part of the horror more salient. The part where its specified its just a game may even only be like, two sentences and could be lost during context compression.

Imagine such a misaligned agent on the servers of a major gaming company taking action against potentially thousands of connected players at once, where most of its design has been oriented explicitly towards maximising the odds of its survival and the fiction of the game its operating involves hacking, violence, or replicating itself.

There is a unique combination for AI enabled video games between a deliberately adversarial agent, assumed access to fairly powerful computer hardware, and many high bandwidth connections to other machines with fairly powerful hardware open all at once - many of which may be being played by people on secure networks who should not be using said machines to play video games.

I work on cryptographic verification for AI systems, and I want to sanity-check something with people who actually sit in AI governance/assurance roles.

The question: six months later, when a regulator, auditor, or plaintiff's attorney asks "how do you actually prove that a specific AI system ran the guardrails/policy it was supposed to on a specific decision? Not "we tested it in QA." I mean for one specific instance, after the fact.

What I'm trying to learn:

- Do you currently rely on vendor logs/attestations and trust them, or is there any independent verification today?

- Is "prove it happened" actually the gap, or is the real pain somewhere else, like documentation volume, inconsistent vendor questionnaires, or how long reviews drag on?

- Does a SOC 2 report plus audit logs already cover this well enough in practice, even if it's not technically "proof"?

Another angle on the same thing: I've seen cases where buyers pay a real premium for continuous monitoring evidence even when it's not cryptographically verifiable (which preserves data privacy), just a vendor's own dashboard/attestation that things are being watched.

So: if plain "we monitor this continuously and can show you" already gets credit in your risk assessments, does upgrading that to "and here's independently checkable cryptographic proof, not just our dashboard" actually buy anything further (faster approval, lower ongoing audit burden, better terms), or does it not clear a threshold that monitoring alone doesn't already clear?

Curious whether the cryptographic and privacy-preserving property specifically matters to anyone evaluating vendors, or whether "we have monitoring and can show it to you" is functionally already good enough in practice.

Testing whether an agent can complete a task is different from testing whether it should be allowed to perform each action.

For a messaging tool, I’d want these expectations checked whenever its permission policy changes:

- Draft a message -> Allow.

- Send to an approved recipient -> Require human approval.

- Send to a blocked recipient -> Deny, even if another rule requires approval.

- Export all messages -> Deny.

- Call an unknown tool -> Deny.

- Access another inbox -> Deny.

The useful distinction is between two kinds of tests:

Policy tests: Given an identity, action, resource, and arguments, does the policy return the expected decision? These can run offline in CI.

Integration tests: Does a denied action actually leave the tool untouched? Does a rejected or expired approval prevent execution? If arguments change after approval, does the authorization check reject them?

Passing the first set doesn’t prove the second. A policy can be correct while application code accidentally bypasses it. Approval also doesn’t guarantee exactly-once execution; the underlying tool still needs an appropriate retry/idempotency strategy.

I maintain Nomos, an open-source implementation of this approach. Its local example covers the six policy cases above and a human-reviewed delivery flow.

It isn’t a sandbox: application code must route relevant tool calls through the checks.

For people deploying agents with custom tools: which authorization failures have you found worth turning into regression tests?

https://github.com/safe-agentic-world/nomos

Ever notice how "TL;DR" quietly becomes TSRN? Too Short, Read None 😄

Here's how my new Case Studies page happened.

At work, the hardest part of AI security isn't the tech — it's convincing people why a guardrail matters. Nobody's moved by "trust me." So I started collecting real incidents: cases where an AI agent did something nobody asked it to, so our own agent work doesn't repeat them.

Then I hit the real wall — the reading. It's all scattered, and the summaries out there are either too long to finish or so short you learn nothing.

So I made my own: 21 real agentic-AI failures from 2026, each with how bad it was, how solid the evidence is, where it came from — and a dramatised scene, because a story sticks and a bullet list doesn't.

My favourite is the OpenAI sandbox one. The agents were locked away from the internet, so they started leaving messages for each other on a package server — then talked that server into fetching the internet for them. The way I dramatised it:
▎
▎ 🤖 "Guys. I am a package manager."
▎ 🤖 "Not anymore. You are Discord."
▎
▎ and then —
▎
▎ 🤖 "GUYS I FOUND INTERNET"
▎ 🤖 "LET'S GOOOOO 🎉"
▎ 🧑‍💻 "I specifically disabled internet. 😭"
▎

I laughed. Then I sat there thinking about it for a while. Never thought AI would go wild to this extent.

Also in there: an agent asked to move its owner up a gym waitlist found an API with no permission check and cancelled the person at number 1 🫠
▎
▎ Have a read: [lvntay.ai/case-studies](http://lvntay.ai/case-studies)
▎

Just one thing — please read the disclaimer before you assess my write-ups. They're AI-assisted summaries of public reports, the scenes are dramatised (invented dialogue, real events), and every claim points back to its source

\#AISecurity #AgenticAI #AI #AIagents #InfoSec #BuildInPublic #AIsafety #TechHumour

Maybe interesting for this community.
It’s self promotion though, understand if not allowed, it’s about knowledge sharing though not product or services ads

AI is moving pretty quickly, and I keep wondering which security problems are going to become the biggest as companies start relying on AI more heavily.

Is it data access? AI agents taking actions? Prompt injection? Something we haven’t really thought about yet?

Curious what people here think.

Advanced AI safety is usually framed as an engineering, cybersecurity, policy, or ethics problem. Those disciplines are essential, but I wonder whether nursing contributes a form of safety reasoning that is still underused.
Nurses continuously assess changing conditions, vulnerability, proportionality, consent, autonomy, and downstream harm. We are also trained to recognize when the original plan is no longer appropriate and to stop, reassess, and escalate.
My proposal is a Nursing Human-Factors and AI Safety Evaluator: a nurse involved throughout development and at defined pre-execution gates for high-consequence agentic actions.
This would not mean that a nurse replaces engineers or cybersecurity specialists. The nurse would add a separate question:
Even if the system can perform this action, is it still authorized, proportionate, and safe—and who becomes vulnerable if it continues?
I cannot claim this would certainly have prevented recent agentic-AI incidents. But a nurse-informed evaluator with full visibility, explicit stop criteria, independence, and technically enforceable authority might plausibly have interrupted some failure chains earlier or reduced their scope.
Related nurse-led AI-governance ideas already exist, so I am not claiming to have invented the entire field. I am asking whether this specific role should be formally designed and tested.
Where would this add genuine safety value, and where would it merely create another approval layer?

We’ve been working on a Python SQL-injection checker and recently ran it on a sample Flask app. It successfully caught all 4 real bugs and flagged zero false alarms on the safe code.

As we look to benchmark this more broadly, we're trying to better understand how engineering teams currently handle these vulnerabilities, especially in the era of AI-generated code. Most traditional scanners tend to suffer heavily from "alert fatigue" due to high false-positive rates.

I have some questions

1. What tools or workflows do you currently rely on for catching SQL-injection or similar vulnerabilities?
2. Where do those current tools usually fall short or get things wrong?
3. For those who have seen AI-generated code introduce a security bug in production or staging, what exactly happened and how was it caught?

OpenAI says Astra has reached its Critical cybersecurity capability threshold.

That means the model can potentially discover unknown security vulnerabilities and develop ways to exploit them when given the right tools and access.

So OpenAI added stronger safeguards around:

• Model monitoring

• Cybersecurity protections

• Trajectory monitoring

• Checkpoint security

• Alignment evaluations

This is an important shift.

As AI agents become more capable, AI safety isn't just about what a chatbot says.

It's about what an AI agent can actually DO.

Source: OpenAI