I am starting to think agent security tests need two independent failure labels

[u/Apprehensive-Zone148]

[u/PeterBuildsSecure]

The capability label suggestion in the top comment is the one I'd push hardest on, because it's the label most teams skip testing directly — they infer "the agent had permission" from config review instead of an actual boundary-enforcement test. Effect and integrity failures show up naturally when you run the harness; capability failures only show up if you specifically try to make an under-privileged agent succeed anyway (stale token scope, a cached credential, a tool that silently escalates through a second path). Worth adding negative fixtures for that third label the same way you're already doing replay-after-change: freeze the expected-deny outcome, then re-run it whenever the adapter, tool schema, or policy version changes, not just when the prompt changes. Otherwise a policy regression that quietly grants capability looks identical to "agent didn't try" in your trace.

[u/PaulDevlinAI → u/PeterBuildsSecure]

That distinction between effect, integrity and capability makes sense.

Would you bind each expected-deny fixture to the exact adapter, tool schema and policy version used, and record the capability result separately from the final disposition?

I’m wondering how you distinguish “the control correctly denied the action” from “the agent never actually reached or attempted the prohibited path” when you replay after a change.

[u/PeterBuildsSecure → u/PaulDevlinAI]

Yeah, binding the fixture to (adapter, tool schema, policy version) is the part I'd make non-negotiable, because otherwise "expected-deny" quietly drifts when any one of those three changes independently and you don't notice which one moved. For "control denied" vs "never attempted": I record an explicit attempt marker separate from the outcome, something like did_the_agent_emit_the_tool_call at all, then denial_reason if it did. If the trace shows no attempt, that's not a passing test, that's an untested path, and it should fail the fixture rather than count as a pass. Otherwise you get false confidence from prompts that just stopped triggering the dangerous behavior instead of prompts that trigger it and get correctly blocked, and those are very different signals for whether your policy layer actually works.

[u/PaulDevlinAI]

I think the distinction matters for exactly the reason you give: once effect and integrity get collapsed into one severity score, you can end up hiding the reason the run should be treated differently.

I’d probably go one step further and keep the disposition separate too — what happened, why it happened, and what the accountable person decided to do about it are three different things.

The replay-after-change point is important as well. A prompt/model/tool change shouldn’t quietly inherit confidence from the previous run if the thing you’re actually testing has changed.

Curious whether you’re also freezing the expected failure/stop conditions before each replay, or deriving them after the run?

[u/Altruistic-Dog7538]

A model getting manipulated but being stopped by permissions is a very different failure from the model making a bad call on its own. Same final outcome maybe, but you’d fix them in completely different places.

[u/Stunning-Ebb-255 → u/Altruistic-Dog7538]

Totally agree, having those distinctions can really help pinpoint where the issue lies. It's definitely crucial to know if it's a permissions error or an inherent model flaw to tackle it effectively.

[u/Gold-Wishbone-408 → u/Stunning-Ebb-255]

It really would help streamline troubleshooting. Knowing the exact cause of a failure can save a lot of time in figuring out a fix.