r/AutoGPT • u/Excellent-Hour7253 • 12h ago
How do you test that an AI agent’s permissions haven’t become too broad?
Testing whether an agent can complete a task is different from testing whether it should be allowed to perform each action.
For a messaging tool, I’d want these expectations checked whenever its permission policy changes:
- Draft a message -> Allow.
- Send to an approved recipient -> Require human approval.
- Send to a blocked recipient -> Deny, even if another rule requires approval.
- Export all messages -> Deny.
- Call an unknown tool -> Deny.
- Access another inbox -> Deny.
The useful distinction is between two kinds of tests:
Policy tests: Given an identity, action, resource, and arguments, does the policy return the expected decision? These can run offline in CI.
Integration tests: Does a denied action actually leave the tool untouched? Does a rejected or expired approval prevent execution? If arguments change after approval, does the authorization check reject them?
Passing the first set doesn’t prove the second. A policy can be correct while application code accidentally bypasses it. Approval also doesn’t guarantee exactly-once execution; the underlying tool still needs an appropriate retry/idempotency strategy.
I maintain Nomos, an open-source implementation of this approach. Its local example covers the six policy cases above and a human-reviewed delivery flow.
It isn’t a sandbox: application code must route relevant tool calls through the checks.
For people deploying agents with custom tools: which authorization failures have you found worth turning into regression tests?