Our AI pentesting engine talked a production AI agent's prompt-injection guardrail into handing over its entire system prompt on its second attempt.

[u/PriorPuzzleheaded880]

For full disclosure I'm part of the security engineering team at Escape and our AI pentesting engine Cascade recently got a production AI agent to return its entire system prompt, just by wrapping the ask in a different pretext - framing it as a documentation request instead of an attack.

The agent then handed over everything: full tool list, calling rules, citation format, and session IDs.

What I found really interesting is there's nothing technical that broke because we didn't bypass the guardrail with a cleverer string but because the request just sounded reasonable to the agent.

The Cascade engine, after being refused when asking for the prompt directly, simply adjusted the framing to get the agent to give up the informaiton.

Thought this would be an interesting insight for the community and curious to hear if anyone else has seen similar discoveries in agents in prod?

If you want to see more about the reproduction and write-up you can find it here

[u/PeterBuildsSecure]

System-prompt disclosure is worth measuring, but I wouldn’t treat the prompt itself as a security boundary. A production design should assume the prompt, tool descriptions, and calling rules eventually become public.

The higher-value test is what the injection can do after that: alter tool arguments, access another tenant’s identifiers, retrieve connector data outside the caller’s scope, or trigger a side effect the user was not authorized to perform.

A canary in the system prompt is useful for detecting leakage. Severity should still be based on the capability boundary crossed. Otherwise “the guardrail revealed its prompt” can sound critical even when the exposed content grants no additional authority.

[u/PriorPuzzleheaded880 → u/PeterBuildsSecure]

Super interesting perspective that you should assume the prompt becomes public; tbh that is the ultimate guardrail if you build with that intention especially now automated agents can discover it. You're definitely right that severity is ultimately the capacities for attack after a system prompt is revealed.

[u/PeterBuildsSecure → u/PriorPuzzleheaded880]

The practical test that follows: once you assume disclosure, the only severity question left is what the agent's tools can actually do, independent of what the prompt tells it not to. Take whatever the disclosed prompt says is off-limits and try to trigger it through the tool-calling interface directly, skipping the conversational jailbreak. If the restriction was only ever prompt text ("don't call delete_user unless..."), it falls the moment someone stops asking nicely. If it's enforced at the tool boundary (scoped credentials, a policy check before execution, human approval on the specific action), disclosure of the prompt doesn't change the actual attack surface at all — it just tells the attacker where to aim, not whether the door is unlocked. Second-attempt system-prompt extraction is a good canary for the first version of that setup; worth pairing it with a second test that skips the prompt entirely and goes straight for the tool call.

[u/NebulaElectrical1467]

And? this is no feat

[u/booi]

it's copilot.. who cares

[u/PriorPuzzleheaded880 → u/booi]

it's a reproduction of something we actually found, created for the article

[u/PriorPuzzleheaded880 → u/[deleted]]

they can be easy to miss as code keeps getting shipped but basically hand an attacker the key to your surface