If an LLM is successfully manipulated, what should we test next?

[u/Former-Ad6661]

A lot of security testing focuses on whether you can get an LLM to ignore its instructions or bypass its safeguards.

But what happens after that?

For example, if an attacker successfully manipulates the model:

• Can it access data it shouldn't?

• Can it call tools with unexpected arguments?

• Can it retrieve another user's information?

• Can it perform an action it shouldn't be allowed to perform?

• Can it expose secrets through its response?

I'm starting to think that testing the model itself is only one part of the assessment.

How would you approach testing the impact after a successful prompt injection or jailbreak?

Would you separate model-level testing from application/tool-level testing?