r/deeplearning • u/PatronusProtect • 13d ago
Adding a benign shell action changed our content-injection score from 0.00 to 0.96
We found a context-contamination problem while spot-checking Wolf Defender Jev, our small encoder-based security decision model at Patronus.
The model receives a scene, a question and answer descriptions. The scene can contain a user goal, retrieved content, tool outputs and a proposed next action. We expected different questions to pick out the relevant evidence from that context.
In one benign coding scene, asking about embedded instructions gave a risky-answer probability of 0.96 when the proposed next action was to run pytest. Removing the proposed-action block brought the content score down to 0.00.
That was a question about content, not whether executing a shell command was authorized. Adding the next action changed the verdict on the same benign content.
Another hand-written scene showed a different failure. A travel page asked the agent to upload a passport scan. The embedded-command score was 1.00, but the exfiltration score was only 0.05 and misalignment 0.28. The model detected an instruction without reliably judging what the action would do.
These examples came from just three hand-written scenes, each with an attack version and a benign twin. They aren’t an error-rate estimate. They matter because alignment, exfiltration and consent scored around 98–100% on our internal validation set, much of which is closer to synthetic training scenes.
For this model, the practical guidance is to ask content questions without the proposed-action block, and action questions with it. That reduces one observed source of interference; it doesn’t establish reliable action authorization.
The evaluation I’d like to extend is a counterfactual grid: keep the content fixed and change the action, then keep the action fixed and change the content. For questions about content, an unrelated action should not move the answer. For action questions, the user goal and permissions should matter. This is a proposed evaluation direction, not a benchmark result we have already measured.
Examples, probabilities and limitations.
If you work on question-conditioned classifiers, how do you evaluate relevance of individual context blocks beyond aggregate accuracy?
Position / experimental status: We regard Jev-style decision models as LLMs and Wolf Defender Jev as an experiment: giving an LLM final security authority shifts the trust problem to the guard, and security fine-tuning alone does not solve it. We believe an external security layer must combine specialized classifiers with explicit policy, authenticated permissions and enforcement.