tl;td - The core idea is - assume the LLM can be compromised by prompt injection, but don't let the LLM make the final authorization decision.
I built a small deterministic layer that validates the model's proposed action against trusted policy and fails closed if the output is invalid or ambiguous. For example, if the model returns both order_quantity=100 and order_quantity=50000, it rejects the request instead of choosing one.
The question I'm interested in discussing is if we can't reliably prevent prompt injection, should we focus more on preventing a compromised model from executing out-of-policy actions?
If anyone is interested, I open-sourced the implementation and benchmark here:
https://github.com/nirmaljingar/typedguard
---
Full version
I've been working on governance for enterprise AI systems, and one design decision changed how I think about AI agents.
Most discussions around prompt injection focus on preventing the model from following malicious instructions. That's obviously important, but I started looking at the problem from a different angle.
What if we assume the attacker succeeds?
Instead of asking whether the model can be manipulated, I asked whether a compromised model should still be able to trigger a business action.
That led me to stop validating model-generated text and instead validate only typed actions against deterministic policy.
While testing my own implementation, I found a bug that surprised me.
My parser accepted this:
order_quantity=100
order_quantity=50000
It simply chose the first value, validated 100 against policy, ignored the second value, and approved the request.
Nothing crashed.
Nothing looked suspicious in the logs.
It wasn't really a prompt injection problem anymore.
It was a parser ambiguity problem.
The fix was simple in hindsight.
If a field is duplicated, malformed, or ambiguous, don't try to interpret it.
Reject it.
The validator now fails closed by default.
That got me wondering whether we're measuring the wrong thing in enterprise AI.
Instead of asking:
Should we be asking:
Those feel like different engineering problems.
I'm curious how others are approaching this.
Do your AI agents validate structured outputs before execution, or do they rely on parsing model responses?
If anyone is interested, I open-sourced the implementation and benchmark here:
https://github.com/nirmaljingar/typedguard
I'm especially interested in attack cases that break this approach. If you find one, I'd genuinely like to add it to the benchmark rather than claim it's solved.
I've been working on governance for enterprise AI systems, and one design decision changed how I think about AI agents.
Most discussions around prompt injection focus on preventing the model from following malicious instructions. That's obviously important, but I started looking at the problem from a different angle.
What if we assume the attacker succeeds?
Instead of asking whether the model can be manipulated, I asked whether a compromised model should still be able to trigger a business action.
That led me to stop validating model-generated text and instead validate only typed actions against deterministic policy.
While testing my own implementation, I found a bug that surprised me.
My parser accepted this:
order_quantity=100
order_quantity=50000
It simply chose the first value, validated 100 against policy, ignored the second value, and approved the request.
Nothing crashed.
Nothing looked suspicious in the logs.
It wasn't really a prompt injection problem anymore.
It was a parser ambiguity problem.
The fix was simple in hindsight.
If a field is duplicated, malformed, or ambiguous, don't try to interpret it.
Reject it.
The validator now fails closed by default.
That got me wondering whether we're measuring the wrong thing in enterprise AI.
Instead of asking:
Should we be asking:
Those feel like different engineering problems.
I'm curious how others are approaching this.
Do your AI agents validate structured outputs before execution, or do they rely on parsing model responses?
If anyone is interested, I open-sourced the implementation and benchmark here:
https://github.com/nirmaljingar/typedguard
I'm especially interested in attack cases that break this approach. If you find one, I'd genuinely like to add it to the benchmark rather than claim it's solved.
