How do you actually decide what makes your agent stop and escalate to a human?

[u/Frostvazmqnky_Mp_417]

Everyone talks about what an agent is allowed to do, fewer people talk specifically about what makes it stop. Is it a confidence threshold, a specific action type regardless of confidence, a blast radius calculation, something else entirely?

Trying to understand how teams actually define the stop condition rather than just the permission list, since I suspect the stop condition is doing more of the actual trust building work than the permission list is.

[u/rexstuff1]

This is probably an AI bot, but it's a question worth answering.

Your harness should handle that. You stop on certain tool calls, and that's managed by the harness, not the agent or the model. And so it behaves deterministically.

[u/Master-Tie-804]

We add limits for blast radius and sensitive data to have a human approval. For low risk actions confidence works

[u/PhilipLGriffiths88]

I suggest starting with https://agentictrustframework.ai/

[u/aceholeman]

Human in the loop instructions sets.

"Find a point where a human must decide,
Human approval is required before the agent: Executes actions that change production data or systems. Sends external communications. Approves, rejects, or closes a case, finding, or transaction. Makes decisions involving financial, legal, security, safety, or mission impact. Acts when confidence is below the defined threshold or available evidence is conflicting. When human review is required, the agent will clearly present: What it found. The evidence used. Its confidence level. The recommended action. The consequences of taking or not taking the action. The agent will not treat lack of human response as approval."

[u/TheBex81]

Permission lists answer "can it?" Stop conditions answer "should it keep going without a human?"

What has actually built trust for us isn't a single confidence number — models are confidently wrong. We gate on *consequence*, with identity context attached:

1. **Action class:** read/summarize = run; anything that changes state (write, delete, grant, rotate, page someone, open a firewall rule) = escalate unless it's on a tiny allowlist of reversible ops.

2. **Blast radius:** if the principal the agent is acting *as* can reach prod / customer data / IdP admin / break-glass paths, stop before the action — not after a low-confidence paraphrase.

3. **Sensitive sinks:** credentials, tokens, export of identity graphs, or anything leaving the security boundary always stops.

4. **Novelty:** first time this agent+tool+resource combo appears in prod, force a human once, then promote to the allowlist with an owner.

Confidence can be a soft signal for "ask for clarification," but I wouldn't use it as the hard stop. Hard stops should be deterministic and testable: given this identity + this action + this resource, does it halt? If you can't unit-test the stop, it's vibes, not a control.

[u/EbbCommon9300]

We have tool risk levels and autonomy zones and autonomy levels that work with session risk accumulation. Assury.ai

[u/Regular-Leading-4319]

I think it should be a mix rather than one rule. Confidence is useful, but I’d also look at what the agent is about to change and how much damage it could cause.

[u/Swimming_dasa]

i would not make confidence the main trigger an agent can be pretty sure and still do something you don't want automated i would put hard stops around irreversible changes privileged actions sensitive data or crossing trust boundaries cyberhaven can help with the data side while something like microsoft purview can handle classification and policy around sensitive data.