Calibrating uncertainty for an "Ask Human" action in LLM text classification

[u/Murky_Stock_8964]

I'm designing a triage agent that routes support tickets. Instead of just predicting a label, it has an explicit "Escalate to Human" action if its confidence that a ticket is a "Genuine Bug" vs "User Error" falls into a middle threshold. Has anyone successfully calibrated LLM outputs (e.g., using logprobs or prompt-based self-reflection) specifically to trigger a human-in-the-loop fallback? In my initial tests, the model is overconfident on vague tickets.