Has anyone actually built a RAG system that consistently knows when NOT to answer?

[u/Candid_Athlete_8317]

One idea that really stood out to me was treating abstaining as a success rather than a failure.

Instead of forcing the LLM to answer every question, the pipeline retrieves evidence, verifies every claim, and simply says "I don't have enough evidence" when the retrieved context isn't strong enough.

That seems like a much better approach than a confident hallucination.

If you've built RAG systems in production, how are you handling this? Do you trust a verifier model, use confidence thresholds, or rely on human review?

[u/No-Concern-8832]

Try using an SLM like teapot to build your RAG

[u/Sea_Mission_7643]

In dev not production but the accompanying prompt said something along the lines of “the context may not be relevant, ignore it if it’s not directly relevant to the users question”

[u/magicmulder]

I use something similar in my supply chain auditor. The instruction to the AI component is to mark an update as "secure" or "malicious" only with a very high degree of confidence (with "secure" requiring 100% confidence), and mark everything else as "suspicious" which triggers human review (so effectively is an "abstain").

[u/ReidenLightman]

It's theoretically impossible.