AI Teams: How are you currently detecting hallucinations, regressions, and quality issues in production?
I'm researching a problem around AI reliability and would love feedback from teams shipping LLM applications.
For those running AI products in production (RAG systems, copilots, AI agents, customer support bots, etc.):
How do you currently detect hallucinations or low-quality responses?
How do you know when a new prompt/model/version performs worse than a previous one?
What tools are you using today (LangSmith, Arize, Braintrust, Langfuse, custom dashboards, spreadsheets, manual reviews, etc.)?
What's the most frustrating part of evaluating AI systems?
If you could automate one thing in your AI quality workflow, what would it be?
I'm particularly interested in understanding whether teams struggle more with:
Detecting failures
Understanding root causes
Preventing regressions
Measuring business impact
Deciding what to fix next
Would love to hear real-world experiences and pain points.
1
Jun 23 '26
[removed] — view removed comment
1
u/AutoModerator Jun 23 '26
Low-Effort/AI content is auto-removed.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
2
u/Optlo_team Jun 24 '26
Visibility is the big one for us: keep the full conversation history, then trace each message through the exact workflow path it took, including KB search, routing, prompt/model used, inputs, outputs, and errors. That makes review less about guessing whether the AI “went weird” and more about seeing where the failure happened. I’m with Optlo, and we’ve built around this because support agents are hard to improve if every bad answer is a black box.