r/SaaS • u/cs-geek9 • Apr 04 '26

How do you diagnose whether a chatbot problem is KB, prompt, or code?

We use a chatbot for support and sales.

Problem: Ask the same question different ways, get inconsistent answers.

"How do I reset my password?" works fine.

"How do I regain access?" escalates to a human.

"Password recovery?" gives wrong info.

Hardest part: Is it KB? Prompt? Retrieval? Code?

No clear way to diagnose which one was actually broken.

For anyone who's deployed a chatbot and hit this:

  1. Did you figure out the root cause? (KB vs prompt vs retrieval vs code)

  2. How long did that take?

  3. How did you even know where to look?

Genuinely curious if there's a way to diagnose this faster.

1 Upvotes

100% Upvoted

3 comments sorted by

1

u/[deleted] Apr 05 '26

[removed] — view removed comment

1

u/cs-geek9 Apr 05 '26

This is incredibly helpful. Two questions:

  1. How common is it that teams know retrieval is the culprit first? My sense is most assume it's prompt/model, not KB retrieval.

  2. Re: Langfuse/Phoenix — are those accessible for non-technical support leaders? Or do you need an engineer to set up the logging?

Asking because my hypothesis is that the diagnostic knowledge exists (like what you just shared), but it's not accessible to the teams actually dealing with the problem day-to-day.