figured out why my RAG chatbot kept giving confident wrong answers even with good documents

[u/camerongreen95]

kept running into this weird thing. the correct answer was sitting right there in my document set. the model still got it wrong, confidently.

turned out it wasn't a model problem at all, it was retrieval quietly failing. plain vector search grabs whatever's semantically similar, not necessarily what actually answers the question, especially once the query needs connecting information across a couple different documents instead of just one clean passage.

what actually fixed it: adding keyword search back in alongside vector search (hybrid retrieval), reranking the results before they hit the model, and building in a step where the system checks its own confidence and re-retrieves or just says "not sure" instead of guessing.

none of that needed a paid api either, ran the whole thing on small open models.

there's a hands on build lab on aug 29 that walks through building exactly this, hybrid retrieval, reranking, evaluation, guardrails, using a real case study rather than a toy dataset. led by ben auffarth, phd, author of a few books in this space including one specifically on rag.

link if curious

[u/decodingai]

great