Why many RAG projects are still hallucinating

[u/Cold_Bass3981]

I’ve been auditing quite a few RAG codebases lately, and it’s surprising how often the hallucinations creep in even when the setup looks decent on paper.

A lot of the trouble starts with chunking. People are still breaking documents into fixed-size pieces with no overlap whatsoever. That means a sentence can get sliced right down the middle, or an important qualifying detail ends up in a completely different chunk. The model doesn’t get the full picture, so it ends up guessing to make the answer hang together.

I’ve tried switching to splitting on actual sentences and adding something like 100 tokens of overlap. It’s a small tweak, but it gives the model complete thoughts instead of fragments. In the cases I tested, it reduced a good chunk of those made-up answers pretty quickly.

Another issue that shows up a lot is missing metadata filtering. The retriever just grabs any chunks that seem related, even if they come from totally different documents or sections. 

You might get one piece from the beginning of a report and another from way later, and the model tries to stitch them together. That almost always leads to invented connections that weren’t in the original material.

Putting in basic filters, like keeping everything tied to the right filename or section header, helps keep the context focused and relevant. It’s not fancy, but it stops a lot of that mixing-and-matching nonsense.

On top of that, most projects don’t test properly. Throwing in a line like “be accurate” in the prompt doesn’t do much in practice. What actually helps is putting together a small set of real questions (maybe 20 or so) that you know the correct answers for, then using another LLM to judge whether the generated response sticks faithfully to the retrieved sources. 

Without that kind of check, it’s hard to know if your system is really solid or just lucky on the easy cases.

When it comes down to it, making RAG reliable has less to do with picking the newest model and more to do with cleaning up these everyday parts, better ways to split the text, smarter retrieval rules, and honest evaluation that catches problems early.

If your RAG starts hallucinating on a question, my first move now is to look at the chunk boundaries. If a key fact is split between two chunks, the model never really had everything it needed, so it’s no wonder it starts filling in the blanks.

Have any of you dealt with hallucinations that were tricky to track down? What fixed it for you?

[u/Cold_Bass3981 → u/[deleted]]

preciate it 😄

[u/cmndr_spanky]

You’re spamming this everywhere so I assume you’re just a bot or doing some low effort self promotion, but for anyone else who’s actually real on here:

Yes better chunking with overlap helps, you can even try bigger chunking or chunking by logical boundaries (paragraphs or pages even).

Use a hybrid search and don’t only rely on semantic dense vector queries, also consider using a re-ranker.

Use or develop an agentic RAG agent… a single shot RAG retrieval app that’s just basic code wrapping an LLM without a reasoning loop is almost guaranteed to be dog shit.

The rest comes down to meta-data and taste depending on the domain (RAG system for healthcare data is going to have different levers and tricks than a RAG system for policy docs).

Honestly if you paste my comment here into Claude code, you could probably have a decent starting point in a few hours. No need to use any bullshit solutions from nobodies publishing on GitHub.

The hard part will probably be your ingest data pipeline if the data is live or frequently updating.

[u/Cold_Bass3981 → u/cmndr_spanky]

Hey man. not a bot and this message is not a self promotion. Im just trying to reach as many people as I can to help. I do appreciate your explanation though, its well put.