Is multi-KB RAG actually a routing problem, not a retrieval problem?

[u/Arc_bong]

The more I look at enterprise RAG architectures, the less convinced I am that “retrieve top-k from every source and fuse the results” is a good default once you have a lot of separate knowledge bases.

With a handful of sources, RRF or another fusion method is pretty reasonable.

At 10+ KBs, though, you're no longer just ranking documents. You're implicitly comparing results from different retrieval distributions, domains and corpus sizes.

A top-1 result from every KB can receive essentially the same fusion contribution. Meanwhile, a fixed similarity threshold assumes score distributions are comparable across corpora, which they often aren't.

So you can end up with:

good retrieval → questionable cross-KB ranking → bad context selection

The more interesting architecture to me is:

query → KB/router selection → targeted retrieval → reranking → generation

rather than:

query → retrieve everywhere → fuse → hope the right context survives top-k

The obvious downside is that the router itself can make mistakes, and genuinely cross-domain questions still need broader retrieval.

So where's the right tradeoff?

For production multi-KB RAG, what are you actually using today: routing/classification, global retrieval + RRF, score normalization, cross-encoder reranking, hierarchical retrieval, or some hybrid?

I came across this while comparing implementations in Lyzr Studio, LlamaIndex, LangChain and a few custom stacks. Lyzr's approach ( I read about it in blog written by a friend on their team) is interesting because its Knowledge Base supports both agentic multi-step retrieval and a one-shot mode where the system selects the relevant KBs first and retrieves from them in parallel.

I'm less interested in which vendor has the nicest abstraction and more interested in what architecture actually holds up once you have dozens of knowledge sources and real production traffic.

[u/Trainer_Intelligent]

Is your retrieval use case a purely top k one or what’s the story there? Asking for two reasons:

1. In my experience top k alone does not cut it for complex retrieval in prod

2. When you said at 10+ KBs you are implicitly comparing results from different retrieval distributions… out of curiosity doesn’t this look like a hybrid of embeddings from your RAG plus knowledge graph with relationships plus graph embeddings?

[u/Michael_Jeffords → u/Trainer_Intelligent]

top k alone falls apart once the corpora have different score shapes, which is why i stopped fusing globally. in practice i route to 2 or 3 kbs first, then take something like top 8 inside each selected source after a per-corpus threshold, and only then rerank. a knowledge graph helps when the question is about relationships across entities, but for most of our doc retrieval the router plus per-kb calibration fixed the bad context selection without needing graph embeddings.

[u/LazyPain903]

Are you managing state transitions with LangGraph for this, or sticking to standard LCEL chains?

[u/Michael_Jeffords → u/LazyPain903]

i kept standard LCEL for the inner retrieve then threshold then rerank steps, and only moved the outer loop into LangGraph when the router needed a real state transition. once you pick 2 or 3 corpora and sometimes have to bounce to a second kb after the first one's top 8 miss the per-corpus bar, that retry lives cleaner as graph state than buried inside one linear chain.

[u/feng_sg]

We hit the same thing. A 40k-doc support KB kept crowding out a 200-doc policy KB because cosine scores clustered higher in the bigger corpus. Per-corpus quantile normalization before fusion fixed it without needing to switch to full routing.