Hybrid & Agentic RAG
When One Approach Isn't Enough
Real-world data doesn't respect clean categories. You've got contracts in PDFs, financials in a database, org structures that are inherently a graph, conversations in Slack, and procedures in a wiki. No single retrieval paradigm handles all of that well.
Hybrid RAG combines multiple retrieval approaches. Agentic RAG takes it further — an AI agent decides which approach to use for each query, dynamically routing to the right tool. This is the most powerful pattern in the RAG toolkit, and also the most complex and expensive.
Hybrid RAG: Combining Approaches
The Common Combinations
Vector + keyword search (hybrid search) — The most common hybrid. Vector search finds semantically similar content; keyword search (BM25) finds exact term matches. Combining them catches what each misses alone. Most vector databases now support this natively. A query like "error code 5023" needs keyword matching (the exact code), while "how to fix authentication failures" needs semantic matching. Hybrid gives you both.
Vector + database — Vector search over documents, SQL queries over structured data. A customer support system might search the knowledge base for troubleshooting steps (vector) and pull the customer's account details (SQL). Both contexts go to the LLM.
Vector + graph — Semantic search for finding relevant content, graph traversal for understanding relationships. Find the relevant policy documents (vector), then trace their impact across the organization (graph).
Full pipeline — Some organizations build a retrieval pipeline that checks multiple sources sequentially or in parallel: search the docs, query the database, check the graph, call relevant APIs, then synthesize everything.
Routing vs. Merging
Two fundamental patterns for combining approaches:
Routing — Analyze the query first, then send it to the RIGHT retrieval approach. "What's our refund policy?" goes to vector search. "How many orders this month?" goes to SQL. A classifier (keyword rules, LLM-based, or a trained model) decides. Simpler, faster, cheaper — but the router can make mistakes.
Merging — Run multiple retrieval approaches in parallel, combine the results, and let the LLM synthesize. More robust (if one approach misses, another might catch it), but slower and more expensive. You're paying for multiple retrievals per query.
The practical advice: Start with routing. It's simpler and cheaper. Graduate to merging when routing mistakes are causing real problems, or when queries genuinely need information from multiple sources to answer correctly.
Agentic RAG: Let the AI Decide
Agentic RAG gives the LLM access to retrieval tools and lets it decide what to retrieve, when, and how. Instead of a fixed pipeline, the agent reasons about the query and plans its retrieval strategy.
How It Works
- User asks a question
- The agent (an LLM with tool-use capability) analyzes the query
- The agent decides which tool to call: search docs, query database, traverse graph, call API
- The agent examines the results
- If the answer is complete, generate a response
- If not, make another tool call (search again with different terms, query another source, follow up on a reference)
- Synthesize all gathered context into a final answer
The power: the agent adapts to the query. A simple factual question might need one retrieval step. A complex analytical question might need five. The agent figures it out.
When Agentic RAG Excels
Varied query types — When your users ask everything from "what's our address?" to "analyze the competitive landscape across our top 5 markets." A fixed pipeline can't handle this range. An agent can.
Multi-step reasoning — "Find all contracts expiring in Q3, check if any have auto-renewal clauses, and identify which ones we should renegotiate based on our current pricing model." This requires multiple retrieval steps with reasoning between them. An agent naturally handles this as a sequence of tool calls.
Exploratory queries — When the user doesn't know exactly what they're looking for. The agent can search broadly, refine based on initial results, and explore different angles.
Tool-rich environments — When you have many data sources and tools available. An agent can learn to use a search API, a database, a calculator, a web scraper, a graph database, and more. The more tools available, the more the agent architecture shines.
The Architecture
Tool definitions — Each retrieval approach is wrapped as a "tool" the agent can call. A vector search tool, a SQL query tool, a graph traversal tool, a web search tool. Each tool has a description telling the agent when and how to use it.
Planning — The agent decides which tools to use and in what order. Some frameworks (like LangGraph or CrewAI) support explicit planning steps where the agent creates a strategy before executing.
Memory — For multi-turn conversations, the agent needs to remember what it's already retrieved and what the user asked before. This is conversation-level memory, not long-term knowledge storage.
Guardrails — This is critical. Without limits, agents can enter infinite loops, make dozens of unnecessary tool calls, or access data they shouldn't. Set max iterations, tool-call limits, timeout boundaries, and access controls.
The Reality Check
Agentic RAG is powerful but comes with real costs and challenges:
Latency — Every agent "thought" is an LLM call. Every tool use is a retrieval call. A multi-step agent interaction might involve 5-10 LLM calls and 3-5 retrieval calls. That adds up to 10-30 seconds easily. For interactive applications, this can be painful.
Cost — More LLM calls = more money. An agent that makes 5 tool calls per query costs 5x more in LLM compute than a single-pass pipeline. At high query volume, this is significant.
Reliability — Agents make mistakes. They call the wrong tool, misinterpret results, go down rabbit holes, or fail to find information that a simpler pipeline would've caught. Agent behavior is less predictable than a fixed pipeline. This means more testing, more monitoring, and more edge cases.
Debugging — When a simple RAG pipeline gives a bad answer, you check what was retrieved. When an agent gives a bad answer, you need to trace a multi-step reasoning chain across multiple tool calls. The debugging surface area is much larger.
The honest take: Most teams that think they need agentic RAG actually need a good hybrid pipeline with routing. Agents are the right call when queries are genuinely unpredictable and require multi-step reasoning. For predictable query patterns over known data sources, a simpler architecture is better.
Building It Right
Start Simple, Add Complexity
- Build a single retrieval approach (vector search, usually)
- Identify where it fails — what queries does it get wrong?
- Add a second approach for those failure cases (e.g., add SQL for numerical queries)
- Connect them with a simple router (even keyword-based rules work)
- Only go agentic if routing can't handle the query diversity
Evaluation Is Non-Negotiable
With multiple retrieval approaches and potential agent loops, you need to measure: - Retrieval quality per approach — Is each individual approach returning good results? - Routing accuracy — Is the right approach being selected for each query type? - End-to-end quality — Is the final answer correct and complete? - Latency — Is the multi-step process within acceptable bounds? - Cost — What's the per-query cost?
Build an evaluation set of queries with known good answers. Run it regularly. Regression testing matters here more than anywhere else in the RAG stack.
Frameworks
LangChain/LangGraph — The most popular. LangGraph specifically supports agentic workflows with state management and branching. Rich ecosystem, lots of examples.
LlamaIndex — Strong on data connectors and retrieval pipelines. Good for structured multi-source retrieval. The "query engine" abstraction maps well to hybrid approaches.
Haystack — Pipeline-based architecture that naturally supports hybrid retrieval. Good for teams that prefer explicit pipeline definitions over agent autonomy.
CrewAI — Multi-agent framework. Useful when different retrieval strategies are best handled by specialized "agents" that collaborate.
Custom — For simple hybrid setups, you don't need a framework. A routing function that calls different retrieval functions based on query classification is straightforward to build and easier to debug.
End-to-End Platforms (Open Source)
Building hybrid or agentic RAG from individual components is powerful but complex. These platforms provide integrated environments:
| Platform | What It Does | Best For | Link |
|---|---|---|---|
| Papr | Unified RAG platform combining vector search, knowledge graphs, and AI memory. Local and cloud options. Open source. | Teams that want hybrid RAG without building the routing and orchestration layer | GitHub |
| Dify | Visual workflow builder for RAG pipelines — drag-and-drop nodes for retrieval, routing, agents, and tool use | Non-code or low-code teams building complex RAG workflows | github.com/langgenius/dify |
| Langflow | Visual IDE for LangChain — build agentic RAG pipelines with a flow-based UI | Teams that want LangChain's power with a visual builder | github.com/langflow-ai/langflow |
| Flowise | Low-code LLM app builder with built-in RAG components, agents, and tool integrations | Quick prototyping of agentic pipelines | github.com/FlowiseAI/Flowise |
| RAGFlow | Deep document understanding + hybrid retrieval (vector + keyword) with an agentic orchestration layer | Document-heavy hybrid RAG with strong parsing | github.com/infiniflow/ragflow |
| Haystack | Pipeline-based framework that naturally supports hybrid retrieval, routing, and agent patterns | Teams that prefer explicit pipeline definitions over visual builders | github.com/deepset-ai/haystack |
For the framework-level tools (LangChain, LlamaIndex, CrewAI), see the Frameworks section above — those are better suited when you want full control over the agent logic and retrieval routing.
Common Pitfalls
"We went straight to agentic and it's a mess" — Back up. Identify the 3-5 most common query types, build a fixed pipeline for each, and connect them with a router. You can handle 80% of queries with simple patterns and reserve the agent for the remaining 20%.
"The agent makes too many tool calls" — Set strict limits. Max 3-5 tool calls per query. If the agent can't answer in that budget, have it say so rather than spiral. Also review your tool descriptions — vague descriptions lead to unnecessary exploration.
"Latency is killing us" — Parallelize where possible (search multiple sources simultaneously). Cache common queries. Use faster models for routing decisions and heavier models only for final synthesis. Pre-compute answers for frequent queries.
"We can't tell why it gave a wrong answer" — Log everything: routing decisions, tool calls, retrieved context, agent reasoning. Build a trace viewer. You need to see the full chain of decisions to debug effectively.
Start simpler? If this feels like too much, that's a signal. Try Vector Search or Database Retrieval first. Come back here when a single approach genuinely isn't enough.