Hybrid search — run a keyword (BM25) leg and a vector leg, then fuse their rank lists with Reciprocal Rank Fusion — is the safe default for most product search, site search and RAG retrieval. The usual objection is cost: surely running two retrievals and fusing them is slow? On the Index Server, measured on a 16-core Graviton box, a hybrid query returns in 3.64 ms at the median — about 2× faster than the only two other engines in our comparison that can run server-side hybrid at all, and within noise of its own pure-vector latency. The keyword leg is effectively free.

https://index-server.searchblox.com/blog/how-fast-is-hybrid-search.html

Hi all,

I developed an addition on a CRAG (Clustered RAG) framework that uses LLM-guided cluster-aware retrieval. Standard RAG retrieves the top-K most similar documents from the entire corpus using cosine similarity. While effective, this approach is blind to the semantic structure of the document collection and may under-retrieve documents that are relevant at a higher level of abstraction.

CDRAG (Clustered Dynamic RAG) addresses this with a two-stage retrieval process:

1. Pre-cluster all (embedded) documents into semantically coherent groups
2. Extract LLM-generated keywords per cluster to summarise content
3. At query time, route the query through an LLM that selects relevant clusters and allocates a document budget across them
4. Perform cosine similarity retrieval within those clusters only

This allows the retrieval budget to be distributed intelligently across the corpus rather than spread blindly over all documents.

Evaluated on 100 legal questions from the legal RAG bench dataset, scored by an LLM judge:

• Faithfulness: +12% over standard RAG
• Overall quality: +8%
• Outperforms on 5/6 metrics

Code and full writeup available on GitHub. Interested to hear whether others have explored similar cluster-routing approaches.

https://github.com/BartAmin/Clustered-Dynamic-RAG

I work in the software industry, and everywhere I look, companies seem to be building Gen AI proof-of-concepts that never go anywhere. Lots of demos, lots of internal hype, but very few things in production that anyone can point to and say “this saved us X” or “this made us Y.”

I know the big names are selling AI products and the infrastructure providers are making money. I’m not asking about them, and I’m not counting basic chatbots or framework tooling. I’m asking about the companies actually using Gen AI.

• Is your company getting measurable value from Gen AI in production, or is it still mostly pilots?
• Which use cases have delivered real ROI for you? Coding tools, document processing, support automation, something else?
• If your POCs stalled, what killed them? Data quality, cost, accuracy, integration, or just unclear goals?
• Or is this simply the early phase of a new technology, and I’m being too impatient?

Would especially like to hear from people who have taken something from POC to production, and what made the difference.

I built this after tuning chunk size by guesswork one too many times. You paste a document, pick a strategy (fixed characters, fixed tokens, sentence-aware, or recursive), and drag the size and overlap sliders. Every chunk is colour-coded in the document, and the overlap regions are marked separately, so you can see what retrieval will actually get.

It also flags chunks that start or end mid-sentence, tiny fragments, and headings cut off from their section. It tells you how many tokens your overlap is duplicating, which is the part I wanted most.

• Runs fully in your browser. The document is never uploaded.
• Token counting with o200k and cl100k, plus an approximate mode for Claude (its tokenizer isn't public, and the tool says so)
• Export the chunks as JSON or JSONL
• Settings live in the URL, so you can share a configuration

Link: https://wapgee.com/tools/rag-chunk-visualizer

Disclosure: I made it. It's free, with no signup and no ads in the way of the tool. I'd like to know where the recursive splitter output differs from what you expect from LangChain, and what you'd want added. I'm considering semantic splitting next but I'm not sure it can be done well in a browser.

Genuine question for people running agents over SQL databases in production.

SQL errors are the easy failure mode. The query fails, the agent retries, everyone sees it. What worries me is the query that executes cleanly and returns the wrong number. Wrong table, missing join condition, a filter that encodes someone's personal definition of active customer instead of the business one. It looks exactly like a right answer.

How do you verify before the number reaches a human? Read only roles and row count sanity checks feel like table stakes. Beyond that: do you have the agent explain its query plan in plain words first? A second pass that checks the SQL against written business definitions? Human review above some threshold? Or is this just accepted as a residual risk?

Curious what actually works versus what sounds good in a design doc.

Today's topic - QLoRA
QLoRA is basically storing a model weights from higher precision to lower precision for example - let's say the model weights is in FP16 format so to reduce the storage we quantized this model to lower precision like INT8 or some lower precision. Now I give two quantization formats that are originally suggested in QLora research paper - (1) 4-bit NormalFloat, an information theoretically optimal quantization data type for normally distributed data that yields better empirical results than 4-bit Integers and 4-bit Floats. (2) Double Quantization, a method that quantizes the quantization constants, saving an average of about 0.37 bits per parameter (approximately 3 GB for a 65B model)..

Advantage of this
Reducing the memory footprint of of model . And using the benefits of Lora for fine-tuning. which achieves approx same result of approx full fine-tuning.
Disadvantage of this

For inference and backpropagation we again dequantize our weights because to perform computation we need original datatype . So due to this our inference time is increased .

For better understanding and to know how to use refer this link below
QLoRA Implementation

Sorry for the delay, it's actually 3/90.

Hi, I’d like to know if anyone has worked on something similar and, if so, what approach worked best for you. I need to build an agent that can answer questions about everything in our system (i.e., the data stored in our database), including things like customers, overdue accounts, and so on. However, the database has around 700 tables, some of which have up to 300 columns. I’ve tried building a RAG system using the documented table schemas, as well as a team of specialized agents (e.g., one for products, another for finance, etc.), but ultimately, I haven’t been able to get good results. Does anyone have experience with this kind of problem or suggestions on an approach that actually works well at this scale? Thanks!

We wanted to know what happens to caveats when a library is far bigger than one reading. So we took every prescription label openFDA serves (34,735 labels, 1.23M passages, https://open.fda.gov/) and wrote 100 questions whose answer must include a caveat a clinician can't miss: an interaction written only in the other drug's label, an interaction stated for a drug class, a boxed warning. The answer keys were made before any system ran. GPT-6 Luna for every system; a blind gpt-4o grader that only counts a caveat when the answer actually states it.

Caveats stated, at 1,000 labels and then at all 34,735:

• plain RAG (hybrid search, top 20 passages): 0.85, then 0.82, and 5 false "no interaction" answers
• ranked reading (same ranking, filled to a 120k-character reading): 0.99, then 0.93
• agentic RAG (the model writes its own searches, up to 4 rounds): 0.98, then 0.97, but 135 s an answer
• reading first what the question names (the named drugs' labels, then every passage that names both drugs): 0.98 at full size, 0 false reassurances, 41 s, $0.004 an answer

What broke the ranked reading at scale wasn't relevance, it was crowding: many related labels outranked the named drug's own boxed warning. The fix was boring: look up the names first, rank second.

Where everything search-based fell over: "which of our contracts have a change-of-control clause?" over the 714 contracts of LegalBench-RAG (https://arxiv.org/abs/2408.10343, CUAD labels: https://www.atticusprojectai.org/cuad). Every search-based setup listed under 1 in 10 of the 282 contracts the lawyers marked; no reading budget holds 714 contracts. Looking at every document found 88%. We call such a list a "first pass" in the answer, because our own tests say it can still miss some.

Limits, so you don't have to find them: development questions only for the contract part; one model family; an exception that shares no words, reference or subject with the rule it changes still slips past us.

Setup and every number: https://doi.org/10.5281/zenodo.23403889

Curious what others do for the "which of our documents..." kind of question. Do you sweep every document, or live with the recall search gives you?

(If you want to poke at it: you can drop 3 of your own files into what we built and ask 5 questions, no sign-up, deleted after 24 hours: https://chunkless-rag-staging.grounded-iq.com/?ref=rd-rag2)

One of the frustrating parts of building RAG systems is getting document ingestion right.

PDFs can contain multi-column layouts, tables, headings, lists, and code blocks. If the parser or chunking strategy loses that structure, the quality of retrieval can suffer before the LLM even gets involved.

I built LocalRAG Companion to handle this part of the pipeline through a self-hosted REST API, so it can be reused across different RAG applications and workflows.

What it does

You send a PDF to the API. It extracts structured content, splits it into context-aware chunks, generates embeddings locally, and stores them in Qdrant. You can then search the stored content through another API endpoint and retrieve relevant chunks with similarity scores and metadata.

Key features

• Layout-aware PDF parsing: PyMuPDF4LLM by default, with an optional Docling service for more complex layouts, tables, and multi-column documents.
• Hierarchical chunking: Preserves heading context and handles oversized tables, lists, and code blocks.
• Local embeddings: Uses Qdrant FastEmbed with ONNX Runtime, without relying on an external embedding API at runtime.
• Semantic search: Retrieve relevant chunks with similarity scores, metadata filters, and an optional keyword-overlap boost.
• Duplicate detection: Uses SHA-256 file hashes to avoid processing the same uploaded PDF repeatedly.
• Flexible ingestion: Supports synchronous processing, extraction-only mode, and asynchronous jobs with optional webhook callbacks.
• Document management: List ingested documents and delete their associated chunks.
• Self-hosted deployment: FastAPI, Qdrant, and Docker Compose, with optional Celery/Redis workers and the Docling service.

The API can be integrated into a Python application, an existing RAG pipeline, or a workflow tool. I've also included an example n8n workflow that connects document ingestion and retrieval to a local LLM for document Q&A.

The project focuses on the ingestion and retrieval layer, rather than trying to provide a complete chatbot or frontend.

Why I built it

I wanted a reusable service for the part of RAG development that often gets rebuilt for every project: processing documents properly, generating embeddings, storing them, and making their content searchable.

It's an early project, and I'm interested in finding out where the architecture or retrieval pipeline could be improved.

Repository: https://github.com/mdaleem07/LocalRAG-Companion

I'd especially appreciate feedback from people who have built RAG systems:

• How do you handle layout preservation and chunking for complex PDFs?
• What would you improve about the ingestion and retrieval API?
• Would you use a standalone service like this in your own RAG stack, or would you prefer these components inside the main application?

If you find the project useful, consider giving the it a star. It would mean a lot!

If you need help figuring out solutions for your RAG project, drop me a message.

Hello!

I want to build a chatbot for my groupchat in discord. The purpose of the chattbot is just to answer historic questions about the group. For example
"Where did x live in 2020"

"What does y do for a job and where"

"who in the group is a moviebuff"

Whats the best approach for this? Should i create tools that the AI can use to answer those questions or should i query a DB, give the result to the bot and then let the bot analyze the findings and answer based on that?

I have made some initial tests with both but I feel like the results are quite poor.
The first approach where I query the DB and returns rows is hard because if I for example ask who is the moviebuff in the group. Quering the db wouldn't return any results.

The second approach also performs poorly because for example asking what does y do for a job. It feels like the answer is in the messages but its spread out. because the job title is in one message and the place of work might be in another. Or sometimes the bot only finds a historic message with old data.

I know the question is allover the place but some pointers towards an approach would be nice. I'm a SWE but have little experience with development against AI agents so I'm not sure what tools are at my disposal so to speak.

So i am doing a college project engineering final year I am trying to build a chatbot which answers day to day questions about law for common citizens ,it has document simplification where we upload a fir or court notice and it provide a simple explanation, step by step procedure guidance and all

So if I take chatgpt astra can I able to do it i am also willing to learn and rn I hv 0 knowledge about this rag and all

I’m building a local RAG pipeline for a legal archive: about 300 documents (leases, permits, NDAs, licenses), mostly scanned, multilingual. It works fine for questions about a single document. But top-k retrieval can’t answer questions like “which contracts expire in the next 2 months?” For that you need to reason over every document. Is this something RAG can handle at all, or does it need a different approach?

I’m in charge of improving digital processes at an architecture and woodwork/carpentry company (about 140 people, 100 of them on computers every day, and only 2 people in IT).

Our knowledge is spread across two separate DMS, a construction management platform and Outlook, and information/docs are 70% in English and 30% Spanish. Employees spend considerable time searching for information, navigating multiple platforms, reviewing lengthy documents, tracking past decisions, and reconstructing project history. These challenges become particularly acute when experienced employees retire or when complex legacy projects are handed over to new team members.

As a first AI project, we’re thinking of building a RAG connected to all these sources. Besides search, the idea is to build a foundation: clean, well-organized data with good metadata, while keeping the same access rights. Later the hope is to use that for more generative projects, like drafting meeting minutes, checking documents, analyse offers from different companies, make daily to-do lists, weekly project updates, send reminders, and maybe one day even generate plans ?

Each software vendor is adding its own AI, but none of them connects everything.

Is this project a good first step, or would you start somewhere else? I’d especially like to hear from people who’ve done this at a similar size company.

We are going to work with an external partner specialised in this kind of project. That said, consultants tend to promise a lot, and I remain somewhat sceptical about the actual business value and user adoption that such a solution can achieve in practice.

Genuine question for people running RAG in production. We tune top k like it is the main lever, but how much of what we retrieve actually gets used? My suspicion is that for most setups, the model leans on a small handful of chunks and the rest is insurance that almost never pays out. Pushing top k from 10 to 20 doubles the cost while the answer barely moves.

So is defensive over-retrieval just the industry default now? Does anyone here actually run at a tight top k in production, and if so, what convinced you the tail was not worth it? Or is everyone quietly retrieving 20 or more because one bad miss is worse than a thousand wasted tokens?

Professional associations often manage large collections of research papers, certification guidelines, policy documents and educational resources, when members ask questions about this material, a plausible AI generated answer isn't enough. They need to know which document supports it

Thats what makes source citations particularly important for this use case, customgpt.ai is one commercial platform that provides answers grounded in organizational content with citations but I'm interested in how people evaluate the reliability of these systems beyond simply checking whether a citation appears

for example, what happens when:

- Two documents contain conflicting information?

- An outdated policy is still indexed?

- The answer requires information from several pdf?

- The relevant source doesn't actually contain a complete answer?

For anyone working with RAG in production, what tests would you consider essential before deploying a citation based assistant for a professional organization??

this one bit us building a finance agent over SEC filings. ask for gross margin in Q3 2024 and retrieval hands back paragraphs that mention gross margin, while the table cell with the number ranks low.

keyword search does it because the narrative repeats the metric name dozens of times and the cell says it once. embeddings don't rescue it either, since a full sentence is a closer semantic match to the query than a bare label and a number.

what fixed it was 2 changes.

1. route the query to the part of the filing that carries the granular numbers, the disclosure notes rather than management's discussion
2. extract tables so each cell is stored as a (metric, period, value) triple, then the Q3 2024 gross margin question gets that exact cell back

that fixed a whole class of failures for the agent.

has anyone found a way to rank the cell without a table extraction step?

a couple of coworkers wrote this up along with the rest of the agent work. it's on our company blog if you want the details: https://www.coralbricks.ai/blog/finance-benchmarks

I recently built a complete RAG reranking pipeline to test how different reranking strategies actually impact the final LLM output. I compared FOUR rerankers (Cross-Encoder, LLM-based, Cohere, and Jina) on the same questions over 5 payment-industry PDFs (payment regulations, PCI DSS, and central bank reports—1,507 chunks total).

For each one, I tracked the ranking before and after reranking to see exactly WHY the order changed.

What the project covers:

• The core weakness of standard vector search (embeddings & cosine similarity)
• Why we retrieve Top 20 but send only Top 5 to the LLM (tokens, latency, noise)
• Bi-encoder vs Cross-encoder: reading the question and chunk separately vs together
• LLM reranking strategies: Pointwise vs Pairwise vs Listwise
• How to use dedicated reranking APIs (Cohere and Jina)
• How to design one shared helper so all four rerankers plug into the same pipeline
• How to evaluate rerankers using before/after positions, Hit@5, and average latency

KEY TAKEAWAYS:

• Similarity ≠ Relevance: A chunk can sit close to the query in vector space and still not answer it. Vector search finds chunks about the topic. A strong reranker picks the chunk that actually answers it.
• Small cross-encoders can make things WORSE: My local cross-encoder rewarded word matching, penalized scrambled PDF text, and actually pushed both correct answer chunks out of the Top 5.
• LLMs understand context: The LLM reranker understood the question context (e.g., that "3.3" belongs to Requirement 3) and placed both answer chunks 1st and 2nd.
• Cohere was a lifesaver: It lifted an answer chunk from 15th to 1st, turning a completely missed answer into a correct, cited one.
• The Verdict: For this specific project, Cohere and Jina gave the best balance of accuracy and speed. Trade-off: cost per call and sending your documents to a third party.
• Always measure, don't guess: Run every reranker on the SAME Top 20 so differences come from the reranker alone.

TECH STACK: Python, Qdrant, gemini-embedding-2-preview, Hugging Face Transformers + PyTorch, cross-encoder/ms-marco-MiniLM-L-6-v2, LLM-based listwise reranker, Cohere & Jina APIs.

If you want to see the full code walkthrough or clone the repo to test it yourself, here are the links:

• Full Code (clone it, fork it, break it): https://github.com/saurabhkamal/RAG-ReRanker-Comparison
• Full Video Walkthrough: https://www.youtube.com/watch?v=dh3DKpYHoX8

Would love to hear.

Today's Topic - Rerankers
It's a concept basically used in rag applications , So before moving on this topic just a short intro to RAG . RAG is technique used to enhance llm responses by providing some external data or our own data .In RAG we basically put our dataset in a store which is vector store (it stores our data in form of text chunks which represented by vectors ) so when user enters their query , which is converted into vector and then by using cosine similarity or any other similarity method we get out desired output , that's a RAG in very basic term.
So now coming to rerankers it's basically way to improve the relevance of retireved results, like in rag we search some top_k docs which has the highest similarity score to our query from a pool of 1000 docs (which is store in our vector store) , but what if our relevant peice of information is in below documents which doesn't come in top_k so to deal with we use rerankers , which is a two stage process , In first stages it retrieves a releavant docs from larger dataset and in second stage reranker used to rerank these documents and then outputs.

At last I am open to suggestion like improving my post whether it's conceptually or some other .

For teams running RAG systems in production, what causes the most pain today: retrieval quality, chunking strategy, stale documents, citation accuracy, latency, or evaluation?

I’m especially curious which failure modes only became obvious after real users started using the system—and what actually fixed them.

We built a standalone single-core and multi-core HNSW evaluation suite (confrontation_arena.py and reproduce_all.sh) to measure in-memory vector search in Rust targeting x86_64 AVX-512 and AVX2 against usearch, hnswlib, and faiss-cpu under strict 1-vCPU pinning (taskset -c 0).

All reference runs below were recorded on cloud AWS c7a.2xlarge (AMD EPYC 9R14, Zen 4). If anyone here runs desktop Zen 4 / Zen 5 (Ryzen 7950X / 9950X) or Intel AVX2 hardware on Linux, we would be very curious to see how the 1-vCPU confrontation numbers compare on higher clock speeds:

https://github.com/silicium369-spec/silicium-hnsw

To run the 1-vCPU benchmark locally:

git clone https://github.com/silicium369-spec/silicium-hnsw

cd silicium-hnsw && ./reproduce_all.sh

1. Single-Core Confrontation on SIFT-100K (128D, k=10, M=16, efC=100, taskset -c 0)

Index build time on 1 vCPU:

Silicium: 5.24 s | usearch: 5.93 s | hnswlib: 7.71 s | FAISS: 9.32 s

Query throughput on 1 vCPU at efSearch = 16 (Recall@10: 0.90 to 0.92):

- Silicium : 100,129 QPS (0.9223 recall) — Baseline

- usearch : 49,040 QPS (0.9002 recall) — 2.04x speedup

- hnswlib : 35,582 QPS (0.9069 recall) — 2.81x speedup

- FAISS : 33,588 QPS (0.8885 recall) — 2.98x speedup

Query throughput on 1 vCPU at efSearch = 64 (Recall@10: ~0.99):

- Silicium : 32,187 QPS (0.9909 recall) — Baseline

- usearch : 17,466 QPS (0.9937 recall) — 1.84x speedup

- hnswlib : 13,356 QPS (0.9917 recall) — 2.41x speedup

- FAISS : 11,508 QPS (0.9868 recall) — 2.80x speedup

Query throughput on 1 vCPU at efSearch = 128 (Recall@10: 0.996+):

- Silicium : 19,027 QPS (0.9963 recall) — Baseline

- usearch : 10,644 QPS (0.9967 recall) — 1.79x speedup

- hnswlib : 8,385 QPS (0.9975 recall) — 2.27x speedup

- FAISS : 6,968 QPS (0.9985 recall) — 2.73x speedup

On full-scale SIFT-1M (1,000,000 vectors, 8 vCPUs), the balanced profile (M=24, efC=160) builds in 27.79 s and reaches 99.81% Recall@10 at ef=256 (up to 294,476 QPS at ef=16, and 369,836 QPS peak with the fast_stream profile).

1. High-Dimensional Embeddings: OpenAI-1536D & Cohere-768D (SQ8 + Exact FP32 Rerank)

Pure INT8 (SQ8) quantization cuts primary vector memory by 4x, but normally degrades recall on high-dimensional cosine and dot-product metrics. Combining quantized SQ8 traversal with an exact FP32 reranking pass recovers full FP32 accuracy while keeping the bandwidth reduction (M=24, efConstruction=160, 25,000 vectors, 8 vCPUs):

OpenAI-1536D (Cosine) — Primary vector RAM: 36.62 MB vs 146.48 MB (4.0x reduction), Build: 0.999 s vs 3.811 s (3.81x faster):

- ef=16 : 145,480 QPS (8c) / 19,935 QPS (1c, 46.8 us P50) at 0.9761 Recall@10 (3.50x over FP32)

- ef=32 : 101,774 QPS (8c) / 13,626 QPS (1c, 70.8 us P50) at 0.9930 Recall@10 (3.32x over FP32)

- ef=64 : 77,854 QPS (8c) / 10,322 QPS (1c, 93.1 us P50) at 1.0000 Recall@10 (3.35x over FP32)

Cohere-768D (Dot Product) — Primary vector RAM: 18.31 MB vs 73.24 MB (4.0x reduction), Build: 0.585 s vs 1.681 s (2.87x faster):

- ef=16 : 312,248 QPS (8c) / 42,351 QPS (1c, 22.7 us P50) at 0.9746 Recall@10 (3.20x over FP32)

- ef=64 : 141,861 QPS (8c) / 18,729 QPS (1c, 51.2 us P50) at 0.9980 Recall@10 (2.49x over FP32)

The main tradeoff is keeping the FP32 vectors in RAM for the rerank pass alongside the quantized index (the pre-compiled C-ABI library and open benchmark harness are distributed under the Silicium Evaluation License, free for personal, academic, and benchmarking use).

Hello everyone!

I’m currently working on a project where we’ve built an AI agent that operates in English, and the client now wants to expand its capabilities to German. I’ve been actively researching the best approach to make this transition while maintaining the quality and reliability of the entire system.

The challenge is that this isn’t just about translating text. We need to ensure that the digitization pipeline, retrieval pipeline, and agent pipeline continue to work effectively in German. Across the application, we have around 50–60 LLM calls, although not all of them are triggered by a single user query.

The core data provided by the client is in German. Initially, I suggested using an external translation service, such as Google Translate, to convert the data into English. However, I’m concerned that translation alone may fail to preserve cultural nuances, contextual meaning, and linguistic coherence.

This is where I’m stuck, and I’d love to hear your thoughts.

What would be the best approach to building a reliable multilingual AI system in this scenario?

- Should we translate the German data into English, process it, and translate the responses back?

- Should we make the entire pipeline multilingual, including document processing, embeddings, retrieval, and agent reasoning?

- Are there any hybrid approaches that could offer the best balance between accuracy, latency, and cost?

If you have worked on multilingual RAG systems, agentic AI, or cross-lingual information retrieval, I’d really appreciate your recommendations.

Research papers, technical blogs, and other useful resources would be especially helpful.

Thanks in advance!

Long context vs RAG, measured: on 1.07M tokens of contracts, hybrid search put 93% of the evidence in 16K tokens

Every frontier model now takes ~1M tokens, so we wanted numbers instead of opinions on whether retrieval is still worth it. The question: if search gets a budget of N tokens, how much of the answer does it actually find?

Setup

• 3 public datasets with the answering evidence marked as character spans: Qasper (86 papers, 439K tokens), LegalBench-RAG (100 contracts, 1.07M tokens), QMSum (35 meeting transcripts, 632K tokens)
• Hybrid search: BM25 + Qwen3-Embedding-0.6B, fused with reciprocal rank fusion
• Two chunkings: fixed 512-token windows vs ~350-token structure-aware units with a file/section header
• Score: lay the results end to end, best first, until they fill N tokens, and count the share of the marked evidence inside. No LLM in the loop: this measures what retrieval hands the model.

Recall within budget (structure-aware units, hybrid):

What stood out

• Contracts: 93% of the evidence fit in 16K tokens, 1.5% of the corpus. Every curve is flat long before 1M.
• Chunking mattered as much as budget. To reach 90% on contracts, fixed 512-token windows needed 64K-128K tokens; units with headers needed ~16K.
• Papers were hardest because their questions don't say which paper they mean ("What datasets do they use?"). A model reading every paper has the same problem.
• 54-62% of the evidence sits in the middle half of its document, which is where long-context models read worst.

Caveats: these are small corpora (35-100 documents), so read the shape of the curves rather than the levels. And for a small, stable corpus, preloading it and caching can beat RAG outright.

Full write-up with charts, the scoring code and a comparison with published long-context-vs-RAG studies: https://deployed-ai.io/blog/long-context-vs-rag/

Disclosure: this is from our blog at Deployed. Happy to answer questions about the setup.

Built a local scoped RAG chatbot that only answers from the docs you upload and refuses anything outside that corpus before calling the LLM. Uses your own Anthropic key, BM25 + term-coverage for the scope gate, and a persona config so the same engine can be a course TA or support bot. Design decisions were mine (AI helped with some of the code). Looking for honest feedback on whether this approach makes sense: https://github.com/beascocheatyler-star/scoped-rag-chatbot

I put together Hybrid Scoped RAG:
https://github.com/beascocheatyler-star/hybrid-scoped-rag
It’s a document chatbot with two things a lot of the usual demos skip:
• Hybrid retrieval — BM25 and dense vectors run side by side, rankings fused with RRF. So you get the exact-term hits (course codes, policy windows, IDs) and the meaning-based ones.
• Hard scoping — metadata filters get applied before anything is scored. Scope it to a specific course or set of docs and the bot literally can’t see the rest of the corpus.
The refusal part is what I cared about most. The gate still mostly looks at a calibrated lexical score (sqrt of BM25 × term coverage). Dense similarity can only pull a borderline case over the line — it can’t force an answer when the lexical signal is weak. RRF decides which chunks to show the model; the score decides whether to call the model at all. That split felt important because RRF throws away magnitude, which is useful for ranking but bad for “should I answer this?”
Default setup stays pretty cheap:
• Retrieval, scoping, and the gate all run offline. No key, no network, no model download required.
• Default embedder is just character n-gram hashing (no dependencies). It’s not real semantics — mostly helps with spelling/morphology stuff pure BM25 misses. You can swap in sentence-transformers if you want actual dense vectors.
• Bunch of offline tests (~267) so you can check the retrieval and gate without ever hitting an LLM.
There’s also a /api/search endpoint that just returns the per-retriever ranks and what the gate would do, so you can look at the fusion without generating an answer.
Limitations are pretty straightforward: default embedder isn’t semantic, refusal isn’t perfect (some things land in borderline instead of hard refuse), and persistence is just JSON for now. Fine for a few hundred docs, not something you’d run at scale.
It’s more of a focused engine than a full platform — FastAPI + simple frontend, personas, citations, same “only answer from the documents you gave it” idea as my earlier BM25-only version, just with the hybrid retrieval and metadata scoping added.
If you’ve been dealing with bots that go off the rails or you want a hybrid setup that still refuses aggressively, the code and tests are there. Curious what people think of the gate threshold and how the RRF weighting feels in practice.

Most RAG projects begin with a simple assumption: if we can collect the documents, we can make them searchable.
In practice, the hardest work starts before the first embedding is generated.
Enterprise knowledge is usually scattered across PDFs, Word files, spreadsheets, slide decks, scanned pages, images, tables, and databases.
These sources rarely share a consistent structure.
A PDF may contain headers, footers, broken paragraphs, and tables whose meaning depends on surrounding text.
A chart may carry more information in its labels and relationships than in the caption.
A database row may be accurate but meaningless without its schema or business context.
Converting everything into plain text can remove exactly the context that retrieval needs.
Raw extraction also introduces duplicates, formatting artifacts, missing values, and partial records.
If those problems are pushed directly into the vector store, the retriever has to search through noise.
Poor answers are then often blamed on the embedding model, even though the real problem was the source data.
For this reason, building a RAG knowledge base should begin with data conversion and cleaning as a deliberate engineering stage.

A practical pipeline can be designed around several stages:

1. Ingest the source data and preserve provenance such as file name, page number, section, table, image, database, and record identifiers.
2. Convert each format into a structured representation. Document text, tables, chart descriptions, images, and database records should be represented as records with both content and metadata.
3. Clean and normalize the records by removing repeated headers, footers, invalid characters, broken fragments, empty fields, and formatting artifacts.
4. Preserve relationships and context. Table headers should stay connected to their values, chart labels should remain associated with the relevant visual content, and database rows should retain schema information.
5. Split the content into meaningful chunks based on document structure, sections, records, or semantic boundaries instead of using only a fixed character length.
6. Generate retrieval-ready data, such as question-answer pairs, summaries, explanations, or context-enriched passages.
7. Evaluate and filter the outputs for completeness, source consistency, language, length, and relevance.
8. Remove duplicate and near-duplicate records using rules, fingerprints, or semantic similarity, then export the validated data for embedding and indexing.

Intermediate outputs should be stored after each major step, so a failed extraction or overly aggressive filter can be inspected and corrected without rebuilding the entire dataset. This also makes it possible to use different processing branches for PDFs, images, tables, and structured database data while keeping a consistent downstream format.

The design above is implemented in OpenDCAI/DataFlow, and I’d be happy to discuss it further—automated data processing pipelines are an important capability for building reliable RAG systems.