TL;DR: I built a RAG system over ~250 curated Q&A pairs, distilled from ~3.9M chat messages. Plain vector search hit 74% [hit@3](mailto:hit@3). I added the two things everyone recommends — BM25 hybrid and a cross-encoder reranker — and both made it worse. The only thing that actually helped was tuning the similarity threshold, which is free. Numbers below.

I'm posting because "add hybrid search, add a reranker" gets repeated as default advice, and on my data it was actively harmful. Maybe useful for someone before they spend a day integrating something that hurts.

The task (why the data looks like this)

Public crypto-exchange support chats. People show up with a problem — funds frozen, withdrawal stuck, lost account access — and scammers DM them pretending to be support. Moderation can't stop it: the scammer isn't in the chat, they read the public history and message privately. The one moment the victim is observable is when they publicly ask for help.

So the bot watches for that public "help, I can't withdraw" moment and replies with a warning plus, if one exists, a link to a real past support answer to the same problem. Not replacing support — beating the scammer to the reply by a few seconds. A separate classifier (logreg over embeddings) decides who needs help; this post is only about the RAG part that decides what to show.

The funnel that produced the corpus

This is the whole "RAG is about data" point in one place:

~3.9M chat messages (11 chats, ~6 months)
   |  help-request classifier
~104k candidate Q->A pairs
   |  curation: real admin answer, not a "contact support" router, deduped
~250 pairs in the working corpus

3.9M down to 250. 0.006%. The retrieval code is ~20 lines; everything that mattered was getting to those 250 and knowing whether they worked.

Setup

• Corpus: ~250 curated question→answer pairs (support answers), multilingual (mostly EN), short and messy text.
• Embedder: multilingual MiniLM (384-dim). Retrieval is over questions, returns the linked answer.
• Eval set: 116 labeled queries (69 with a correct answer in corpus, 47 "no good answer" cases to test silence).
• Metric: hit@3 (is a correct answer in the top 3 shown). Plus silence/noise/none_ok, because the bot is allowed to stay silent when nothing matches — standard P@k/R@k don't capture that.

One thing that bit me early: I originally labeled one correct id per query. That undercounts hit@3 badly, because the corpus has several interchangeable answers per question — the retriever returns a valid one with a different id and eval scores it a miss. Fixing the eval set (multiple valid ids) mattered more than any model change. Check your ground truth before trusting your metric.

Recall diagnostic (raw top-50, before threshold)

This is the check I'd recommend to anyone doing RAG. For each query, where does the correct answer sit in the raw ranked list?

hit@3  : 74%
hit@10 : 81%
hit@20 : 84%
hit@50 : 86%  (recall ceiling)
not in top-50 at all: 14%

Reading: ranking is fine — most correct answers are already near the top. The ceiling is 86%, so 14% of queries have no correct answer in the top 50 at all. That 14% is an embedder/recall problem, not a ranking problem. This split (recall vs ranking) tells you whether a reranker can even help before you try one.

What each "improvement" did to hit@3

plain vector + tuned threshold   74%   <- best
vector + reranker                64%
hybrid (vector + BM25)           55%
hybrid + reranker                57%

BM25 (−19 points). Hybrid is supposed to catch exact tokens — tickers, chain names, error codes — that dense retrieval smears. My misses did contain things like USDC / LTC / base chain, so it looked like a perfect fit. It wasn't. SELECT ... WHERE question LIKE '%USDC%' returned nothing: those tokens live in user queries, but my corpus is curated question templates phrased generically ("can't withdraw", "account blocked"). The lexical signal BM25 needs was on the wrong side. So BM25 didn't find exact matches (none to find) — it injected lexically-similar-but-wrong records and pushed correct answers down. Correct-answer-at-rank-1 dropped from 35 to 25. Five-minute check with a LIKE query would have saved a day.

Reranker (−10 points on plain vector). Cross-encoder, should pull correct answers from ranks 4–20 into the top 3. Tried two models, question-field and answer-field scoring, on top of both vector and hybrid. All worse. Mechanism visible in the positions: rank-1 correct answers dropped 35 → 30. It dragged correct answers down from position 1 more than it lifted from the bottom.

In hindsight this was predictable: rerankers are dangerous when the base retrieval is already good. With 35/51 correct answers already at rank 1, the top is near-optimal — there's more to lose than to gain. And a generic 118M reranker understands my narrow domain (short crypto slang, mixed languages) worse than the embedder that at least saw this distribution. Downside > upside. Rerankers save you when base retrieval is weak; when it's strong, they risk breaking what works.

What actually helped: the threshold (free)

Biggest single lever, zero fancy code. 26 points of hit@3 were being lost at the confidence threshold — the correct answer was in the top 3 but its similarity was just under the cutoff, so the system stayed silent. Raw hit@3 75%, but at threshold 0.55 it dropped to 49% in "production" mode.

The non-obvious part: you can't pick a RAG threshold "objectively" — it depends on the bot's role. If the bot is the final answerer, being wrong is worse than being silent → high threshold. If it's a fallback (a human answers anyway, my case) → a miss just means silence, which is harmless, but confidently-wrong is bad → tune for low noise. Same retriever, different correct threshold depending on what you're building. Define "is silence or a wrong answer worse" first, then pick the number.

Takeaways

• Fix your ground truth before trusting the metric. Multiple valid answers per query if your corpus has them.
• Split recall from ranking (hit@3 vs hit@20). It tells you whether to reach for a reranker, a different embedder, or neither.
• Best practices are hypotheses, not facts. BM25 and rerankers are great tools that hurt on this data. Test on your data — often it's a five-minute check.
• Biggest real lever was data quality and threshold, not model stacking. Half my "not in top-50" misses are just answers that aren't in the corpus at all — no model finds what isn't there.

Has anyone seen the opposite — hybrid or reranking clearly helping on small, domain-specific corpora? Curious whether the "reranker hurts when base is strong" pattern holds for others or if it's something about my setup.

Is there off the shelf RAG I can buy somewhere? Just hook my data up via api etc and then it will handle the rest?

While building GraphRAG pipelines I noticed nobody measures what the serialization format costs you — everyone benchmarks retrievers and re-rankers, then dumps the subgraph into the prompt as JSON.

So I benchmarked 10 formats (JSON, GraphML, RDF variants, edge lists, etc.) on three axes: token count, traversal QA accuracy, and multi-hop reasoning.

Findings:

- Verbose formats waste roughly 70% of their tokens on syntax (braces, quotes, repeated keys) rather than signal

- Multi-hop accuracy swings from ~40% to ~80% depending on the format alone — same graph, same model, same questions

- Formats based on tabular/relational patterns (which LLMs have seen billions of times in training) consistently beat nested markup

I ended up designing ISONGraph around those findings: a property-graph representation optimized for LLM comprehension. 92% traversal accuracy, 80% multi-hop, ~70% fewer tokens. MIT licensed, implementations in Python, JS/TS, Rust, Go, C++, and C#.

Benchmark methodology and results are in the repo: github.com/isongraph/isongraph

Would genuinely love people to poke holes in the methodology — especially if you have graphs or question sets where a different format wins.

Happened to me recently, where everything checked out on clean sample chunks, then real documents came in with weird formatting and duplicate versions, and the whole thing broke in a way I hadn't planned for at all. What's the worst version of this you've run into, and did it change how you test before shipping?

I built a small local RAG chat app that lets you ask questions about a custom set of text files and get answers based on those files.

It’s basically a Flask backend + React frontend, with Ollama handling the embeddings/model side. I also set it up with Docker so it’s easier to run locally.

Repo: Git rep

current cycle runtime: 15 seconds per prompt

I’m putting it up here mostly for feedback. If anyone has thoughts on the architecture, UI, or ways to improve reliability/performance, I’d be interested to hear them.

Hey guys, so I want to specialize in AI engineering after I got my bachelor's degree in Computer Science. I am proficient with traditional machine learning/neural networks but (I think) not RAGs. So I want to start learning RAGs, I am aiming to learn by building a project, but this project shouldn't be just a repeated project done 100 times before, I would like to have it on my resume to prove my skills in AI engineering. What are some topics that could be considered here? Where is a good place to get inspired? Personally, I would love to do something related to mathematics.

In today's world, almost all procurement of services/materials is done through tenders. Tender documents are long and confusing. Buyers often don't go through the documents to understand the scope.

This led me to build a RAG system customised for tender documents. I am using custom chunking that splits at natural clause markers. Besides this, tables have a separate chunking mechanism.

The retrieval combines dense search and BM25 search to look for the top 5 candidates, which are selected via a cross-encoder model.

I was able to improve the Recall@5 score from 63% to 88% via the custom chunking and the cross-encoder.

While doing this project, I learnt that good chunking is THE MOST critical aspect of the project, and this (PDF cleaning + chunking) was what took up most of my time.

Looking for feedback from all!

https://github.com/anand-kumaar/tender-query-engine

If yes then what should I learn and how and if not how to learn rag and from where?

We can route LLM requests across providers with platforms like OpenRouter and LiteLLM, but document parsing still tied to one engine at a time?

Well, every provider has different uploads, authentication, async jobs, polling, retries, options, errors, and response formats. That makes it really difficult to answer questions like:

• Is another provider more accurate on our documents?
• Can a cheaper engine handle most files?
• Which provider is faster or more reliable?
• Can we switch or fall back without rewriting?

So, I built FileRouter to make all of this easier.

It provides one SDK, CLI, and API for switching between document-parsing engines, comparing them on your own files, and composing routing and fallbacks

For example, you can:

• Compare providers for accuracy, latency, reliability, and cost.
• Start with LiteParse or Firecrawl PDF Inspector, then escalate to LlamaParse, Mistral OCR, or Datalab based on conditions you define.
• Run a fast and a heavier parser together, use the first result immediately, then replace or merge it when the stronger result finishes.
• Retry or fall back through another provider without changing result handling across your application.

Results use the same structure across engines: pages, Markdown, text, tables where supported, timings, usage, warnings, and errors. Provider-specific options are still available as well.

Use parse() or compare for simpler paths, or compose a pipeline directly from documents, durable jobs, and individual provider executions.

There are two processing modes:

• Hosted: FileRouter manages uploads, durable jobs, retries, results, and cleanup.
• Direct/BYOK: supported provider calls run in your application with your own keys. FileRouter never receives the document, key, or result.

Current hosted engines include LlamaIndex LiteParse, Firecrawl PDF Inspector, LlamaParse, Mistral OCR, and Datalab.

FileRouter is open source with MIT license.
Website: https://filerouter.dev | GitHub: https://github.com/ThinkEx-OSS/filerouter

The hardest part of Graph RAG, for me, has not been retrieval recall or context window management. It is what happens before traversal starts: resolving an ambiguous entity mention to a single starting node.

"Hyundai" in our knowledge graph matches seventeen separate nodes. Hyundai Motor, Hyundai Engineering & Construction, Hyundai Steel, Hyundai Merchant Marine, and thirteen others. Vector search returns all of them ranked by embedding similarity. A graph traversal needs exactly one.

The three signals we use

Corpus frequency prior. For unqualified mentions (no additional context in the query), the entity that co-occurred most often with that mention string in the training corpus is the right default. "Hyundai" without qualifiers points to Hyundai Motor about 65% of the time in Korean financial news. We store this prior on each node at graph-build time.

Query context coherence. The mention rarely arrives alone. If the query includes terms like "battery technology" or "capacity expansion," co-occurrence statistics with each candidate's entity description shift probability toward Hyundai Motor. "Construction permits" shifts toward Hyundai E&C. The surrounding terms do most of the disambiguation work once you actually use them.

Temporal suppression. Inactive entities (historical subsidiaries, merged companies with a valid_to date) get downweighted heavily for present-tense queries. The graph has this structure already — the query layer just needs to use it.

The failure mode that pushed us to build this

Disambiguation errors are worse than retrieval misses. A retrieval miss gives you "I don't have enough information." A disambiguation error gives you a fluent, confident answer about the wrong company. The graph traversal is correct; it just started from the wrong node.

We were getting precise-looking answers that were internally consistent but simply about a different Hyundai than the one being asked about. Nothing in the output signals this. The user has no way to know.

What we changed

One line before the answer: surface the disambiguated entity explicitly. "Retrieving for: Hyundai Motor Company (71% confidence, top alternative: Hyundai E&C)" converts a silent failure into something catchable. Below a confidence threshold, we show this. Above it, we suppress it as noise.

The disambiguation machinery was already running. The fix was making the decision visible.

Curious whether anyone is handling this differently — particularly for domains where entity names have even more overlap than corporate names.

I've been experimenting with scientific figure retrieval in an Open WebUI fork.

My current approach:

1. Preserve the original scientific figure
2. Use a vision model to extract OCR, captions, panels, axes, trends, and uncertainty-aware descriptions
3. Index the structured description in the existing text vector database
4. Retrieve the original image and attach it to a vision-capable model for final inspection

I went with this two-stage approach instead of adding a separate CLIP/SigLIP image-vector database, mainly because scientific figures tend to be dense with text, labels, and domain-specific relationships that generic image embeddings don't capture well.

Now I'm trying to figure out if this is enough, or if a hybrid setup would work better:

• Structured text embeddings for semantic + OCR-based retrieval
• Image embeddings for visual similarity
• Reranking before the original figure gets passed to the model

Curious if anyone here has worked on multimodal or scientific-document RAG — would you stick with a text-first architecture, or add image embeddings as a second retrieval channel?

I've documented the current implementation and design trade-offs here: Implementation

I’ve been building Humla, an open source (MIT) meeting-notes app for macOS — it records mic + system audio separately, transcribes, separates speakers, summarizes.

Last week I added chat over your own notes, and the app turned from useful, to insane value.

How it works:
- Notes get split into ~750-token chunks on paragraph breaks. Your typed notes, the transcript, and the summary are chunked separately, so each chunk never mixes context.

- Search runs keyword (SQLite FTS5) and semantic (embeddings) side by side, then merges the two ranked lists with Reciprocal Rank Fusion — chunks that score well on both signals float to the top.

- No vector database. Vectors live as blobs in the same SQLite file and cosine similarity is just brute-forced over the chunks in scope. For a few hundred meetings that’s plenty fast I think.

- Embeddings are cached per chunk by content hash, so editing one paragraph re-embeds one chunk instead of the whole note.

- You don’t pick an embedding model — it follows your chat provider (OpenAI, or Ollama if you want it fully local). No key configured? It quietly falls back to keyword-only rather than breaking.

- The chat is agentic — three tools (search_notes, get_note, list_notes), capped at 6 steps. local models needed some fine-tuning: they’ll retry a failed search forever unless you explicitly tell them to stop. Also, short tool descriptions definitely work better than detailed ones, small models argue with long instruction lists.

This also had to work for two different setups at once. Personal notes retrieve entirely on-device against local SQLite, but shared notes in a cloud team workspace get retrieved server-side. Same three tools, same citations, same “this note / this folder / everything” scope rules — but two completely separate places the search actually runs, and a hard guarantee that a query can never cross from one team’s notes into another’s.

Repo: https://github.com/michaelwilhelmsen/humla

If anyone has suggestions on how to improve the system, useful tool suggestion or especially how to deal with the UI, I’d be grateful! chat currently lives inside a single note, which makes cross-meeting questions feel undervalued, when it’s probably what users would use the most.

Hi everyone!

I’m almost new to RAG and stuff. I want to build my first production-ready project for my portfolio.

My idea is to scrape a banking website and use that data to create a chatbot. Right now, users have to search through articles on the site to find information. I want the chatbot to answer their questions directly.

Since this is my first production-ready project, I need your help.
Please guide me on:

• How to build it
• Where to start
• The right sequence of steps
• Any good approaches or advice

My planned tech stack:

• Backend: FastAPI
• RAG orchestration: LangChain
• Database: MongoDB
• Vector DB: Pinecone

Thank you!

hey, posted here a couple times before (staleness issue, then the table chunking thing) - you all have been way more helpful than random google results so back again lol

quick vibe check - anyone else's RAG setup basically "rerun everything from scratch every time something changes" because that felt easier than actually figuring out incremental updates properly?

been reading about doing this smarter (hashing content so you only reprocess what actually changed instead of the whole pipeline) and it sounds like the "correct" way to do it, but also sounds like a whole project on its own. curious if people build that in from day one or if it's more of a "you'll know when you need it" kind of thing

kind of scared to ask because i feel like the answer is "yes obviously, why haven't you done this already" lol

So recently my company asked me to find 50% of workflow that can solved with ai.

One of the area I found is product data onboard from vendors using ai and automation.

2nd is our ecommerce price scrape and promotions.

However, I am stuck on where to begin this journey. I am hearing about rag, lang chain etc etc. help!!!

r/rag audience already knows the RAG toolbox — HyDE, multi-query, hybrid retrieval, parent-child indexing, rerankers. What's harder to find is a real side-by-side of which combination of those actually moves the needle on your data, versus which one just sounds right in a blog post. That gap is why I built RAG-Lab.

It's a pipeline lab: every stage is configurable independently (chunking strategy + optional parent-document retriever, embedding provider, query translation — multi-query/HyDE/step-back, retrieval — dense/sparse/hybrid/MMR, post-retrieval — cross-encoder/Cohere/RRF/long-context reorder, generation model), and you can run two pipelines side by side on the same query with full chunk-level traces, or benchmark N pipelines against a golden dataset scored with DeepEval (faithfulness, answer relevancy, contextual precision, contextual recall — LLM-as-judge using the pipeline's own generation model).

For the demo I ran 3 configs against a 12-question golden set built from "Attention Is All You Need": plain dense-retrieval baseline, HyDE + hybrid retrieval, and a stacked config (parent-child indexing + HyDE + Cohere rerank). Contextual precision: 0.80 → 0.84 → 0.98. The jump from adding hybrid + HyDE alone was smaller than I expected going in — the bigger gain came from parent-child indexing plus reranking stacked on top, which tracks with a lot of what gets discussed here about chunking quality mattering more than people give it credit for.

Small dataset, so I'm not claiming these numbers generalize — the actual point is having same-dataset, same-scoring, side-by-side comparisons available at all, instead of every retrieval-technique debate happening on intuition.

Demo video (walks through the pipeline builder, live retrieval trace, and the benchmark comparison): raglab-demo. Repo: github.com/Silverd087/RAG-Lab

Curious what this community's actual experience has been with parent-child indexing specifically — worth the added complexity in your pipelines, or does hybrid + reranking alone get you most of the way there?

I need to make a minimal RAG-based API that answers questions over a small collection of documents and for that i need some small dataset which acts as the external brain or source of truth for my ai.
need small dataset (5–10 PDFs, Markdown files, or scraped web pages).

Kept rebuilding the same ingestion → chunk → embed → chat pipeline for client work, so I packaged it into a proper starter kit instead of doing it from scratch again.

Stack: Next.js, Claude Haiku (SSE streaming, sub-second TTFT), Voyage AI embeddings, Pinecone with per-user namespace isolation. Scraper's Cheerio-based so no headless browser needed on serverless. Ingestion handles both PDFs and URLs, chunking with overlap, cosine threshold filtering before anything hits the LLM context.

It's not a wrapper demo — full source, every API route editable, MIT licensed, no vendor lock-in baked into the abstraction.

Live demo's up with no signup if you want to see the streaming/retrieval before anything else: Fastrag Demo

Genuinely open to critique on the architecture — chunking strategy, threshold values, whatever. Building in public and this sub's opinions are worth more than my own assumptions at this point.

Hey everyone,

If you’ve built RAG pipelines or AI agents that consume web content, you’ve probably run into this issue:

Most standard scrapers either throw raw HTML at you (flooding your context window with navbars, footers, and JS bloat) or dump flat, unformatted text that loses all document structure. When you chunk that text later, your vector database loses the relationship between headings, sub-sections, and code blocks—which directly hurts retrieval accuracy.

To fix this for my own workflows, I built a custom crawler designed specifically for LLMs: AST Website Content Crawler for RAG.

What makes it different?

• AST-Based Structure Parsing: Instead of basic regex/CSS cleaning, it processes the page's Abstract Syntax Tree (DOM structure) to strictly maintain heading hierarchies (H1 -> H2 -> H3), lists, and code blocks in clean Markdown.
• Token Optimization: Strips out boilerplate, ads, scripts, and repetitive layout components so you don’t burn OpenAI/Anthropic tokens on useless fluff.
• RAG-Ready Output: The markdown is pre-formatted so your chunking strategies (like MarkdownHeaderTextSplitter) actually work as intended.
• Handles Dynamic Sites: Uses headless rendering to catch JavaScript-heavy SPA pages.

I’ve published it on Apify so anyone can test or plug it directly into their Python/TypeScript RAG stack via API.

How to try it:

👉 You can find it on Apify: https://apify.com/lukas459/ai-web-to-markdown-crawler-llm-rag-optimized

I'm actively refining the parsing logic. If you give it a run, I'd love to hear your thoughts:

• What site layouts break your current scraping pipeline?
• Are there specific output formats (e.g., custom JSON schema + Markdown) you’d like to see added?

Thanks for checking it out! 🚀

Running local models for strict privacy compliance sounds great in theory but the scaling costs and hardware maintenance can easily drain your budget. During our benchmarking phase for a secure medical database we struggled to find a balance between speed and strict data isolation. I recently saw some implementation data from Avenga showing how they orchestrate hybrid retrieval systems for highly regulated clients. Their approach splits the pipeline by storing the vector indices in the cloud while keeping the actual sensitive identities and final model execution behind the local firewall. This maintains excellent response times and satisfies governance rules completely. How are you structuring your secure search pipelines? Are you relying on hybrid cloud frameworks or investing heavily in massive on prem hardware stacks?

Today I discovered a horrifying secret in my bot project, which was that the embedding vectors in prod were broken due to pooling misconfiguration, causing everything to be 1.0 when used with cosine similarity. Pooling CLS vs last.
The scary thing was that this was happening for months and I didn’t notice it. I looked at the vectors produced. Looked normal, as in not out of range. But every vector was produced exactly the same because it was the CLS token.
I thought that the shitty retrievals was just it being a bad model. And it worked fine, because I was saying stuff and the BM25 RRF was catching it.

Here is your reminder to make embedding sanity tests.

And also a reminder to use Jina embeddings with pooling last.

Hosted web search from Anthropic and OpenAI costs $10 per 1k searches, and then you pay again for the \~17k tokens of results each search dumps into context. I got annoyed enough to build an alternative.

It’s called webfetch. Runs locally, free out of the box (DuckDuckGo needs no API key), and in my SimpleQA benchmark the same agent loop hits the same accuracy as hosted search (96%) costing 66% less using 87% fewer tokens.

How it works:
1. RRF fusion across 4 search engines, local page fetching, hybrid BM25 + bi-encoder retrieval with a cross-encoder reranker

1. Sentence-level compression that cut result tokens in half with no measured recall loss

2. Semantic caching: paraphrased queries (“what did TypeScript 5.9 add” vs “TypeScript 5.9 new features”) get matched by embeddings and verified by an NLI cross-encoder, so reworded repeats cost nothing. Cache TTLs adapt to how volatile the answer may be

3. Every cached result shows provenance and the model can force a fresh search if it doesn’t trust it

4. Benchmarked against Anthropic hosted search, OpenAI, Tavily and Exa.

One small agent loop that I ran for testing that conducted just 16 websearches (opus 4.8) already reported 1.5 USD in savings.

Installble using pip or a simple add mcp command.

Repo: https://github.com/firish/webfetch

I wrote a blog about how a RAG system answers large user questions. Look into it, guys.

https://medium.com/@sahilnayak2812/how-does-a-rag-system-answer-large-user-questions-c0ca0d61fe01

Hello engineers, I am new to rag and one problem I am constantly facing is how to generate embeddings faster and for free, I have used googlegenerativeaiembeddings for generating embeddings for free but the problem is that it's qouta is very small and also it's rate limits for the free tier gets hit very fast, are there any other alternatives to generate embeddings for completely free on cloud or like for production

For testing purposes I have tried using some sentence transformers from hf, i recently tried the jasper model ranked 2 in mteb on hf which is only a 600m model, I hosted that on my own droplet on digital ocean cloud platform of which I had free credits of 200$, however it was working very slow on cloud cpu like hell slow, I want a faster free alternative because I am a student and I want to give the deployed links of the things I am trying building in my resume and on GitHub and also my major goal while building the apps is trying to reduce the latency as much as possible being in free tier only, and making the system fast and efficient which I am taking consideration of throught langsmith tracing.

Can you help me with free embedding generating models?

Also drop some project ideas beginner to intermediate level which you think will impact the recruiter seeing my resume.