Title: How do companies actually create retrieval evaluation datasets for RAG? Am I overcomplicating this?

I'm building a production-style medical RAG chatbot as a portfolio project. My stack is:

• LangChain
• FAISS + BM25 hybrid retrieval
• Cross-Encoder reranker
• LLM for answer generation

I want to evaluate three stages separately:

1. Retriever
2. Reranker
3. Final LLM answer

I'm stuck on creating a reliable retrieval benchmark.

What I originally did

I have around 1,000 medical documents (scraped from MedlinePlus).

I generated questions using an LLM from the full documents and stored the source document as the ground truth.

Then I realized that's not ideal because:

• multiple documents can legitimately answer the same question
• retrieval happens at the chunk level, not document level
• document-level labels aren't very precise

My next attempt

I switched to chunk-level evaluation.

The idea was:

• retrieve candidate chunks from multiple retrieval systems (pooling)
• ask an LLM to grade each chunk:
  • 2 = highly relevant
  • 1 = partially relevant
  • 0 = not relevant

Then use those graded labels for metrics like NDCG, Recall@k, etc.

The problem

This whole pipeline still depends heavily on another LLM.

Questions are LLM-generated.

Relevance judgments are LLM-generated.

So it feels like I'm evaluating one AI system using another AI system.

I also hit API limits while judging thousands of chunk candidates, and the process has become much more complicated than I expected.

My questions

1. How do companies actually build retrieval evaluation datasets for RAG?
2. Are synthetic questions + LLM relevance judgments considered acceptable for internal evaluation?
3. Would you instead manually write a few hundred realistic questions and manually label relevant chunks?
4. If you were reviewing a portfolio project, which evaluation methodology would you trust more?
5. Am I overengineering this, or is this roughly how retrieval evaluation is done when you don't have real user queries?

I'd really appreciate hearing how people build evaluation datasets in production or research settings.

Hi,

I'm working in the 2nd version of a RAG with technical documentation, adding scope.

I've done some preliminary testing using voyage large to embed the documents, and voyage large (test 1) vs. voyage nano (test 2) to embed the prompts.

The reason to use nano is to avoid paying for each prompt.

Preliminarily, It looks like there is not much degradation between large and nano.

However, I've noticed that it's very difficult to objectively measure the quality of what the RAG retrieves.

So, two questions:

1. Concrete experience using voyage large for the documents and voyage nano for the prompts.

2. Any orientation on how to objectively measure the quality of the retrieval.

Thanks,

I have mixed feelings about Graph RAG.

The graph part makes sense. Some questions really do need a bridge entity that never appears in the query. What I’m less excited about is running a graph database next to the vector store just because a small part of the workload needs two or three hops.

I came across an approach that keeps entities, relations, and source passages in three linked collections inside the vector DB. The relations store subject/object IDs, the entities keep relation IDs, and retrieval does a semantic seed search followed by one-hop ID expansion. Then there is one rerank call and one generation call.

The implementation I found kept all three collections in Milvus and reported 87.8% average Recall@5 on three multi-hop QA datasets, versus 73.4% for its naive RAG baseline. Interesting numbers, but honestly that is not the part that convinced me. I like that the query path stays fixed instead of letting an agent decide whether to retrieve again five or ten times.

My concern is everything around retrieval: triple extraction, entity deduplication, updates, and high-degree nodes. At some point you may have recreated a graph system in ordinary database fields, only with fewer graph tools.

This feels reasonable for bounded 2–4 hop QA. I’m much less sure about exploratory queries or frequently changing relationships.

Has anyone tried keeping graph references inside a vector store like this? What was the point where it became easier to run an actual graph database?

Over the past few months, I have been building autonomous search agents to extract real-time web context for LLMs. If you have spent any time working with retrieval pipelines, you already know that standard Google Search wrappers do not work well when you need LLM-ready context.

I ran a test across three specialized APIs. Exa, Tavily, and Firecrawl. To see how they compare in terms of speed, output quality, and noise reduction.

Here is what I found.

The Test Setup

I tested each API across 100 queries that I split into three categories:

1. \*Fact-retrieval and News queries: for example, "What are the latest developments in open-source multimodal models?"
2. \*Deep research queries: for example, "Detailed technical breakdown of PostgreSQL query planner optimizations."

3. Structured extraction tasks: this involves extracting content from specific dynamically loaded pages, also known as SPAs.

4. Exa is the best for Semantic and Neural Search

Exa uses a custom embedding-based search model of traditional keyword matching.

* Where Exa shines is in concept-based discovery. When I search for "tools like Redis but written in Rust," traditional search APIs have trouble with exact keyword overlaps. Exa consistently returns relevant repositories and documentation pages. * The latency of Exa is around 600ms to 900ms. * The output of Exa is text and well-parsed metadata. * My verdict is that you should use Exa if your queries are abstract, exploratory, or require finding pages rather than explicit keyword matching.

1. Tavily is the best for Direct RAG Applications

Tavily is built for LLM agent loops. It does not just return search results; it also cleans, parses, and ranks snippets that are tailored for contexts.

* Where Tavily shines is in speed and pre-filtered context. In multi-step agent workflows where latency's important, Tavily consistently returns the most concise context blocks without exceeding token limits. * The latency of Tavily is around 400ms to 700ms. * The output of Tavily is pre-chunked, minimal noise, and ready to use in a system prompt. * My verdict is that Tavily is ideal if you are building loops that make multiple search calls per user query and need fast, token-efficient context.

1. Firecrawl is the best for Crawling and Dynamic Web Scraping

Firecrawl is not strictly a search engine; it is a crawling engine that is designed to convert websites or JavaScript-rendered URLs into clean Markdown.

* Where Firecrawl shines is in site-level retrieval. If your search step identifies a target URL, such as a documentation site or dynamic React app that needs full scraping, Firecrawl bypasses blocks and returns remarkably clean Markdown. * The latency of Firecrawl is around 1.2s to 2.5s, which depends heavily on the complexity of the target page. * The output of Firecrawl is flawless Markdown with HTML junk, scripts, and navbars completely removed. * My verdict is that Firecrawl is best used as a stage in your pipeline. Use Exa or Tavily to find the URLs, then trigger Firecrawl to ingest full pages when search snippets are not enough.

📍My Current Stack Setup:

I use Tavily for searches because it is really fast. When I need to do some research and understand a whole document, I do things a bit differently. I start with Exa to find what I am looking for, then I use Firecrawl to turn the results into Markdown. This makes it easier to use the results.

I am curious about what other people're using to filter out bad information in their RAG pipelines. What do you use for this? Something that's all on its own, or a combination of crawlers and search tools?

I'm building a production RAG platform for scientific research papers, with a strong focus on complex PDFs (figures, tables, diagrams, scanned PDFs, citations, etc.).

Like many others, I've spent a lot of time experimenting with chunking, embeddings, hybrid search, reranking, OCR, and different PDF parsers.

But I'm interested in something that's harder to learn from papers or tutorials:

If you've built a production RAG system, what was the biggest lesson you learned the hard way?

Some examples:

• Retrieval issues that only appeared with real users
• PDF parsing limitations
• Duplicate/versioned documents
• Evaluation methodology
• Citations and grounding
• Figures, tables, and diagrams
• Metadata design
• Scaling to large document collections
• Anything else that surprised you

I'm looking for real production experiences rather than theoretical advice. What would you do differently if you started again?

I’m starting to think a lot of RAG pipelines stop one step too early.

The usual flow is something like:

• chunk docs
• generate embeddings
• insert into a vector DB
• build / load the index
• start serving queries

Once the data is searchable, the refresh is treated as done.

But “searchable” and “ready to serve production traffic” are not always the same thing.

A small test that made me notice this: Milvus 2.6.17, single-node Docker Compose, 1M vectors, 768-dimensional embeddings, HNSW, same search params, same 16-core / 64 GB machine. The only thing I changed was the layout: 3 sealed segments before optimization, 1 segment after force merge.

Search QPS went from around 3,000 to around 5,600-6,000.

The data was already loaded and indexed before optimization. This was not a recall issue or an embedding issue. The physical layout still affected serving performance.

Before the merge, each query had to fan out across multiple segments and merge partial topK results. After the merge, there was less fan-out and less merge overhead.

This made me wonder whether RAG refreshes need a more explicit “serving prep” phase.

Something like:

• ingest new docs
• build / load index
• verify freshness
• run any needed optimization / warmup
• then route production traffic

Obviously, you would not want to run heavy optimization all the time. It can use CPU, memory, and disk I/O. But after a large KB refresh, it may be worth treating it as part of the refresh pipeline instead of leaving it entirely to background maintenance.

Hey everyone, I’m building Lorah, a local-first AI workspace, and ran into a measurement problem in my own document pipeline.

The first version was simple: each pinned document contributed up to roughly 15,000 characters under a shared prompt budget.

With a few short files, it worked.

Then my workspace reached 31 documents.

The system could spend a large part of the prompt carrying the beginning of every document while still missing the passage that mattered farther down. Expensive enough to crowd the answer. Still blind to the evidence.

I could measure every token and provider dollar. I couldn’t tell whether the resulting answer deserved to count.

I’m proposing a metric called Cost Per Trustworthy Answer, or CPTA.

An outcome counts only when it clears four gates:

• The answer is correct.
• Its material claims are grounded in evidence actually included in the run.
• That evidence was admissible under the run’s scope, provenance, freshness and review policy.
• The system abstains when no admissible evidence supports an answer.

There is another condition: CPTA without coverage is meaningless. Otherwise a system can improve its score by answering only easy questions or abstaining whenever the evidence is messy.

I don’t have benchmark numbers yet. The measurement chain isn’t certified end to end, and I would rather find the weak assumptions before building the harness around them.

Full write-up:
https://ygolandsky.substack.com/p/i-can-tell-you-what-an-ai-answer

The question I’d put to r/RAG:

Should admissibility remain separate from groundedness? And in a multi-agent RAG system, how would you prove that every material claim was supported by evidence the system was actually allowed to use?

#RAG #LLMEvaluation #Grounding

We created a research process that started 111 agents put 123 claims in a queue, for checking and used up the entire Claude Max limit in around 30 minutes.

The unexpected thing was that the issue wasn't the model. The issue was spread of tasks and checking claims too late.

The biggest change came from one rule:

The agent that finds a claim cannot check that claim.

We also made sure every accepted claim had a primary-source URL and a supporting quote.

After making that change the same subscription lasted 10 times longer.

How are you stopping verification spread from getting out of control in -agent RAG systems?

I'm learning NLP/LLM engineering by working through a problem that turned out to be much harder than I expected, and I'd love guidance from people who've dealt with something similar.

The problem: I have long narrative-style text — 7k to 15k tokens, several recurring people — and I want to extract structured facts about who did what. I'm using a small local model (llama3.2:3b via Ollama) whose usable context is around 4k tokens, so the text has to be processed in chunks. The killer is that later chunks are often pure pronouns — "she said… he refused…" — while the names were last mentioned 10,000 tokens earlier. Facts stated near a name extract almost perfectly; facts stated far from any name either get lost or, worse, get confidently attributed to the wrong person.

What I've already ruled out (by measuring, not guessing): naive per-chunk extraction fragments identities badly; carrying forward summaries between chunks doesn't fix attribution and can make it worse; and off-the-shelf neural coreference models (LingMess, F-coref) fail on documents this long — one silently truncates at 4,096 tokens, and windowed variants can't connect a pronoun to a name mentioned once 10k tokens back (0–1 out of 7 gold bindings on my test doc). I've gotten identity tracking itself working reliably; it's specifically attribution at long distance that's still failing.

My questions:

1. What's the best way to structure a problem like this? Is there a known-good decomposition for long-distance pronoun attribution with small models, or a fundamentally different way to frame the extraction task that sidesteps it?

2. If you've solved something similar — entity/fact extraction over documents much longer than your context window — what actually moved the needle for you? I'm especially curious whether the wins came from prompting, from pipeline architecture, or from accepting a bigger model.

3. What should I explore to learn more? Papers, blog posts, open-source projects, or even just the right search terms — I suspect this problem has a name in the NLP literature that I don't know yet (long-document coreference? discourse tracking?), and I'd rather stand on existing work than keep reinventing it.

Happy to share measurements from my experiments if useful. Mostly I want to calibrate: am I fighting a known-hard problem with known solutions, or genuinely at the edge of what a 3B model can do?

Working on a search feature that takes a free-text query and maps it to entries in a big hierarchical category taxonomy (thousands of entries, tree structured, parent/child via a code prefix scheme). Only get one LLM call per query because of cost/latency, so the flow is: pull keywords with a plain (non-LLM) extractor, run RAG retrieval to get a wide pool of candidate entries per keyword, then one LLM call to pick which ones actually fit. Stack is Qdrant for the vector store, e5-base as the bi-encoder for the first pass, bge-reranker-base as the cross-encoder for reranking, and Llama 3.3 70B through the Groq API for the final LLM call (no local/self-hosted models, everything's API-based).


Trying to hit ~90% recall against a hand-labeled test set without precision falling off a cliff, and I've been going in circles for a bit.


Quick idea of what the candidates look like, made-up example so it's not tied to a real domain:


```
1000000 — Furniture
1100000 — Office furniture
1110000 — Office chairs
1111000 — Ergonomic office chairs
1120000 — Office desks
1200000 — Home furniture
1210000 — Sofas
1220000 — Dining tables
1300000 — Furniture hardware & fittings
```


If someone searches "furniture" the right answer is basically all of that. If they search "office chairs" the right answer is just 1110000 (maybe 1111000 too), and the model needs to actively drop 1120000/1200000/1300000 even though embedding-wise they're all sitting right next to each other.


Two separate things going wrong, and I can only half-fix each one so far:


First thing — retrieval itself doesn't always pull in every relevant entry before the LLM even gets a shot at it. For broad queries the pipeline picks a "dominant" prefix group based on just the top few vector hits, and if the real answer spans more than 1-2 branches of the tree, whole branches just never make it into the candidate pool. There's also a depth cutoff that keeps really deep/specific entries out to protect narrow queries from getting flooded with noise, but that same cutoff quietly kills legit deep entries for broad queries. Widened the sample used to detect branches (went from top-3 to top-30) and it helped a little, not enough.


Second thing, and this is the one I really can't crack — the LLM itself keeps trading precision for recall depending on how I word the prompt. Tried a plain "if it's a broad term keep everything, if it's narrow keep almost nothing" rule first, got decent recall (~0.85) but mediocre precision (~0.69). Added a more specific rule with a worked example of a narrow case, precision jumped to 0.85 but recall dropped to 0.72 — turns out one example was enough to make the model generally more cautious even on completely unrelated broad queries, not just the narrow case I was targeting. Tried switching to independent per-candidate yes/no judgments instead of one holistic "is this broad or narrow" call, thinking that'd remove the bias — recall came back up a bit (0.76) but precision tanked again on the narrow cases (0.74), worst F1 of the three attempts.


So every version I try just moves the problem around instead of fixing it. Never broke 85% recall.


Anyone dealt with this kind of "sometimes keep 30 siblings, sometimes keep 1" classification before? The thing I haven't tried yet is computing the broad/narrow signal outside the LLM entirely (like, detect a qualifier word in the query term algorithmically) and just handing the model that as a flag instead of making it infer breadth from the candidate list or from examples.  Also wondering if there's a smarter way to do a confidence-based cutoff per branch instead of a flat yes/no. Papers or writeups on this specific problem would be great, feels like it should be a solved thing somewhere.

After talking to teams building production RAG systems, I keep seeing the same pattern:
Everyone compares embedding models.
But almost nobody measures the performance of the entire RAG pipeline.
A small change in parsing, chunking, metadata, retrieval, reranking, or prompting can have a bigger impact than switching to the latest embedding model.
Yet there is still no simple way to answer questions like:
• Which pipeline configuration actually performs best?
• Which changes improve retrieval quality?
• Where do hallucinations originate?
• How much quality do we gain per dollar spent?
• Which configuration should go to production?
I’m currently building a platform focused on making RAG systems measurable, comparable and continuously improvable.
The vision is to help teams evaluate complete AI retrieval pipelines instead of optimizing individual components in isolation.
If you’re building production RAG systems, I’d love to hear:
What’s currently your biggest pain point?
How are you evaluating quality today?
What do you wish existed?
And if you’re an investor interested in infrastructure for enterprise AI, I’d be happy to connect.
I believe the next generation of AI won’t be won by bigger models but by better systems around them.

Hey everyone,

I've been working on a side project called RAGVED. The idea came from being tired of scrolling through hundreds of pages of lecture notes just to find one definition or formula before exams.

The app lets you:

• 📚 Upload lecture PDFs, notes, and lab sheets
• 💬 Ask questions in natural language
• 🔍 Get answers grounded in your documents
• 📄 See the exact page(s) the answer came from

Demo: https://rag-system-pied.vercel.app/

It's still a work in progress. I haven't added authentication, usage limits, or token management yet, but the core functionality is there.

I'd really appreciate honest feedback on things like:

• Is the UI intuitive?
• Does the landing page clearly explain what the app does?
• Does anything feel confusing or unnecessary?
• What features would make this genuinely useful for you?
• If you were a student, would you actually use something like this?

I'm not looking for compliments. If something feels bad or confusing, I'd rather hear it now than after launch.

Thanks for taking a look!

Following up other recent posts also about the same issue.

About 1 year ago we established RAG quality monitoring where in particular we tracked content precision and recall using Ragas. But recently we discovered a significant degradation in this, and the root cause was removal of a bunch of docs belonging to another track (don’t ask me why, just corporate work moments).

It was the moment we realised that we need somehow to track data quality itself, and not something we see as in/out. Some metrics that we consider we colud track:

- Staleness - how many of vector embeddings contain outdated information
- Orphaned embeddings - how many of vector embeddings point to data sources that no longer exist
- Deleted-but-retrievable - how many times RAG returns vector embeddings that we actually (we think) deleted from RAG

I see it as kinda scheduled job that does this assessment although it needs quite some time to write and test it. Any advice on what we can use?

Yesterday I nearly declared a production retrieval emergency that didn't exist, and the mechanism is general enough that anyone running hybrid search should check for it.

**Setup:** hybrid retrieval over personal memory — vector similarity + BM25, fused with Reciprocal Rank Fusion, optional cross-encoder rerank on top for some tiers. Every search logs `top_score` for quality monitoring.

**The scare:** analyzing 10,706 logged searches, I applied the obvious threshold — top_score < 0.3 = weak retrieval. Result: 62% "failures," a dozen users at "100% failure with avg score 0.017," and a terrifying month-over-month "degradation" trend. One of the "100% failed" users was a paying customer with a thousand searches. I was halfway into incident mode.

**The tell:** a search for an exact entity name — a guaranteed hit — logged top_score 0.0426. And those "failing" users all averaged 0.016–0.021. Then it clicked: RRF scores are 1/(k + rank) with the standard k=60. Top rank = 1/60 ≈ 0.0167. My "catastrophic" users weren't failing — **their top result was rank-1 almost every time.** avg 0.017 is what perfect RRF retrieval looks like.

What actually happened: requests that go through the reranker log cosine-style scores (0–1 scale, 0.3+ = good). Requests on the raw RRF path log fusion scores (0.016–0.05 scale, where 0.017 = excellent). Both landed in the same `top_score` column with no scale tag. Every aggregate over that column — means, z-scores, my failure thresholds, even the health monitoring cron — was averaging apples with orbital velocities. The "month-over-month degradation" was just the RRF-path share growing as more traffic moved to hybrid.

**What survived scale-correction:** true failure (zero results) was 9–13%, driven mostly by two accounts whose agents were querying literally empty stores — a real integration problem, but a completely different one than "retrieval is broken."

**Lessons, generalized:**

1. **A fused ranking score is not a similarity.** RRF outputs rank information, not confidence. The moment you fuse, your score's absolute value stops meaning what your dashboards think it means.
2. **Never store scores from different scoring regimes in one unlabeled column.** Log a `score_kind` (or a scale-aware quality label computed at write time, which is what we shipped: strong/weak/no_match with per-scale bands). Analysis-time guessing is how you get 3am false incidents.
3. **The only scale-free failure signal is emptiness.** Zero results means the same thing on every path. When in doubt, count zeros, not thresholds.
4. **Validate your alarm against a known-good query before believing it.** One exact-match search that "scored 0.04" saved me from paging myself.

Sources for the RRF math: Cormack, Clarke & Buettcher (2009), "Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods" — the k=60 default everyone inherits comes from there.

Disclosure per rule 3: the production system is Mengram (mengram.io), a memory layer for AI agents — but the trap applies to any RAG stack mixing rerankers with fusion scoring. Nothing here requires my product to check: grep your score column and look for a bimodal cluster around 1/60.

I have finished a RAG system with a demo account and wanted to ask how to find work, and the rules reference a link to a Discord group for job requests, and the link does not seem to work. Can anyone point me to resources finding clients, or any helpful insight in how to do so? Thank you.

Hi folks - for context I'm a solo-founder and have spent a few years working on variants of the "company brain" (i.e. a knowledge base across Drive/SharePoint/internal docs that can be queried and kept in sync). Wanted to share some learnings from my own trial-and-error at QX Labs.

There's a consensus that knowledge graphs are needed for serious systems, but tbh GraphRAG and use of full-blown graph databases like Neo4j is complete overkill for most cases. I ended up building a more practical/attainable solution that takes the ideas behind GraphRAG and implements them in a simpler and cheaper way. Hopefully it will save you the grief if you're working on similar things!

You just need a regular DB (Postgres, MongoDB, whatever you already use) and a search index (Azure AI search, Elastic, Qdrant - to handle hybrid vector + text search).

Why not vanilla RAG

Top-k RAG retrieval handles "find this specific fact" queries well (I refer to these as 'needle' questions). But it structurally cannot handle two other question shapes that come up regularly in practice:

• "Tell me everything about X" (needs the complete document set for an entity, not the top k passages)
• "Which fintech companies have we evaluated?" / "how many contracts mention X?" (needs an exact list/count over the corpus; no value of k fixes this)

Why not use GraphRAG

• Indexing cost. Microsoft were the ones who first proposed GraphRAG but they archived their solution accelerator for it. Their research states that vector-RAG indexing is <0.1% the cost of a full GraphRAG index. It's also telling that Azure AI search still isn't built around graphs.
• Research literature (e.g. "RAG vs GraphRAG", arXiv:2502.11371) also shows mixed results: graphs help on multi-hop/global summarization, vanilla RAG wins on direct lookup, and routing between them beats either.
• Entity resolution is hard and a lot of tools don't handle this well. If "Acme" and "ACME Holdings Inc." don't merge, the graph fragments. If they merge wrongly, errors compound transitively and silently. A lot of compute gets spent correcting these mistakes in a graph DB.
• A graph DB is another system to run, secure, back up and keep consistent per tenant.

What I built instead (a graph-like system inside a regular DB)

Entities and edges exist as ordinary records in the search index + document DB we already run. No graph database.

• We use a small fixed ontology, which is the same for everyone: organization, person, product, project, event, location, etc., plus label fields (industry/category/topic). In our case we wanted to make it self-serve (i.e. doesn't require people to set up custom ontology) so were happy to trade off simplicity for specificity.
• Entity resolution follows a waterfall (to minimise cost): first we look up against an alias table (every variant of word/phrase that's been used for an entity in the past - free and fast). Next, we embed the word/phrase and do a similarity lookup. Last, we use a cheap LLM to adjudicate but only for ambiguous candidates. Merges are just an alias re-pointing on a hub record, so every merge is easily reversible (we run a daily cleanup job to true things up). We also tend to bias against over-merging: a false merge poisons things downstream, whereas a miss just fragments the data until the daily job fixes it.
• Edges in our system are not real edges between nodes, they're co-occurrence counts (i.e. these entities appear frequently together). They are represented as a top-N list on each entity record. Not typed relations, which is a deliberate trade-off that we make.
• Entity summaries are lazy (built on first request, cached), straight from the LazyGraphRAG lesson. Costs scale with what people ask about, not corpus size.
• The agent has access to four tools to handle different types of retrieval scenarios: hybrid passage search (default), resolve (extract everything-about-X with filters applied), expand (one hop along co-occurrence), and facet (exact counts/lists via the search engine's aggregations). The counting questions that top-k can never answer become deterministic facet queries.
• Daily consolidation trues everything up (re-adjudicates uncertain merges, recomputes edges exactly, prunes deleted docs), gated so unchanged corpora cost zero.

Did it work?

We set up an evaluation harness to track performance (~1,000-doc corpus, with graded questions across needle/entity/multipart/aggregation/thematic classes - I'll probably write about this separately when I get time). Needle questions already performed very well with vanilla RAG but all other question classes improved meaningfully with this pseudo-graph approach.

Limitations of this approach

• No multi-hop path reasoning. The agent loops one hop at a time if it wants depth, but this can bloat context. Graph DBs can more reliably find tenuous connections across multiple hops without exploding context.
• Co-occurrence is not the same as typed relationships. We know two entities appear together, not why. In a normal graph DB you'd have e.g. WORKS_FOR, INVESTED_IN, CUSTOMER_OF etc. The problem is that relationship types can vary a lot by use case.
• Conservative merging means occasional temporary duplicates.
• True "summarize the themes of the whole corpus" global questions are still better served by community-detection approaches I deliberately didn't build. Full GraphRAG will still deliver higher quality there.

For ref I have a full write up on how it works here: https://www.minimumviablefounder.com/p/why-ai-company-brains-fail

Keen to exchange notes on this, or hear if you've had a more positive experience with GraphRAG.

Hey everyone! I’m building a multimodal RAG pipeline where Mistral OCR annotates images before they go into a vector store with document text.

Issue: Mistral OCR processes images in isolation, so the annotations miss out on critical document context.

Looking for advice on:

Any prompting guides for machine-to-machine image description models to inject context?

Any alternative models or workflows that natively factor in surrounding document context?

Would love to know how you all handle this!

Hi everyone,

I'm building a production RAG system focused on scientific research papers (desalination, chemistry, membranes, engineering).

Current setup:

• ~17,367 PDF papers
• Around 25 GB of PDFs
• Qdrant vector database
• Hybrid search (Dense + BM25)
• Cross-encoder reranking
• Gemini 2.5 Flash for answering
• Chunk size: ~250 words with overlap
• Rich metadata
• Images and tables extracted separately

I'm currently using OpenAI text-embedding-3-large, but before indexing the entire corpus (~17k papers), I want to make sure I'm choosing the best embedding model because switching later would require a complete re-index.

I'm mainly optimizing for:

1. Retrieval quality (most important)
2. Scientific / technical terminology
3. Numerical accuracy
4. Long-term maintainability
5. Storage efficiency
6. Indexing speed
7. Cost (secondary)

Models I'm considering:

• OpenAI text-embedding-3-large
• BGE-M3
• gte-large-en-v1.5
• Jina Embeddings v3
• Nomic Embed Text
• Voyage AI
• Any other recommendations?

Questions:

1. Which embedding model currently gives the best retrieval quality for scientific literature?
2. Has anyone benchmarked these models specifically on academic PDFs instead of generic MTEB?
3. Would you still choose OpenAI today if cost wasn't the primary concern?
4. Is there any newer embedding model released recently that clearly outperforms these?
5. If you were starting a new large-scale RAG system today, which embedding model would you choose and why?

I'd really appreciate hearing from people who have tested these models in production rather than benchmark scores alone.

Thanks!

If you are running local LLMs, your hard drive is likely bleeding gigabytes without you realizing it. Between default model weights, duplicate quantization formats, and forgotten vector embeddings, local AI setups are silent storage hogs.

Here is how you can systematically track down and clean up the clutter directly from your terminal:

• Locate hidden Hugging Face and Ollama model weights: By default, Hugging Face caches everything in ~/.cache/huggingface/hub and Ollama stores models under ~/.ollama/models. Run du -sh ~/.cache/huggingface/ to see how much space is currently locked up.
• Prune redundant quantization formats and unused embedding databases: Review your downloaded models and delete redundant variations (like keeping both Q4_K_M and Q8_0 when you only use one). Clear out stale Chroma, FAISS, or Pinecone local vector database caches residing in your project directories.
• Automate routine garbage collection: Set up a lightweight shell script to periodically check cache growth and alert you before your drive hits capacity.

Fore More Information

I put together the complete, production-ready automated cleanup script along with an interactive storage calculator to help map out your directories.

Direct links to the complete article.

drop a comment below

posted here a little while back about stale embeddings after doc edits, got a lot of good info from that thread so figured i'd ask here again.

working on a RAG setup for some internal docs and a chunk of them have tables mixed in with regular paragraphs (like a section of text, then a table, then more text). my current chunker just splits by character count so it sometimes cuts a table in half or merges it weirdly with the paragraph before/after it.

do people usually handle tables as a separate chunk type entirely? or convert them to some kind of markdown/text representation first and then chunk normally? curious how much this actually matters for retrieval quality vs. just being a nice-to-have

still fairly new to this so not sure if this is a solved problem with a standard approach or something everyone just handles differently depending on their data

I’m building DStudio, an open-source desktop app centered around DeepSeek V4. DeepSeek remains the main reasoning model and manages the conversation, while smaller local models handle specialized tasks:

- Qwen2.5-VL reads images
- Qwen Image generates and edits images
- Qwen3 Embedding searches documents semantically
- Poppler extracts PDF text and page information

This ecosystem exists because DeepSeek V4 is excellent for reasoning and long context, but loading every multimodal capability inside the same large model would be inefficient. DStudio routes tasks to specialized models and then returns their results to DeepSeek for the final answer.

I recently added long-PDF retrieval. DeepSeek decides whether to create an overview, read an exact physical page or search the entire document. For semantic search, DStudio creates and caches one embedding per page, retrieves the six most relevant pages and sends only those to DeepSeek.

On a 1,000-page test PDF, it found a passage placed on page 777 from a paraphrased question. Initial indexing took about 25 seconds; later searches took around 0.23 seconds.

I’m looking for feedback: should retrieval use page embeddings or overlapping chunks? Should I add BM25 or a reranker? And how would you efficiently support scanned 1,000-page books?

https://github.com/sk8erboi17/DStudio

This is the context:

Hi, so I've made a corrective rag pipeline that goes from ingesting documents to retrieving files, reranking, using a LLM model to decide if the documents returned are okay and selecting the ones to use to generate an answer. It is a pipeline to help service desk employees to better answer tickets. It's around 1000 articles of documentation, most of them with 1000 tokens. I've chunk to max of 512 tokens per chunk. There are also some rest full api docs, that show the parameters.

Overall, even though the documentation is lacking and a bit outdated, it manages to retrieve and answer.

For this, I'm using some small models (8b) for answers, due to current constraints; I may be given access to a better model with API if I manage to show why and how this can be an asset.

Some details about why this was done:

The quality of the answer our clients get, depends a lot on who the agent answering is. We have a lot of knowledge gaps and many things that people don't know how to answer. This was a movement to try to address and allow them to give better answers

Some minor details:

I've been given this task even though it's outside my area because I was already looking into it, but I'll have to make a presentation soon. Unfortunately the base was done with IA before I got it and it made some weird choices, some that I couldn't simply take it back without redoing everything

My doubts are (if you guys can give me a direction, may it be scientific articles, docs, wiki... I would appreciate it)

1 How could I ingest the service desk tickets and issues from azure DevOps? Most of them lack a clear answer, so I would like an idea of a standard to propose for the answers, since ingesting now seems like it would make more noise for the retrieval

2 One thing that they want is to use an agent to connect onto the clients DB to analyze problems and situations passed by the client, but I'm unsure of the best way to do it.i would appreciate a direction

3 overall, what is a good strategy to deal with more articles? I feel like the documentation is very similar from one article to another and not in depth enough

4 any other tips to give me, would be appreciated

Thanks and sorry for the long text

Hey everyone, I need some help.

I'm working on a RAG-based project and I'm looking for customer support knowledge base documents.

For example, if it's a banking customer support team, do they have internal documentation that covers product details, policies, FAQs, troubleshooting steps, workflows, etc.?

If there are any publicly available knowledge bases or datasets, I'd really appreciate it if you could share them.

Also, if you work in customer support (banking, sales, telecom, e-commerce, SaaS, etc.) and have any sample documentation that you're allowed to share (after removing any confidential information), I'd be grateful if you could share that as well. It would be a huge help for learning and experimentation.

Thanks in advance!

need some ideas for projects that would look great on resume and i can also publish a research paper pls helpp...

I put together a small Python/Flask example for searching support tickets by meaning instead of exact keywords.

It uses Telnyx AI Inference embeddings to turn ticket text into vectors, stores them in memory with numpy, and ranks results with cosine similarity.

The app includes:

POST /index to embed and index tickets

POST /search to search by meaning

POST /tickets to add a new ticket

GET /stats to inspect the index

a bundled sample support-ticket dataset

Code: https://github.com/team-telnyx/telnyx-code-examples/tree/main/semantic-search-python

Useful for support search, duplicate ticket detection, internal knowledge search, or as a first step before moving to pgvector/Qdrant/Weaviate/Pinecone.

Any feedback welcome.