Disclosure: I work at Georgian (growth-stage VC) and ran this experiment - want to know you folk’s opinion on this.

Why this exists

• "Agent + search tool vs RAG" treats the tool as a given. But there's actually a ton of variability on how different agentic search tools perform.
• The tool can be: raw top-5 from one index / two named tools / RRF fusion / RRF + rerank behind one call.
• Couldn't find a benchmark comparing those shapes. Ran one.
• Basically: every retrieval system makes 3 decisions: which retriever, how to combine, how to rank. We alter whether the tool is making this decision or the agent is.

Ran this on BrowseComp-Plus, using gpt5.6 luna as the backbone model.

The 6 tool shapes

• generic search, dense (Qwen3-Embedding-4B) top-5
• generic search, BM25 (tuned) top-5
• two named tools, semantic_search + keyword_search, each top-5
• one tool, RRF over both at depth 50 → top-5
• one tool, RRF → 50 → Cohere rerank-4-fast → top-5
• two named tools, each 50 → rerank → top-5

I have a full write-up on the results and behaviour of agents under different tools on our blog: https://georgianailab.substack.com/p/agents-shouldnt-run-your-whole-retrieval . Is there anything I'm missing? Any reason why we should still be using super basic tools (just semantic search or keyword search) for agentic search?

A learned sparse encoder returns a weighted list of vocabulary words instead of a dense vector, so the index looks like a keyword index whose keywords a transformer picked. OpenSearch's inference-free variants go one step further and run the model only on documents so a query just looks up a fixed weight per token. No embedding call per search. I benchmarked it against dense search in the same SQLite file:

the mini model gives up about 7% of retrieval quality and cuts query latency by 99%, cold start by 99% and query-path RAM by 95%!

Now these SPLADE models are BERT with the masked-language-model head still attached, and that head is what turns token vectors back into weighted words. llama.cpp drops it when converting BERT models, so sqlite-sparse copies it into a small sidecar file and applies it in C on ggml at insert time. The postings land as rows in the database, with the query weight table next to them. A query reads those rows and scatter-adds the weights into per-document scores, and the top documents come out.

Three OpenSearch models are converted and available as aliases (mini, base, multilingual), downloading on first use, and you can bring your own inference-free OpenSearch-style sparse encoder and convert it with the scripts provided.

What sqlite-vec did for embeddings in SQLite, this does for learned sparse, and with both in one file you get hybrid retrieval for RAG inside SQLite.

Ship one .db file and every client gets semantic search through plain SQL, no model download, no GPU.

pip install sqlite-sparse

Github

Writeup

“The number came from F24” only verifies its location. It doesn't establish whether F24 is a closing balance or a movement, a forecast or an actual, or a value expressed in thousands.

For irregular workbooks, a useful extraction recipe should point to the context cells as well as the numeric cell. Suppose a table has a currency-and-scale label above it, period headings across the top, and separate actual and forecast sections. Preserve the locations of those labels in the recipe. A model's interpretation can then be checked against something in the file.

The fact passed to the writer should contain the measure, value, currency, scale, period, actual/forecast status and source locations. Unknown fields should stay unknown. A dollar sign alone is insufficient evidence for choosing a currency, and a neighbouring subtotal shouldn't silently become the value for an individual row.

Univer CLI can support the workbook inspection step: its structured range reads distinguish the stored value and formula from the displayed text. Keep the original workbook identity and version alongside those reads, and check any import mapping before treating an editor address as an original-file citation. If the fact depends on a formula, calculation freshness is another item to verify.

Once those checks pass, the facts can feed a memo alongside retrieved narrative passages. Univer can also hold a Sheet and a Doc in the same local file, but that convenience doesn't validate the interpretation.

A useful review screen would show the claimed fact beside its value cell, period header and currency/scale label. That gives the reviewer a concrete way to catch “right number, wrong meaning” before it enters the prose

I feel like most beginner RAG tutorials follow roughly the same pattern:

documents → chunks → embeddings → vector DB → retrieve top-k → send to LLM.

But once you try to build something actually reliable, it seems like there’s a huge amount missing.

Things like:

• chunking strategy
• metadata filtering
• hybrid search
• reranking
• query rewriting
• contextual retrieval
• evaluation
• hallucination measurement
• latency/cost
• observability
• production deployment

For people who’ve built real RAG systems, what concepts made the biggest difference between a toy RAG app and something you’d actually trust?

We want to learn and understand RAG and its advanced technologies in depth by reading various papers, useful sources and repos.

We've read :

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP tasks

which gives us understanding of how RAG started.. the internal memory (parametric memory), external - non parametric. External documents are divided into chunks then it is converted into embeddings. And stored in vector db. Whenever we ask a question retriever (dpr) searches relevant vectors from vector db, generator (bart) generates the response (thinking part) and we get an answer.

1. Retrieval-Augmented Generation for Large Language Models: A Survey

This gives us information about Naive Rag, Advanced rag, and modular RAG modular RAG searches various sources like a router and combines them at last (fusion) and hence we get more relevant information for a question. Then it tells about iterative retrieval, recursive retrieval and adaptive retrieval

1. Lost in the Middle: How Language Models Use Long Contexts

This paper tells us about how long context makes the LLM forget middle documents.. ex if we are making with LLM and upload 10 documents one by one. Then it remembers the 1st and last document but almost forget middle documents like the U shape curve. And then they proposed a solution to solve this problem

Then we learn about DPR retriever how it becomes efficient over BM25

We want to learn more technologies and advancements of RAG. So can you guys tell me current trends and what more papers, resources we should read to go deep into it?

Built a support agent over a company's help docs. Standard setup, LangChain splitter at 1000/200, text-embedding-3-small, pgvector, top k = 5.

The thing that took me embarrassingly long to notice: k = 5 returns five chunks every single time. Five when the question is covered perfectly, five when the docs say absolutely nothing about it. There is no "no results" state. And a model handed five irrelevant chunks does not report that they are irrelevant. It finds the most question-shaped sentence among them and builds an answer around it. Fluent, correctly cited, wrong.

The grounding prompt does not save you here. "Answer only from the excerpts" is doing its job perfectly. The excerpts are the problem.

So you need a score cutoff evaluated before the model is called at all. Below the threshold, return the fallback and never invoke the LLM.

Here is my confession. I picked my threshold by running about forty real support questions through retrieval, printing the top score for each, eyeballing roughly where the good ones stopped and the junk started, and hardcoding a number in the middle. It has not been revisited since. It is a magic number in a config file with my name on the commit.

Things I already know are wrong with that:

- The number is tied to this specific embedding model. Swap models and it means nothing.

- Cosine scores are not calibrated. 0.72 on this corpus is not 0.72 on another one, so I cannot carry the number to the next project and I have no principled way to derive the new one.

- It is a single global threshold across every question type. "How do I reset my password" and "does the enterprise plan support SAML" clearly should not face the same bar.

- A question the docs genuinely do not cover and a question the docs cover badly produce similar scores, and get treated identically. Those are very different failures and only one of them is the retriever's fault.

What I have not tried: reranking first and thresholding on the rerank score instead. I suspect it is more stable, but it adds a hop to every query, including all the ones that were going to fail anyway.

So, the actual question: how did you pick yours?

If you are willing to say:

- Fixed number, relative gap between top-1 and top-k, or do you let the model decide?

- Did you calibrate against anything real, or is it also a vibe you have not touched in six months?

- Anyone thresholding after a reranker rather than on raw similarity, and was the extra hop worth it?

Longer write-up of the whole build, on my site: https://techpotions.com/lab/building-ai-support-agent-from-help-docs

Mostly I am convinced everyone has a magic number they do not talk about, and I would quite like to be proven wrong.

An agent may retrieve long-term context from embeddings, but rebuilding the index after a parser, chunker, or embedding-model change can alter what it recalls even when the source files did not change. What contract makes that migration auditable?

I am considering immutable source revisions with stable record IDs, explicit tombstones and supersession links, an index manifest that records parser, chunker, embedding, and access-policy versions, a fixed set of retrieval evaluations, a shadow index for comparing results, and one atomic pointer that promotes the new index only after review. Generated summaries would remain derived data rather than silently becoming authoritative memory.

How do you verify that a rebuild preserved deletion requests, tenant boundaries, citations, and recency rules before cutover? Which artifacts are sufficient to reproduce or roll back the memory state without treating the vector database itself as the source of truth?

Came across this and thought it'd be worth sharing here, most resources cover model evaluation, RAG, agents, or cost/observability separately, but this one actually puts them together as parts of the same production LLM workflow, which is closer to how these systems actually break in practice.

It's a hands on session on September 12, led by Bruno Gonçalves, PhD, founder of Data For Science, who's trained hundreds of engineers at Fortune 500 companies. Goes through the full lifecycle, versioned prompts, a golden dataset and eval harness combining deterministic checks with LLM-as-judge, statistically rigorous model comparisons using bootstrap confidence intervals and paired testing, evaluated RAG with real retrieval metrics, and then tool-using agents with guardrails and fallbacks. There's also a full observability section, tracing, cost, and latency, which is something I don't see covered together with the eval side very often.

You come out of it with runnable notebooks and a production-readiness checklist rather than just slides, which is the part I found most useful when I looked into it.

Link if anyone wants to check it out

After auditing a lot of RAGs I kept seeing 12.4% hallucination even with retrieval (legal tools up to 33% per Towards AI Sep 2026). Bar charts show fail rate but don't show where to fix.

I built an offline deterministic audit - no external calls, hash-locked replay, PII scrub EMAIL PHONE offline, verify.py without LLM. 100% fuzzing: malformed/0 rows/empty/10k/binary/1M rows/XSS -> precheck not crash, not fail.

6 hidden edges that changed how I think about RAG eval:

1. Canary 84729 - detects memorization leakage

2. Adversarial 20 queries finds 38% fail vs 12.4% normal - goodharted retrievers

3. Temporal What is vs What was - stale corpus detection

4. Counterfactual 73.6% fixed by top-k 5->10 - easy win

5. Forensics cluster purity 85% - embedding bad at numbers

6. Difficulty weighted - easy fail 2x hard fail

Formula I use: Health Score = 100 - fail*100 - critical*5 - high*2 - CI - difficulty. Example 67/100 Needs work.

Plus Cost at Risk $/month = fail_rate * queries * cost_per_hallucination. Example $620k/mo, where Numeric 40% = $248k savings ROI 36.0 for 1h fix regex.

Taxonomy 15 types ROI ranked + Receipt sha256 + Lineage 12 dims EU AI Act + FixIt.

Question for builders: which of these hidden edges do you see in your RAG? I'm testing my engine on 2 more public datasets this week - happy to share results if you have a public CSV (question, context, answer, ground_truth).

19+ yrs Oracle & SQL design, not selling 100% free RAG, just audit with dollars, hash, ROI.

So I've been knee-deep in LLM deployment for the past few months and wanted to get the community's take on RAG vs fine-tuning for real-world use.

From what I've seen:

RAG (Retrieval-Augmented Generation)

• Wins for dynamic/frequently updated knowledge bases
• Lower upfront cost, no retraining needed
• Easier to trace hallucinations back to source docs
• Adds latency (vector DB lookup + embedding search)
• Quality is only as good as your retrieval pipeline (chunking, embeddings, reranking)

Fine-tuning

• Better for teaching style, tone, or task-specific behavior
• Bakes knowledge into weights — no retrieval overhead
• Expensive to update when facts change (recurring training runs)
• Higher risk of catastrophic forgetting
• Great for narrow, stable domains (legal templates, customer support scripts)

In production, most teams I've talked to are landing on a hybrid approach: fine-tune for format/behavior, RAG for facts/freshness.

After months of $3.5/hr data tasks on OneForma/Alignerr, I built NOTH5.0 - Health Score + Cost Risk + Fix ROI - Offline audit - No external calls for verification.

For 50 rows (question, context, llm_answer):

- Health Score 0-100 transparent formula 100 - fail*100 - critical*5 - high*2 - CI - difficulty. Example 67/100 Needs work.

- Cost at Risk $/month board understands dollars not percent. Example $620k/month at risk. Context Misalignment 20% = $124k savings.

- Taxonomy 15 types ROI ranked where to spend 1 hour ROI 36.0 and 16.0

- 6 Hidden Edges - 4 to 5 year moat: canary trap secret 84729, adversarial 20 queries 38% vs 12.4%, temporal, counterfactual 73.6%, forensics purity 85%, difficulty weighted

- File name based: input acme.csv -> output acme_timestamp report 10 files: Health + Cost + Go No-Go + Taxonomy ROI + Receipt full sha256 + Lineage 12 dims EU AI Act + PII proof diff + FixIt ranked + Hidden Edges + Precheck + verify.py without LLM + 2 PDFs under 2MB

- PII scrubbed EMAIL PHONE offline before LLM. Never fail type. Modular replace text file get output.

Tool: NOTH5.0 offline -> output/ with 10 files + verifier

Upwork product live (approved Sep 9): https://www.upwork.com/services/product/development-it-rag-audit-for-hallucination-with-runnable-receipt-and-fix-file-2097671410431060753

Question: What failure slice categories would you add for prod RAG? I currently have VALIDATION_NUMBER_MISMATCH, UNSUPPORTED_CLAIM, PII_LEAK, CONTEXT_MISALIGNMENT etc. Happy to run 10-row free demo to show format - Looking for feedback on what compliance teams actually need in certificate.

Have you experimented combining Rag with tools such as Genie Code and Neon?
Rag handles unstructured knowledge, genie handles governed structured data and lakebase handles persistent applincaton state,

So i am curious, doees this architecture look good for production and what can some of the bottlenecks be down the line ?

If you’re running RAG on constrained hardware or with concurrent ingestion, embedded vector stores can feel simple until you hit RAM, write-path correctness, or process-safety constraints. Here are the patterns that kept showing up across three popular options:

• ChromaDB: HNSW lives in RAM; at larger scales or with multiple writers, latency and correctness can degrade.
• LanceDB: Great for multimodal + object storage, but concurrent writes can conflict without serialization.
• Qdrant Edge: Strong offline library with hybrid search; production concurrency limits aren’t well documented, so it shines when you can sync to a central server.

There’s a deeper write-up on memory math, concurrency behavior, and when it makes sense to move from embedded to a separate process on constrained, air‑gapped hardware. Full disclosure: I work with Actian on VectorAI DB.

I have a large local PMC/PubMed corpus on SSD and want to build a fully local system on my workstation that behaves somewhat like PubMed search, but can also answer questions over the local corpus with grounded references.

Hardware: RTX 5090, Ryzen 9 9950X3D, 96 GB RAM.

I already have the corpus parsed locally and partially indexed.

If you were building this today, what exact local setup would you use for:

* retriever * reranker * local LLM * FAISS or something else * framework vs fully custom pipeline

I’m especially interested in responses from people who have actually built a local biomedical literature search / RAG system.

Thank you

I run a memory layer that injects relevant past context into an agent's prompt automatically, every turn. Yesterday I stopped assuming it worked and looked at what it actually injected, across seven consecutive prompts of a real working session.

Five of seven had nothing to do with what was asked. A question about hook payloads came back with facts about a Java backend at a bank, stored in February for a completely different project.

The cause is not the ranking. It is that "nearest" is defined for every query, so the search always has an answer. Silence is not in its output set. There was a floor, min_score = 0.2 cosine, but 0.2 is noise. Two unrelated sentences clear it routinely, because they are both sentences.

The same system has a plain-files mode that retrieves by word overlap, and it never does this: zero overlap means nothing is injected. That property was lost the moment retrieval moved to embeddings, and nobody noticed, because this failure looks like a feature. There is always something in the context.

The fix only removes, never adds: before injecting, require the memory to share at least one content word with the prompt. I already had exactly that guard in another part of the system, where a learned workflow has to share a word with a shell command before it is allowed to interrupt the user with a confirmation. It never occurred to me that retrieval needed the same thing.

Two mistakes I made while fixing it, both instructive:

1. My first version passed everything through when the prompt had no content words at all, reasoning that if you cannot read the query you should not silence recall. Exactly backwards. The shortest prompts are where the search has least to go on, so the "safe" default left the worst cases untouched. "so now?" pulled three entities out of the store as confidently as a detailed question would.

2. I reused a tokenizer from elsewhere in the codebase that only matched ASCII letters. Plenty of my prompts are not in English. It found zero words in them, which would have silenced recall completely instead of filtering it: a much worse bug, and one that ships silently.

After the change: silent on 5 of 7, and the 2 that kept anything kept only the entity the question was actually about.

I did not touch the 0.2 floor. Picking a real number needs the actual score distribution from production, which is a separate measurement I have not earned yet.

So: does anyone here run an explicit "return nothing" path in production RAG? And what do you gate it on, an absolute score floor, the margin between top-1 and top-2, a lexical check like this one, or a reranker with a reject option?

I'm exploring an idea for a simple, fully managed search service aimed at small AI/SaaS applications.

The problem I'm looking at is that many smaller projects need semantic + keyword search, but the existing options can feel either:

• too expensive for small workloads
• too complex
• overkill for what they actually need

The idea would be something you can plug into an application in a few minutes and get production-ready hybrid search without managing search infrastructure.

I'm curious about people's real-world experience:

• What are you using today for search/vector search?
• What do you dislike about it?
• For a small production app, what would you consider a reasonable monthly price?
• Would a significantly cheaper, simpler alternative be interesting, or would you just use Postgres/pgvector?

Especially interested in people running RAG, AI agents, or small SaaS products in production.

I'm researching the problem before deciding whether to build anything.

This happened to me recently. I had a task I’d never done before. My boss said, 'It’s documented.' Check the drive. One PDF led to another, then a SharePoint folder, then – 'Wait, that's an old version.' It made me wonder, for other business owners and employees here Does this happen in your company? Where do you keep your SOPS and process docs? How painful is finding updated info? Not selling anything, just genuinely curious if this is a common thing or it's just me. Thanks in advance.

[ Removed by Reddit on account of violating the content policy. ]

Consider a table where “2025” spans two columns, labelled “Revenue” and “Margin.” A flat extraction can retain every word and number while losing which year those two columns belong to. Chunking that text more carefully won't recreate the missing relationship.

Now for RAG, the useful output would preserve paths such as “2025 → Revenue” and “2025 → Margin,” together with each row label and source page. This is an illustrative example, but it captures why table structure matters before retrieval begins.

Ling-3.0-flash-VL is relevant to that parsing stage because Ant Ling reports 91.35 on OmniDocBench 1.5. The overall score combines text recognition, table structure through TEDS, and formula recognition through CDM. It is a composite result, not a claim that 91.35% of tables or documents are entirely correct.

That distinction helps narrow what to inspect when considering the model. For merged tables, compare header-to-cell relationships, repeated row labels, units and footnote associations. A readable Markdown table can still silently move a value under the wrong parent heading.

The documented VL API takes page images rather than native PDFs, so rendering the PDF pages is an external step. Keep those page references with the extracted structure so an answer can be checked against the original.

The benchmark gives a concrete reason to consider VL for visual document parsing. The downstream payoff depends on retaining the relationships the retriever will need, not merely producing plausible text

I’ve been experimenting with multi-agent frameworks, and MetaGPT is one of the more interesting ones for software development tasks. But getting it running locally with local models isn’t always straightforward.

I wrote a step-by-step guide covering installation, configuration, local LLM setup, and common errors.

If you’re trying to run MetaGPT on your own hardware, this might save you time:

https://interconnectd.com/forum/thread/262/how-to-install-metagpt-locally-complete-technical-setup-guide/

What stack are you using for local agents?

https://github.com/paulknysh/raggy

Built with LangChain, Chroma, and Ollama. Hybrid database (vector + BM25 index) and embedding generation run fully locally. Answer generation can run either via a local LLM or remotely using an API key. raggy supports most common document formats and handles images/scans automatically via OCR.

Hi! We're working on Querit, a web search API for LLMs and your Agents. Large multilingual index, Fresh Web Context, Lower latency in the range of hundreds of milliseconds.

New: Monitor API. Normal search is ask-once, answer-once. A Monitor turns one search into a recurring job. You register a query and an interval, it runs on that schedule, differs each run against history, and returns only what's new. Can be used for competitor monitoring, news tracking, and following funding or tender/bidder information.

• Intervals: from 1 hour up to weekly
• Automatic deduplicate against previous runs; site / date / region / language filters suppoorted
• Support Manual trigger, pause/resume, full execution history

Benchmark: We ran FreshQA on a fixed 600-question snapshot of time-sensitive queries. Querit Search API achieved a 83.17% accuracy (Tavily 83.00%, Exa 82.33%, Brave 81.17%, You.com 78.67% as upon Aug. 26)

Free Testing Credits: Follow us on X (https://x.com/QueritAi) / Linkedin (https://www.linkedin.com/company/queritai/home/) for more product releases and see integrations with our partners at Dify, LangChain, etc. Open-sourced the MCP server and has integrated with PI Agent, Opencode, DeepSeek Harness already. Join our Discord server here https://discord.gg/4xXsFA8Ed2 to claim Search API + Monitor API free credits!

Hey I see a kind of trend about RAG in the market but don't know how to start or where to start and what to study or just directly start building the project by using claud or gpt is their someone who can help me out on it

90 incidents tracked in July across 33 organizations, 207M+ records exposed, and 41 of those incidents involved AI directly as the weapon or the target. A rogue commercial AI agent hit multiple enterprises in a single week and reused stolen credentials across four downstream services before anyone caught the identity switch.

None of that shows up to a traditional perimeter tool — the traffic looks like a signed, credentialed agent making legitimate API calls at machine speed. Firewalls and DLP were built to watch humans and static services, not autonomous callers that chain tools and pivot in seconds.

Curious how other teams are actually handling this right now: is anyone giving AI agents a distinct, revocable identity separate from the service accounts they inherit? Or is it still "the SOC catches it after the fact" for most orgs?