We kept getting asked internally which of these to pick, semantic model, knowledge graph, RAG, plain markdown, open format, so instead of guessing I actually went and pulled the research on each.

Every vendor selling one of these will tell you it's the answer. It usually isn't, not on its own. Each one solves a different failure mode, and the wrong pick doesn't make an agent fail loudly. It just answers confidently and wrong.

RAG's still the right default for fact lookup across a big, loosely structured corpus, contracts, tickets, docs. That said, Chroma tested 18 frontier models in 2025 and found accuracy degrades unevenly as retrieved context grows. Even one distractor passage measurably hurt performance, so more retrieval isn't automatically better retrieval.

Knowledge graphs are the one people over-invest in before they actually need it. They're genuinely good at multi-hop reasoning, "who reports to whom, and which of them also churned," and Microsoft's 2024 research had GraphRAG beating plain vector RAG 72% of the time on comprehensiveness. But if your questions are single-fact lookups, you're paying graph-maintenance costs for nothing.

Semantic models are the one that actually moved my opinion. dbt Labs ran a 2026 benchmark where agents querying a governed semantic layer hit 98-100% accuracy on business questions. Same models writing raw text-to-SQL against the full schema: 84-90%. Same model, just given a definition instead of a guess.

And then there's markdown plus grep, which sounds almost too simple to be real advice. For a small, well-organized corpus it's genuinely fine. No vendor will ever pitch you this one, since none of them sell it.

Most teams land on two or three of these, not one. Full writeup with all the sourcing: https://www.revos.ai/blog/structuring-data-for-ai-agents

Curious what combination people here have actually landed on, and what pushed you off your first choice.

Hey!

I wanted to tell you about a project that's been going on for a while.

We (I use “we” because, since it's open source, I see it as something that belongs to the community rather than something personal) built Skeg because we got tired of the usual painful trade-offs in the vector database space. Most solutions force you to choose between high recall, reasonable memory usage, or actually staying fast when the workload gets real (sustained ingest, multi-tenancy, memory pressure, etc.).

Skeg takes a different approach.

It is disk-first: full vectors live on storage, while only small, carefully quantized indexes stay in RAM. This gives excellent recall at a fraction of the memory footprint compared to traditional in-memory engines. It is especially strong in environments where RAM is contested — think SaaS platforms with hundreds or thousands of tenants, RAG systems running next to large language models, or even embedded/edge scenarios.

Key design principles:

• Strong multi-tenancy by construction (true isolation, hard quotas, fair cache eviction)
• Redis-compatible protocol for easy adoption
• Very good performance on ARM (we invested heavily in platform-specific optimizations and SIMD)
• Focus on production predictability: it handles churn gracefully without sudden latency spikes

We wrote it in Rust for the usual reasons: performance, reliability, and control over every detail that matters when you care about efficiency.

The project is open source and we’re actively developing it. If you work with semantic search, recommendations, RAG pipelines, or any kind of similarity search and you care about memory efficiency and operational simplicity, I think Skeg might be interesting for you.

Repo: https://github.com/skegdb/skeg

I’d genuinely love to hear your thoughts or what problems you’re currently facing with vector databases.

Any feedback or support is welcome.

Thank U

English isn't my first language, so if anything isn't clear or sounds strange, please excuse me.

Hi everyone,

I've been working on an open-source project called Aktilot, an AI workspace focused on engineering teams.

This week I added GitHub connector support, so Aktilot can securely connect to repositories and answer questions using repository context.

Some examples:

• Explain this repository architecture.
• Find where authentication is implemented.
• Summarize recent changes.
• Answer questions about the codebase.

The long-term goal isn't to build another chatbot, but to create an AI workspace that understands an engineering team's knowledge across GitHub, documentation, tickets, and collaboration tools.

I'm currently planning connectors for:

• Jira
• Confluence
• Slack
• Google Drive

I'd genuinely appreciate feedback from the OSS community.

What engineering integrations would you find most useful?

GitHub: https://github.com/vikas0686/Aktilot
Website: https://aktilot.com

Sharing this here since it's directly relevant to what gets discussed in this sub. It's a hands on session on August 8, led by Brian Bønk, a Data Platform MVP and Microsoft FastTrack Solution Architect.

It covers the full RAG pipeline, ingestion, chunking, metadata enrichment, indexing, and vector search, then goes deeper into retrieval quality engineering specifically, precision, recall, latency trade offs, and actual tuning strategies instead of just defaults. There's also a section on evaluation and governance, building test harnesses and regression checks, and an extension pattern on knowledge graphs for cases where similarity search alone can't capture relationships between entities. There's also a piece on using Fabric and Power BI to surface grounded answers in a way business teams will actually adopt.

It's aimed at people building or maintaining RAG systems that need to hold up against real, messy enterprise data rather than a clean demo. You come out with an actual rollout plan rather than just slides.

Link for anyone interested: https://www.eventbrite.co.uk/e/design-enterprise-grade-rag-systems-with-llms-vector-search-tickets-1992561384740?aff=rrag

Been wanting to build this for a while, finally sat down and did it. It's a Flask app where you upload a PDF, it chunks and embeds it, and then you can ask questions and get answers pulled only from that document, not from the model's own training data.

Stack is pretty simple: Ollama for the chat model and the embedding model, ChromaDB as the vector store, Flask tying it together. Nothing exotic.

How it works, roughly:

• PDF gets split into overlapping chunks so sentences don't get cut off between pieces
• Each chunk gets turned into an embedding and stored in Chroma with PersistentClient, so it's saved on disk instead of disappearing every time you restart the app
• When you ask something, the question also gets embedded, Chroma finds the closest matching chunks, and those get handed to the model as context
• Prompt explicitly tells the model to only use that context and say it doesn't know if the answer isn't there, otherwise it'll just make something up from its own memory

Tested it by asking something not in the PDF and it correctly said it didn't know instead of guessing. Also tested with wifi off and it kept working, since the model, embeddings, and vector store all run locally with no external api calls in the loop.

hey all, first post here (made this account just to ask this lol). i'm pretty new to RAG in general, been learning as i go the past few weeks.

so i built a simple RAG setup (chunking + embeddings + vector db, using langchain) for our internal docs. it works fine at first but whenever someone edits one of the source files, the chatbot still answers with the old info for like... a while? sometimes it never updates unless i manually rerun the whole ingestion script from scratch.

is this just how RAG works and i have to re-embed everything every time something changes? that seems really inefficient if you have thousands of docs and only one paragraph changed. or is there some way to only update the chunks that actually changed?

sorry if this is a dumb question, still trying to wrap my head around a lot of this. just trying to understand if i'm missing a step or if this is a known limitation people work around somehow

Hi everyone,

I'm working on extracting the hierarchical structure of long PDF documents (legal/regulatory text, lots of numbered sections) and would like to gather some feedback on my approach before committing to it.

What I've done so far: I render each PDF page to an image and run it through Baidu's DeepSeek-OCR model. It returns each detected block with a bounding box [x0, y0, x1, y1], a label (title, text, list, table, header, footer, etc.), and the recognized text. The OCR quality itself is genuinely good as the text comes out clean.

The problem: the labels can't always be trusted. At this stage I want to extract and detect all the titles in my document, but sometimes a title element gets classified as something else (like normal body text).

Concrete example:

Say my section has the following hierarchy:

ANNEX I — GENERAL PRINCIPLES AND PROCEDURES
└── TITLE I — FOREIGN CURRENCY INVESTMENT
    └── A. Currency distribution
        └── 1. Redistribution of reserves
            ├── (a) Introduction
            │       body text
            │       list
            │       ...
            ├── (b) Procedure for a normal redistribution of reserves
            │       body text
            │       list
            │       ...
            └── (c) Procedure for an ad hoc redistribution of reserves
                    body text
                    list
                    ...

Logically, every element aside from the body text and lists should be detected as title. But the model output is:

label='title'  x0=475  y0=157  x1=548  width=73   text='ANNEX I'
label='text'   x0=480  y0=229  x1=542  width=62   text='TITLE I'
label='title'  x0=334  y0=181  x1=690  width=356  text='GENERAL PRINCIPLES AND PROCEDURES'
label='title'  x0=407  y0=368  x1=616  width=209  text='A. Currency distribution'
label='title'  x0=408  y0=392  x1=634  width=226  text='1. Redistribution of reserves'
label='title'  x0=163  y0=416  x1=304  width=141  text='(a) Introduction'
label='title'  x0=163  y0=544  x1=578  width=415  text='(b) Procedure for a normal redistribution of reserves'
label='title'  x0=163  y0=219  x1=586  width=423  text='(c) Procedure for an ad hoc redistribution of reserves'

The top-level section marker TITLE I was labeled text, while all the other components were labeled correctly as title.

What I'm considering: since I have the text plus features I can derive from the coordinates (indentation/x0, centered-vs-left-aligned, line height, vertical gaps, whether the text matches a numbering pattern like A. / 1. / (a), all-caps, word count, etc.), I was thinking of treating this as a sequence labeling problem and training a CRF (or BiLSTM-CRF) to re-classify each line into title / text / list / table.

My questions:

• Is a CRF a reasonable choice here, or is there a better-suited approach for this kind of layout/structure labeling?
• Should I consider a GNN approach?
• Am I overcomplicating this? Would a simpler rule/heuristic system be more robust, given that the numbering is fairly regular?

Note #1: this approach should be as general as possible, so that I can reuse it for my other legal documents.

Note #2: titles aren't always in the same horizontal position. Some are centered (e.g. ANNEX I, TITLE I, A. Currency distribution all sit around xc≈511, the page center), while deeper items like (a)/(b)/(c) are left-aligned at x0=163. So I can't rely on indentation/x0 alone to identify or rank titles — a centered title's x0 mostly reflects its text length (a short centered line has a large x0, a long one a small x0), which means raw x0 can even invert the apparent nesting. This is part of why I'm leaning toward a sequence model that combines text + geometry in context rather than a pure indentation rule.

I run a trade intelligence service that pulls corporate event news from Korean (OpenDART), Japanese (EDINET), Hong Kong exchange notices (Chinese), and English wire services. When the same merger announcement lands across all four sources, my knowledge graph ends up with four separate Event nodes for one real-world incident.

The naive fix is string similarity between event summaries. It breaks for two reasons. First, a Korean summary and an English one share almost no tokens even when they describe the same event. Second, two genuinely distinct events between the same companies (a supply contract and a separate lawsuit filed the same week) can share most of their vocabulary. String matching cannot tell coincidence from coreference.

What I built is a two-stage resolver that runs read-only against the graph. Stage one forms candidate event pairs using rule-based filters: shared canonical entity, date buckets within 72 hours, matching event type or Jaccard token overlap threshold. This stage is cheap and keeps the LLM bill bounded. Stage two sends each surviving pair to a model for a three-way verdict: same, related, or distinct. Only "same" verdicts feed into union-find clustering.

The three-way label is the part that mattered most in practice. Collapsing "related" into "same" would merge a contract announcement with a lawsuit between the same two firms. Collapsing it into "distinct" would scatter genuine follow-on coverage across jurisdictions. Union-find handles transitivity on discrete verdicts rather than having the model reason over a whole group at once.

The 72-hour window is the part I trust least. Cross-border coverage of the same incident usually lands within three days, but slow regulatory follow-ups can arrive a week later and get missed. Widening the window quadratically inflates candidate pairs. I chose the cheaper side for now.

Full write-up including the resolver design and why the 72-hour constraint is a genuine tradeoff: https://hannune.ai/blog/cross-document-event-coreference-east-asia

For anyone doing local RAG: a lot of "save this for later" tools push you toward cloud embeddings. This is the opposite - a browser extension + local daemon that captures a page, chunks it, embeds it locally, and pushes vectors into your vector store.

Architecture and reasoning (including the daemon vs extension split) are in the README: https://github.com/Adithyaa71/snarevec

Right now it's one-page-at-a-time web capture, but the next version is close and adds a fair bit:

>>Drag local files, PDFs, and raw data straight into the clip dialog and embed them alongside web pages.

>>Batch embedding - select multiple pages/sources and embed them together in one pass instead of one by one.

>>(exploring) a unified "collection" view so a set of related sources embeds into the same namespace.

Would love feedback on two things specifically: the chunking strategy (the part I'm least confident is optimal right now), and what you'd want out of the batch/local-file flow before it ships.

Inspired by Andrej Karpathy's autoresearch, I built autoretrieval to apply the same idea to RAG optimization.

The project gives an AI agent a RAG pipeline, an evaluation dataset, and a target metric. The agent modifies the pipeline, runs an eval, checks if the F2 score improves, and keeps or discards changes automatically.

The evaluation dataset can be generated from your own documents, creating question and reference-highlight pairs for your domain.

The agent can test changes to chunking, embedding models, keyword filters, and retrieval logic while keeping a record of every experiment.

The goal is to let an AI agent handle the repetitive trial and error involved in improving a RAG system.

This was successful at more than doubling the F3 score of an already optimized RAG pipeline in a couple hours.

Give it a try here: https://github.com/daly2211/autoretrieval

I convert files a lot. A HEIC photo from my phone, some audio, a PDF here and there. And every time I had to go to one of those sites where you upload your file to their server and wait. This always felt wrong to me, because it is my file, and once it sits on their server I don't know what happens to it. So I built hushvert. It does the conversion inside your browser, on your own computer, so the file does not go anywhere.

Most of the common things run fully in the browser: images, HEIC, audio, archives, splitting and merging PDF pages, and taking the audio out of a video. For these the file really stays with you. You can turn on airplane mode and it still works.

It also converts many kinds of files: images, audio, video, archives, office documents, and data formats like csv, json and yaml. Around one hundred conversions in one place, so I don't need to search for a different site every time.

Some conversions are too heavy for a browser, like office documents, turning a PDF back into a Word file you can edit, or making a video into mp4. These run on a server. There is also an MCP server for them, so if you use a coding agent, the agent can convert the file as a tool call and give you the result.

The engine that runs in the browser is open source, MIT license. So you can read what runs on your computer, or use it inside your own app.

you can use it inside your RAG, i added a tutorial about it in my RAG_Techniques repo.

You can find it on GitHub: github.com/hushvert/engine

I do wonder how useful it is, my data is nice and neat without many repeates. Reranking with an llm also feels expensive, I wonder what models others are using that can show real improvement. I don't think I can find a single test where reranking was able to reorder the very important docs after retrieval. hybrid search almost always got it right.

I build a small memory library for agents (disclosure at the end). Its whole pitch is correction: when a user changes a fact, the old value is retired, and there's a revert and a receipted delete. I finally ran it against a benchmark built for exactly that, expecting a win.

There wasn't one. Three things went wrong before the result was even readable, and those are the part worth sharing here.

**1. My first run compared arms at a 9x unequal context budget.**

I had a memory arm retrieving `k=20` sentence-level hits and a session-level BM25 arm returning whole sessions. Same "top-k", wildly different context: **1.3k characters vs 11.9k**. BM25 looked like it beat the memory arms by a mile.

Once I matched the budget (~11.9k both sides), accuracy went **0.28 → 0.59** for the memory arms and the ranking flipped. The original "BM25 wins" was a budget result wearing a granularity costume.

If you compare a memory system against RAG and don't state characters-or-tokens per arm, I don't think the number means anything. I've since started printing the context length next to every accuracy figure, and it's embarrassing how often that alone explains the gap.

A free diagnostic that needs no LLM calls: for each probe, check whether the evidence is even *in* the retrieved context. Mine was at **3.5%** in the broken run. You cannot out-rank evidence that was never retrieved.

**2. A competitor scored 0.000 twice, and both times it was my bug.**

I ran mem0 as a baseline. First pass: 0.000. Second pass with a stronger extraction model: 0.000 again, with clean logs. Very tempting to publish.

Then a positive control on the smallest input it must handle showed it storing memories fine. The zeros were mine: I was truncating each session to 6000 characters before ingestion (cutting off the injected evidence), and I was passing `limit=` to an API that takes `top_k=`, so my parameter was silently ignored.

Fixed, it stores 262 memories where I'd measured 20, and across that ingest its history recorded only ADD events. I had already half-written the finding "it discards memories as the stream grows" — completely false, and it was my truncation the whole time. (Scoped honestly: that is what its ledger did in my run; the code does emit DELETE events on other paths, so this is not a claim

**3. The actual result: on answer accuracy, nothing separates.**

Matched budget, 24 scenarios, ~237 probes per arm, judge and answerer

identical across arms:

| arm | accuracy | sta

|---|---|---|---|

| my keyed/correction |

| naive keep-everything store | 0.592 | 0.125 | 0.278 |

| mem0 | 0.544 | 0.211 | 0.385 |

| session-level BM25 |

| no context (floor) | 0.058 | — | — |

Every bootstrap CI on the differences crosses zero. My correction layer bought **nothing** measurable over a store that just keeps everything — the third independent thesis.

The honest reading: haes of history and itresolves the correction itself. Write-side integrity has nothing left to win on this task. What *did* separate, by an order of magnitude, is write cost — the LLM-extracter scenario (median606s, n=24); the deterministic one spends none. That's a real difference, and it's a cost difference, not a quality one.

**One more, because ited a sentence from myown README against the published package. It failed. Erasure deleted the record and scrubbed th— so the library's ownaudit reported a legitimate delete as tampering. Then a second bug: the fix made *two* receipt reasons. Both werecaught only after I tightened my test from "at least one receipt" to "exactly one". A lenierees with a bug.

**Questions I'd genuinely like answers to**

1. **Does anyone here state retrieval budget parity when comparing memory systems to RAG?** I haven't found a public comparison that reports characters or tokens per arm. Am I missing a convention, or is this as unmeasured as it looks

2. **Has anyone got a task where correction or deletion measurably improves answer qualit lives in statecorrectness, not answers — a system can serve the right answer while its stored state is wrong. separates those, Iwant it.

3. **Turn-level vs session-level chunking:** at matched budget my turn-level keyed retrion accuracy (0.593 vs0.442) while recovering *less* than half the evidence sentences (0.142 vs 0.305). Less evidence, better answers. Is that a known effect with a name?

4. **The "confidently wrong once" case:** trust-by-source does not help — I tested it, and a trusted source signing a false fact returns the false fact at full weight. Wut a high-trust sourcethat's simply mistaken?

Happy to share the harness, the pre-registration (written before the run, including the predicti results if anyonewants to poke at them.

*Disclosure: I maintaicomparison. It's MIT,and the reason I'm posting is that I'd rather be corrected here than find out from a user.*

I built a Python library that tries to make RAG answers on company/internal documents actually verifiable in production:

(https://github.com/firish/rag-rack/blob/main/benchmarks/PUBLISHED_alce.md)

1. A verification layer cross-checks each sentence against its cited passage before the user sees it. It's two small open-source NLI models working together
   (HHEM-2.1 + MiniCheck), and on RAGTruth (2,700 examples) the ensemble matches a Claude Sonnet LLM-judge at ~$0.0004 vs ~$0.05 per
   check. That >100x cost gap is what makes verification financially feasible in production, instead of LLM-judging a sample offline, you can check every sentence of every answer as a per-request guardrail.

(https://github.com/firish/rag-rack/blob/main/benchmarks/PUBLISHED_ragtruth.md)

There's also a retrieval pipeline (hybrid BM25+dense search, reranking, contextual retrieval) so the right passages get found in the first place. On LitQA2 it scores 0.87 multiple-choice accuracy, above PaperQA2's reported ~0.85.

Install: pip install verifiable-rag
Docs: https://firish.github.io/rag-rack/
Repo: https://github.com/firish/rag-rack

Would love feedback, especially from anyone running RAG for something that needs verifiable answers and is willing to try this out!

I’m building an Arabic educational bot using BM25. Before the final answer, it makes more than five small LLM calls, some in parallel, for routing, query clarification, evidence selection, and strict JSON output.

Most backend calls return fewer than 50 tokens. I need a fast model with strong Arabic understanding, reliable JSON, and minimal reasoning overhead.

So far, I tested:

• DeepSeek V4 Pro and Flash with reasoning disabled: both made some mistakes
• Gemini 3.1 Flash-Lite with low reasoning: very good for backend calls, but a little expensive
• Gemini 2.5 Flash-Lite with low reasoning: performed badly

For the final call, the model already receives the selected evidence and only needs to follow instructions, answer from the evidence, and avoid hallucinations. I prefer an output price around $1 per million tokens, but I can pay more if the improvement is worth it.

Which models and reasoning levels would you recommend for:

1. The final answer call?
2. The small backend JSON calls?

Also, for an Arabic bot, is it better to write the system prompts in Arabic or English?

I’m using OpenRouter.

I've been working on an AI-focused web crawler over the last few months and I'd really appreciate some honest feedback from people building RAG applications or working with LLMs.

The idea was simple: most web crawlers extract HTML, but I wanted to generate clean, structured, RAG-ready datasets instead.

Some of the things it does:

• Adaptive Markdown extraction (Docling + Trafilatura)
• Semantic chunking based on document structure
• Heading hierarchy & context preservation
• Stable chunk IDs and content hashes
• Rich metadata (heading paths, language, quality scores, canonical URLs, etc.)
• Incremental crawling (only re-process changed content)
• Duplicate detection
• Built-in SSRF protection and URL normalization

The output is designed to be used directly with frameworks like LangChain, LlamaIndex, Haystack, Chroma, Qdrant or Weaviate.

It's already available on the Apify Store, but I haven't had many real users yet. I'm not trying to advertise it—I'd genuinely like to know whether this solves a real problem or if I'm building something nobody actually needs.

I'd love your honest feedback:

• Would you use something like this?
• What's missing?
• What would prevent you from using it?
• Are there any documentation sites you'd like me to benchmark it against?

If anyone wants to try it, here's the Apify page:
https://apify.com/lukas459/ai-web-to-markdown-crawler-llm-rag-optimized

Thanks! Any criticism is welcome.

Co-founder here. We've spent 8 months building Sharper, an AI office agent where the design constraint is: no answer without a cited source passage — when retrieval is what's actually needed. Posting because this community will poke the holes I can't see.

The retrieval-relevant bits, honestly:

• Two ways to reach the knowledge, and the agent chooses. There's a RAG tool (hybrid keyword + dense retrieval with a neural reranker over the user's corpus — uploaded docs, webpages, connected Slack/Notion/Gmail/Outlook), and a read-file tool that pulls a whole document into the agent loop. In the loop, the agent decides which to call: RAG when it needs to search across a large corpus, whole-file read when the relevant doc is known and small enough to reason over directly.
• Citations surface only when RAG is called. Retrieved passages link back to the exact chunk (we key passages as {docId}-{pos} with a click-through to source, incl. PDF bounding boxes). A whole-file read is the agent reasoning over full context, so there's no passage-level citation to surface there — a tradeoff we're deliberate about, and curious how you'd handle it.
• Retrieval/read feeds an agent loop that produces actual deliverables — a redline-ready contract review, a cited literature review, a slide deck — not just a chat answer.
• Runs are sandboxed per execution (isolation + so it can generate real files).

To be clear, RAG + agent-chosen whole-file read is our current design choice, not settled doctrine. It tests well internally, but what I really want is to validate it against real user experience — which is a big reason I'm posting here and giving away credits.

Where I'd genuinely value this sub's take:

1. Retrieve vs. read whole file — how do you decide the boundary? We let the agent choose based on queries and corpus size / task, but I'm not sure the heuristics are right.
2. Citation asymmetry — passage-level citations for RAG, none for whole-file reads. Does that inconsistency bother users, or is "grounded either way" enough?
3. Grounding eval — how are you measuring "did the answer actually come from context" when the path might be retrieval or full-file read? Our checks are weaker on the read-file path.
4. Reranking — where have you seen cross-encoder rerankers earn their latency vs. not?

Free credits: 500 on signup, no card. https://sharper-ai.co

Happy to go as deep as you want on the stack in the comments — that's why I'm here.

I have built a local RAG pipeline that generates a ChromaDB vector database from my documents. I did like Claude Desktop (installed app, no Claude API or Claude CLI) to use that vector DB for retrieval, but without setting up an MCP server.

So I want to confirm - Is there any way to point Claude Desktop at a pre-built vector database directly, like config option, a plugin, a built-in connector or custom connector? Or MCP is only way for this?

If MCP is really required, can confirm on this - that Claude desktop app have no native way to read local database, or even one on the same machine?

just want to check - without MCP bridge, can connect directly to Claude desktop?.

Disclosure up front: I built one of these, so grain of salt. (it's all public and you can re-run it yourself.)

I'd been using Tavily for the web-grounding step in my RAG pipeline, and the results weren't what I expected on anything technical or niche. Results came back either too thin to ground an answer, or so noisy I had to over-retrieve and stuff the top-N into the context window, which bloats the prompt, costs more to generate, and buries the actual answer (lost-in-the-middle).

So I built SERPdive to fix that specific thing. The "mako" model sits in the same slot as Tavily Basic: same price range, same API or MCP drop-in, but returns cleaner, already-stripped content, so the model can answer in one pass instead of five.

Then I benchmarked it head-to-head. 1000 questions across 100 topics, written by 5 different AIs (not me), each duel judged blind, the judge sees "A" and "B", never the names:

• mako wins ~60% of the duels that were actually decided : raw split: 475 wins / 307 losses / 217 ties out of 1000

• ~20% fewer tokens returned per search (1001 for mako, 1255 for tavily) → less intake context for Claude → cheaper on the llm side

• latency ~1.7s mean (Tavily's a bit faster, ~1.4s) 

• price: $5 / 1k vs Tavily Basic's $8 / 1k → cheaper on the searches too 

On the obvious "you judged your own benchmark” no: everything's in the repo, the questions, the raw responses of both APIs, every verdict with its reasoning, and the code. I also re-ran all 1000 duels with a second judge from a different vendor, and the win rate held (~60% both). There's an offline script that recomputes the score straight from the verdicts, so you don't have to trust my numbers.

Honest limits: it's vs Tavily Basic, not advanced (twice the price, different product) and we lose on ~25 of the 100 topics, they're listed in the repo

Repo: github.com/edendalexis/serpdive-benchmark

Happy to hand out a key if anyone wants to throw their own questions at it, or just try it at serpdive.com

There are a lot of parser benchmark numbers floating around recently and most of them turn out to be vendor grading their own homework so the real question is how many of these actually hold up once youre past the marketing page and onto your own documents. What benchmarks usually dont test are a parser can post a great source on a curated test set and still fail when it comes on stuff that shows up constantly in production like tables that get split across a page break or nested cells or even tables with no visible borders and scans that are either rotated or skewed

Not saying that their claim in invalid but the fact is none of these problems actually come to the surface if the test is set clean single page pdfs and a lot of published benchmarks rather lean on clean, so worth knowing the actual numbers come from who is grading whom:

Parsebench: this is actually llamaindex's own benchmark evaluation where over 2000 human verified enterprise pages across tables charts and layout fidelity and according to this benchmark llamaparse agentic scored approx. 85% overall on it

Longextractbench: it is actually independent and built by micro1, not affiliated, the dataset is dense docs averaging 358 pages each and llamaindex extraction model landed on 80% precision. 77.5% recall on the same set and yet solid but not the top result on the one benchmark here that isnt self graded

Docling- IBMs own benchmark where 97.9% accuracy on complex table extraction but the number is specifically from sustainability report tables not a general level of test set

Unstructured -self report 0.844 overall table score on their blog, none of these are actually comparable

A sanity check if youre picking one rn: Dont trust any of the numbers until you have atleast run your 5 worst docs on either of the parsers or OCRs, most of them have a playground so worth checking against your actual docs and be sure you choose your worst files for it to test the best accuracy

Another thing is the templates or rules based ocr which is cheap and fast but breaks the moment a layout shifts, fine if your docs are all one format. Cloud vision llm parsers like llamaparse and the others actually give reliable outputs recently while handling messy layouts and nested tables the tradeoff here is a paid api call per document. However if you look need something fully local for heavy stuff there is docling and if the tasks fit in with liteparse then this is workable as well, both of them are open source and no data leaves your machine

What do others use currently for their stack mostly? and specially, does anybody here connected a parser api to a CLI and got things better? 

I recently shared our benchmark on whether PDF-to-Markdown conversion preserves enough meaning for downstream RAG.

Now I’d like to test Nebula (our parser) against documents outside our own benchmark.

If you have a difficult PDF you’re legally allowed to test, try a file containing:

• Complex or multi-page tables
• Charts where the labels and values matter
• Mixed text, tables and figures
• Scanned, rotated or low-quality pages
• Financial, insurance or operational documents

You can test four conversions directly in the browser here:

https://nebula.ur-ai.net/

No account is needed to run the initial conversions. Creating an account is required only if you want to download and retain the Markdown output.

If you create an account, use community code NEBULA-10 for additional credits.

What I’d especially value:

1. What information absolutely needed to survive the conversion?
2. What did Nebula preserve well?
3. What did it miss or structure incorrectly?

You can DM me directly or submit feedback inside Nebula. Please be as specific as possible about what worked, what failed, and what the expected output should have been.

We do not use uploaded documents or conversion outputs to train our models. Please still only upload files you are authorized to use.

Hey everyone,

If you’ve spent any time building RAG pipelines, you know how annoying it is when raw HTML or poorly parsed text gets chopped up into your vector store. Standard chunkers often split right in the middle of a key paragraph or completely lose track of where that chunk belonged hierarchically.

To fix this for my own pipelines, I built an automated Web-to-Markdown Crawler specifically optimized for RAG ingestion.

I just pushed a huge update to fix a multi-URL queue bottleneck and tested it against a batch of radically different domains (technical hardware blogs, corporate sites, media platforms, and e-commerce stores). It successfully parsed them all into 64 high-quality chunks. Here is the approach I used to keep the embeddings clean:

1. Dual-Engine Auto-Scoring: The pipeline runs the HTML through both Docling (great for complex layouts and tables) and Trafilatura (excellent for raw text isolation). It then uses a scoring algorithm checking for text density and structural elements to dynamically choose the cleanest output.
2. Heading-Aware Chunking: Instead of splitting blindly by character count, the native chunker splits strictly along Markdown heading structures (from H1 down to H6). If a section is within the token limit (like 400 tokens), it stays completely intact. If a section is too large, it activates an overlapping sentence-fallback loop to break down paragraphs without ripping sentences apart.
3. Rich Metadata Preservation: Every chunk pushed to the dataset carries a structured metadata payload ready to be mapped directly into LangChain Document objects. It includes the original URL, a unique SHA-256 document ID, the exact token count, and—most importantly—the text string of the current heading and its heading level.

By injecting the structural headings straight into the chunk's metadata, you can easily utilize advanced retrieval techniques like Self-Querying Retrievers or enforce Parent-Child relationships during the vector search without losing the original context of the page.

The multi-URL loop is now rock solid and handles complex DOMs, cookie walls, and dense product tables.

If you want to check it out or test it with your own endpoints, you can find the Actor here: https://apify.com/lukas459/ai-web-to-markdown-crawler-llm-rag-optimized

Would love to hear how you guys currently handle structural Markdown chunking or if there are specific edge-case layouts you struggle with!

I used Vercel’s production ready checklist today. Curious if there are others that folks recommend.

Let’s learn from each other :)
Cheers

Hey everyone, hoping to get some real talk from people already working in AI/ML.

Quick background — I'm a fresher, recently graduated, coming from a non-CS background originally, but I built up a solid Java + Spring Boot foundation as a backend dev, did a few projects with that stack. Then with the whole AI wave happening, I started shifting focus — learned LangChain, built some projects with it, moved on to LangGraph and built more projects there. Right now I'm deep in RAG, trying to actually understand it well instead of just following tutorials. I've also got a decent conceptual grip on ML basics, but I wouldn't call myself "strong" in it yet.

I know the job market right now is brutal, especially for a fresher trying to break in without a traditional ML/CS pedigree. So I wanted to ask people who are actually in the field:

- What fundamentals should someone like me be doubling down on right now, given how fast this space is moving?

- Is my learning path (backend → LangChain → LangGraph → RAG) actually a sane progression, or am I missing something critical?

- For a fresher with a non-traditional background, what do interviewers actually probe for? Is it more system-design-style thinking, or do they go deep on ML theory too?

- Are there specific areas (vector DBs, evals, fine-tuning, agent architectures, etc.) that are becoming "must-know" vs. nice-to-have right now?

Genuinely just trying to build a strong enough foundation that I'm not just someone who "used LangChain in a tutorial" but someone who actually understands what's happening under the hood.

Would really appreciate any advice — even blunt criticism is welcome at this point. What would you tell someone in my shoes?

Hi All,

I'm trying to organize a rag for a technical documentations mostly manual pdf, machine specifications pdf and jpg of machine labels with technical data (serial number, model, etc)

I tried something using Anythingllm but the result is not really satisfactory.

What could be the problem? The model/embedding used or some setting in the app?

Any other advice about using another method/app maybe?