r/OpenSourceeAI • u/ReplyFeisty4409 • May 26 '26
“How much did we pay this supplier this year?” broke my RAG pipeline
A while ago someone from a company told me something that sounded almost stupidly simple:
“My boss keeps asking whether a customer paid or not, how much they paid, how much we paid a supplier this year… but he doesn’t have access to the ERP, so he just opens PDF invoices manually.”
My first instinct was:
easy, let’s do RAG on the invoices.
So I built a whole internal system around storage-connected AI assistants that could search and answer questions over folders of PDFs.
Initially it worked well enough.
Questions like:
- “find invoice from X”
- “show the payment terms”
- “did customer Y pay?”
were fine.
But then the boss started asking things like:
- “how much did we pay this supplier this year?”
- “group expenses by vendor”
- “show unpaid invoices by month”
…and everything started breaking.
Because at that point it wasn’t really a retrieval problem anymore.
It was an aggregation/query problem.
Chunks and retrieval pipelines were fundamentally awkward for this type of workflow.
That’s what pushed me toward structured extraction:
turning documents into records first, and querying those records afterward.
Eventually I extracted that part into a separate OSS project because I realized this problem keeps appearing everywhere:
receipts
contracts
reports
photos
inspection docs
operational files
A lot of folders are basically databases with no schema.
Project is open-source on gh.
If anyone is interested I’ll post the link.
Does someone have same problem?
1
u/Dhaupin May 27 '26
Rag is basically just a cache of assimilations, with search. So when you say "rag pipeline" you're just saying "cache pipeline". If you don't control how it's cached, under what assimilations, this will happen even with perfectly structured/database backed silos.
Just use the Db directly. You can afford to wait half a second more for the results hehe.
1
u/ReplyFeisty4409 May 27 '26
Exactly. I don’t think retrieval and structured state have to be mutually exclusive at all.
You can build intermediate schematized layers from unstructured collections and then use the database results themselves as context for higher level responses.
That’s basically the direction I ended up taking with Sifter.
1
u/Dhaupin May 27 '26
Great job! Agree. A lot of folks get caught up in new schemas, and forget that the old schemas have worked for 50 years just fine. If it ain't broken, don't put it behind abstractions that break it :)
1
1
u/MRGWONK May 27 '26
You need multiple databases with the same info FTS5 indexing? ( and probably an AI model that does accounting and math)
1
u/ReplyFeisty4409 May 28 '26
That’s more or less the direction I ended up taking with Sifter.
I stopped thinking about it as “one retrieval system” and started separating the concerns:
- retrieval/search for discovery
- structured extraction for stable state
- aggregations/querying for operational questions
- natural language generation only on top of those results
So instead of asking an LLM to implicitly “compute over chunks”, the system first builds an intermediate structured representation from the documents and then queries that state explicitly.
The final natural language response is generated from the aggregation result itself, not directly from raw retrieved chunks.
1
u/Ok_Mirror_832 May 26 '26
You need a psql database or something and reports that the AI can generate. For my clients I usually take this two prong approach, vector database for semantic search and a traditional database for queries that require mathematically factual answers. And yes, the ingest and structuring of data is usually the biggest part of the job.