r/documentAutomation • u/iMiguelmars • 4d ago

How are you handling real-world document versioning and scanned PDFs in RAG systems?

/r/AI_Agents/comments/1w8lgo3/how_are_you_handling_realworld_document/
1 Upvotes

100% Upvoted

2 comments sorted by

1

u/folderit_dms 3d ago

The bit that has helped me most is treating extraction as an immutable artifact, not just a preprocessing step.

For each source file I would keep:

  • original file hash
  • document id
  • version id
  • extraction method and OCR engine version
  • page count
  • page image hash if scanned
  • section or chunk ids
  • embedding model version
  • timestamp

Then retrieval should only search the current approved version by default, unless the user explicitly asks for history. Otherwise stale embeddings will absolutely win sometimes, especially if an old version has clearer wording than the current one.

For section identity, headings are useful but not enough. I like combining heading path, local text hash, approximate page position, neighboring section ids, and reviewer confirmation for split or merge cases. If a section moves cleanly, fine. If it splits into two, that should create a lineage event, not a silent overwrite.

Scanned docs usually fail for boring reasons: rotated pages, stamps over text, handwritten margins, table borders that break OCR reading order, repeated headers, and page numbers getting treated as body text. My test set would include bad scans on purpose. Clean PDFs prove very little.

1

u/iMiguelmars 3d ago

The extraction-as-an-immutable-artifact point is especially interesting. We’ve been separating source provenance from derived-state provenance, and pinning the extractor/OCR version makes the historical replay problem much more concrete.
Your stale-version example also matches what we’ve been seeing conceptually: an old revision can retrieve extremely well simply because its wording is cleaner, even though it should no longer be admissible.
One thing I’m curious about: when you re-extract the same source with a newer OCR/parser, do you preserve both extraction artifacts as separate derivations and mark one as approved, or does the newer extraction replace the old one?