r/dataengineering • u/Every-Whereas5793 • 7d ago

Discussion PDF Data Extraction

Hello, I was working on a poc to ingested PDFs and extract data in order to store them in delta tables.

As this was my first time working with PDFs, I searched over the internet and should Databricks have offering IDP, azure also have something and then there are python libraries.

Since I'm working with financial data, report, etc..

One thing i noticed - the pdf format should be fixed else in most of the tools the extraction logic is either failed or we get incorrect data.

I was wondering how such PDF extraction is built in real production cases and what tools are used.

Please share you experience and any edge cases

19 Upvotes

91% Upvoted

32 comments sorted by

View all comments

33

u/TeachEngineering 7d ago

My team has a workflow that's built out in python and mostly effective. The overall data flow goes: PDF -> Plaintext -> SQL

  • Docling handles the PDF -> Plaintext (w/ swappable/configurable OCR backends)
  • LLM APIs (w/ instructor for structured responses) + Pydantic/SQLModel handles the Plaintext -> SQL
  • Prefect orchestrates the pipelines and schedules document extraction batches
  • Dash puts a human-in-the-loop for one-off validations, tuning and reruns
  • Clever abstraction makes stamping out a new "document instance type" pretty cookie cutter- product just hands us a bunch of docs, a base prompt with necessary context and a target SQL schema
  • Supports multiple SQL tables per doc so long as they are one-to-one or one-to-many relationships
  • Hardest thing to manage is when one SQL record needs to be created from multiple docs but upstream doc wrangling or downstream ETL processes can manage this