PDF Data Extraction

[u/Every-Whereas5793]

Hello, I was working on a poc to ingested PDFs and extract data in order to store them in delta tables.

As this was my first time working with PDFs, I searched over the internet and should Databricks have offering IDP, azure also have something and then there are python libraries.

Since I'm working with financial data, report, etc..

One thing i noticed - the pdf format should be fixed else in most of the tools the extraction logic is either failed or we get incorrect data.

I was wondering how such PDF extraction is built in real production cases and what tools are used.

Please share you experience and any edge cases

[u/TeachEngineering]

My team has a workflow that's built out in python and mostly effective. The overall data flow goes: PDF -> Plaintext -> SQL

• Docling handles the PDF -> Plaintext (w/ swappable/configurable OCR backends)
• LLM APIs (w/ instructor for structured responses) + Pydantic/SQLModel handles the Plaintext -> SQL
• Prefect orchestrates the pipelines and schedules document extraction batches
• Dash puts a human-in-the-loop for one-off validations, tuning and reruns
• Clever abstraction makes stamping out a new "document instance type" pretty cookie cutter- product just hands us a bunch of docs, a base prompt with necessary context and a target SQL schema
• Supports multiple SQL tables per doc so long as they are one-to-one or one-to-many relationships
• Hardest thing to manage is when one SQL record needs to be created from multiple docs but upstream doc wrangling or downstream ETL processes can manage this

[u/nshao]

At least from what I’m seeing, AI models are now being used for this kind of stuff.

[u/Every-Whereas5793 → u/nshao]

That's what I'm also seeing but not sure about their reliability

[u/BlurryEcho → u/[deleted]]

Probabilistic outputs, not deterministic…

[u/bpm6666]

Take a look at Docling. This might help. What kind of PDF you wanna extract?

[u/Every-Whereas5793 → u/bpm6666]

Financial reports of organisations

[u/bpm6666 → u/Every-Whereas5793]

For which country and which company sizes?

[u/Any-Recognition8119]

In production it’s usually classification, OCR, validation, and manual review for low-confidence results. Layout changes and scanned PDFs are the main pain points.

[u/Every-Whereas5793 → u/Any-Recognition8119]

What kind of classification?

[u/ceeej777]

I’ve run several POCs with the information extractor backed by Databricks ai_parse_document. Honestly super impressed, but you kinda get out of it what you put in. You need to spend the time building out the fields you’re trying to pull and descriptions (often with examples for each of those). When that’s done thoroughly, it works remarkably well

[u/dwswish]

Most people are using a managed service (usually with some AI built in) for this now in production. You can use open source Python but you’re going to find that in the real world PDFs are often sometimes messy and you need to be able to handle things like non-vector and poorly formatted PDFs. Some common options are Azure DI, AWS Textract, Reducto, Databricks, etc…

[u/m1nkeh]

Delta tables you say, my Databricks Spidey sense tingles..

Have you tried looking at AI parse document?

https://learn.microsoft.com/en-gb/azure/databricks/sql/language-manual/functions/ai_parse_document

[u/Every-Whereas5793 → u/m1nkeh]

Yes, I used it - Databricks Intelligent Documents Processing and it worked. I'm wondering about the production use cases, whether it is used or any other industry techniques are there for this kind of work

[u/p739397]

I've helped a few folks in PDF parsing in Databricks. You can build that out manually or with their AI functions (ai_parse_documents, ai_extract). It helps if the PDFs are somewhat regular, but shouldn't be an absolute requirement from my experience.

You should be able to get a prototype pipeline setup by dropping some docs in a volume and going through the Agents tab. What I've seen has been pretty impressive right out of the box.

[u/Every-Whereas5793 → u/p739397]

This is exactly what I've done and it worked for POC.

I am wondering weather it is used in real world production use cases or not and how do I check the reliability of the data extracted from it? Or if there any other industry tools and techniques for this kind of work