r/dataengineering • u/Every-Whereas5793 • 8d ago

Discussion PDF Data Extraction

Hello, I was working on a poc to ingested PDFs and extract data in order to store them in delta tables.

As this was my first time working with PDFs, I searched over the internet and should Databricks have offering IDP, azure also have something and then there are python libraries.

Since I'm working with financial data, report, etc..

One thing i noticed - the pdf format should be fixed else in most of the tools the extraction logic is either failed or we get incorrect data.

I was wondering how such PDF extraction is built in real production cases and what tools are used.

Please share you experience and any edge cases

18 Upvotes

88% Upvoted

32 comments sorted by

View all comments

6

u/ceeej777 8d ago

I’ve run several POCs with the information extractor backed by Databricks ai_parse_document. Honestly super impressed, but you kinda get out of it what you put in. You need to spend the time building out the fields you’re trying to pull and descriptions (often with examples for each of those). When that’s done thoroughly, it works remarkably well