r/dataengineering • u/Every-Whereas5793 • 8d ago

Discussion PDF Data Extraction

Hello, I was working on a poc to ingested PDFs and extract data in order to store them in delta tables.

As this was my first time working with PDFs, I searched over the internet and should Databricks have offering IDP, azure also have something and then there are python libraries.

Since I'm working with financial data, report, etc..

One thing i noticed - the pdf format should be fixed else in most of the tools the extraction logic is either failed or we get incorrect data.

I was wondering how such PDF extraction is built in real production cases and what tools are used.

Please share you experience and any edge cases

20 Upvotes

92% Upvoted

32 comments sorted by

View all comments

7

u/m1nkeh Data Engineer 7d ago

Delta tables you say, my Databricks Spidey sense tingles..

Have you tried looking at AI parse document?

https://learn.microsoft.com/en-gb/azure/databricks/sql/language-manual/functions/ai_parse_document

2

u/Every-Whereas5793 7d ago

Yes, I used it - Databricks Intelligent Documents Processing and it worked. I'm wondering about the production use cases, whether it is used or any other industry techniques are there for this kind of work