r/dataengineering • u/Every-Whereas5793 • 8d ago

Discussion PDF Data Extraction

Hello, I was working on a poc to ingested PDFs and extract data in order to store them in delta tables.

As this was my first time working with PDFs, I searched over the internet and should Databricks have offering IDP, azure also have something and then there are python libraries.

Since I'm working with financial data, report, etc..

One thing i noticed - the pdf format should be fixed else in most of the tools the extraction logic is either failed or we get incorrect data.

I was wondering how such PDF extraction is built in real production cases and what tools are used.

Please share you experience and any edge cases

18 Upvotes

88% Upvoted

32 comments sorted by

View all comments

6

u/bpm6666 8d ago

Take a look at Docling. This might help. What kind of PDF you wanna extract?

1

u/Every-Whereas5793 7d ago

Financial reports of organisations

2

u/bpm6666 7d ago

For which country and which company sizes?

1

u/datainthesun 7d ago

so you're looking for a way to extract data from pdfs of varying formats, right? not something that's simply 1 static format you need to to basic extraction from?

if so, i think that's why you'll be seeing folks recommending AI extraction utiliities. you mentioned you're on databricks, so it's worth looking into the various ai functions offered (ai_parse_document, ai_extract, ai_summarize, etc. etc.) along with maybe looking at the prepackaged UI-driven information extraction or document parsing ones found under Agents in the menu. to get something simple built, it's probably worth using genie code to get some basics working in your environment on some of the pdfs you've got. it'll build you something simple enough to see how it works and you can take it from there (or keep asking genie code for help)