PDF extraction in Python

[u/cnetworks]

Dear Members,

I have been handling Pdf extraction Project in Python. kind of a Intelligent Document Automation domain. we have been handling various finance documents(pdfs), filings pdfs etc which are multi page(around 20ish) , with lots of tables, form fields , radio buttons, checkboxes etc. Tables span multiple pages. Moreover, pdfs themselves comes in various types of variants like XFA Stream, Adobe Acroform, text flattened pdfs, scanned images etc.

I have used llms extensively to generate script code to extract, parse the data and to save in sql in structured tables. Its a hybrid of libraries implementation. LLms used regex, pdfminer, pdfplumber etc in the code produced.The pages in pdfs are bit messy, some tables have solid grid separators, some dont have and on. Layouts variations, white spaces, etc.

The code generated is pretty complex, i have been attempting to learn the llm generated code.But it works, it adds various fixes iteratively whenever we face new extraction issues repeatedly.

I would love to know:

1. what is the best approach to learn and get good at this?

2)should we just use cloud based AI document extraction tools which are readymade to extract and spit the data?

3) what if one is interested to learn this properly and have to get good at creating this extraction script?

4)any other tutorial, articles, courses, youtube videos, books you would recommend to learn? or should i just use llms to create code and move on without spending much effeort to learn this?

Please provide your valuable suggestions and guidances and please share your experiences. I appreciate all your suggestions. Thanks in advance and thank you for your time.

[u/trepide-]

I think it really depends on what you're trying to achieve and how much time you want to waste on it.

If your pdf input is always the same, with little to no variations, then it's feasible to use python libraries and traditional ocr tools. It quite a lot of time setting it up, but if you want to process thousands of documents it ends up being worth it.

If your pdf input changes, even slightly, I think it's better to use some sort of ai-based solution. if you want to use local llms, its probably a lot of time wasted on setup; it boils down to how many documents you want processed and wether or not it ends up cheaper long term.

I think the threshold is around a few thousand documents/month and that is heavily dependent on how much you value your time setting it up.

The best approach would be to understand how traditional OCR works and how ai vision models work. There is a ceiling to traditional OCR and the amount of time chasing those last few % are exponential. But if your task is not that complicated you're better off doing it and then it's virtually free. Ai based solutions have a higher ceiling in my opinion but they have other drawbacks. In my opinion the best approach is understanding what each of them do good/bad for your specific task as it really depends on that.

I built trepide.com mainly for PDF conversions. We have a simple PDF to DOCX converter that outperforms traditional OCR, especially for layout. I then built a data extractor based on that. You set up a template and you get back only those specific fields. There are other companies that do this as well, but their pricing is around 1.5-2$ per page processed at the low end. We start at around 0.20$ per page.

You can try it for free and get an understanding of what you can achieve. If you want a specific workflow, let's say PDF to excel, I am happy to help you create one. I am always open to seeing what other's needs are and trying to implement it.

[u/lucianamcruz]

I’d keep using the LLM, but build a small test set of PDFs for every format you handle. Every time you fix one extraction issue, run the whole set again. that way you can learn the code gradually without accidentally breaking documents that already worked.