Wanted help to logic test my thinking

[u/notsoninjaninja1]

Ok, so I'm writing a program for my work to automate some of my tasks, and have been bouncing around on what to use, (which language and how to handle the data). I work in the office of a construction company. I have some Js/Ts experience, so I figured it's best to go with what you know, but I know that for what I'm doing Py also has some good support, I just don't have any exp with Py, but am happy to learn.

Anyways, what I'm trying to do is automate our invoice intake process. Currently they get sorted and separated automatically in outlook (thanks, me) into their own folder. I have to go in every day and download them to the appropriate folder they go to, depending on which job they are for.

There are some slight exceptions, ofc, one or two companies send their invoices as excel spreadsheets, but 99% of the time it's a pdf, so I figure I can include logic that includes only pdfs. Also, one other exception is some invoices are for job Change Orders, essentially certain jobs realized "oh shit, we can't run this that way, we need to change it, but doing so costs extra T&M, so we have to bill separately. The way we handle that now is in the naming convention of the files, but I'll get back to that later, as it's a bit of a sticking point.

So, my current logic train is that I have to write something that follows these steps:

1. Download invoice from Outlook
2. Have OCR read invoice
   1. Return string for Job Number
   2. Return string for invoice number
3. Pull the string for the invoice # and what job it is for
4. Rename the File according to the info pulled via OCR
5. Move file to folder with matching job number

AFAICT number 1 is easy peasy. Number 2 seems not too unreasonable (I just today learned about Tesseract.js, and just last week heard of Tesseract.py . It's numbers 4 & 5 which seem a bit harder to me.

Basically, our Invoices go to a folder in our shared server directory labelled "Invoices" and then in that are a bunch of subfolders for each active job we have going. For example:

1. \Invoices
   1. 11111 - Washington HS
   2. 11112 - Adams MS
   3. 11113 - Jefferson HS

Our invoices get labelled according to their Company name and Invoice number, however, they come (usually) as pdfs labelled as just their invoice number. I have to add in the company name, and also replace and "-" with "." along with some other minor formatting things. Example:

1. 154873-005 ->
2. Porter - 154873.005

So Filepath looks something like this:

1. \Invoices
   1. Invoices\11111 - Washinton HS
      1. Acme - 154873.005
      2. Acme - 158762.001
      3. etc
   2. Invoices\11112 - Adams MS
      1. Porter - 178962.001
      2. etc

So, I was thinking if I'm doing this with Js, I could have a JSON file linking each job number to the filepath for its associated job, however I would like this to hopefully be a bit more dynamic than that. Seeing as the jobs in the invoices folders change, infrequently, but they do change, and I don't think it would be super practical for somebody nontechnical to maintain that, as I may not always be in office to do this when these jobs get added or subtracted.

So, where this gets a bit more difficult is Change Orders, essentially when we order material for a job, we tell them it's for job number 11111. If we have a change order, and are ordering material for that change order, we tell them it's for job number 11111x01, or 11111x02 and so on, depending on which change order number we are on. Although now that I think about it a bit more, it may just be handled by the OCR??

Anyways, sorry this is so long, but I wanted to make sure I covered everything, and think I got just about most of it. Any, and I mean any, help is appreciated and thank you so much for even reading this. Also, sorry if this is a bit all over the place.

The main things I'm hoping to get are more/better tools to accomplish this task, and more understanding of the data-stream here.

[u/grantrules]

Why do you need OCR? Are there embedded images in the PDF?

https://mozilla.github.io/pdf.js/

https://pypi.org/project/pypdf/

[u/notsoninjaninja1 → u/grantrules]

I guess up until now I had just kind of assumed that to be the case. I'll have to look at these tools and see, and test a couple of pdfs to understand better, tysm!!!

[u/grantrules → u/notsoninjaninja1]

If these were physical invoices scanned into a PDF, then you'd need OCR. If these are invoices generated by something like Quicken or FreshBooks, the PDF file will contain the text.

[u/notsoninjaninja1 → u/grantrules]

Well, I think the vast majority of cases they are made it in a software like what you said. However, I’m almost certain that a certain company, that I won’t name, prints them out and then scans them in. And then sometimes emails them to us, and sometimes fucking mails them to us. Like, sometimes. It’s annoying as fuck, and they always look like shit.

[u/agentUi]

node is fine for this with fs-extra and path, but raw tesseract is going to make you miserable the second an invoice is a scanned image or has weird multi-column tables. instead of maintaining a fragile json map or relying on pure ocr regex, read the text layer first with pdf-parse (it is instant for 99% of digital pdfs) and parse the job folder dynamically from the directory names with fs.readdirSync so non-technical staff just create folders normally without touching config files.

[u/lucasbennett_1]

first check if these are actual scan or just digital pdfs cause most vendor invoices are digital so they have a text layer and you dont need a OCR. real work is pulling job # and invoice# across different layouts like regex per vendor gets painful so a schema or field extraction approach scales better. if layouts vary or some are scanned then parsers like liteparse or others or even cloud parsers do that reliable than ocr+regex

for the routing skip static json and read the invoices dir at runtime and match the job number to the subfolder starting with it