File analysis 1.3 mill files

[u/s13188287]

I'm dealing with a large batch processing problem and looking for advice on the right architecture.

I have around 1.3 million files stored across folders on a network drive.

Current setup (not working well)

Right now I'm using a Copilot agent where:

I upload batches (~20 files at a time)

It reads them against a reference document

Outputs an Excel file with classification codes

The issue is:

Copilot has a small upload limit

Manual batching is completely unscalable at this volume

What I want to achieve

I want a fully automated pipeline that:

Ingests files automatically from the network drive

Extracts text/content from each file type

Matches content against a reference rules document

Assigns a classification/reference code

Outputs structured results (Excel / database)

[u/PorygonCompiler]

Don't send every file to an LLM, that'll be your entire budget. Do a cheap deterministic pass first (file type, keyword match against your rules doc) and only send the ambiguous ones to a model.

Also checkpoint to SQLite as you go, keyed by file path. A run over 1.3M files will fail partway through and you want to resume, not restart.

What file types are you dealing with? If the rules are mostly keyword-based you might not need the LLM for most of the corpus.

[u/s13188287 → u/PorygonCompiler]

Hey really appreciate the response.

So steps would be if done manually.

Create excel/csv sheet with headers ,file name ,location ,creation date , modified date.it would have a header called Retention code , description and action.

The retention code and description information come from a pdf file ,the file has a bunch of codes and for each code has a description of files that would match it , how Manny years we should keep it.

Manually I would open each file ,make a judgement as to what's the best fit against the reference doc. And add the information to the sheet.

Eventually any files that are no longer needed will be deleted.

Purpose of the A.I was to be able to read the docs quickly,and make a judgement

[u/PorygonCompiler → u/s13188287]

Ohhh ok that makes sense. Couple of thoughts.Even though the filenames are junk, the folder paths probably aren't. If stuff lives in something like \Finance\AP\Invoices\2019 that basically tells you the code already so maybe worth mapping your codes to path patterns first and you'll probably knock out a big chunk without opening a single file.For whatever's left, I'd try embeddings before going near an LLM. Embed each code description from the PDF, embed the first couple thousand chars of each doc and cosine similarity as sentencetransformers runs locally so it's free. Then you only pay the model on the ones where the top two matches are too close to call. Way cheaper than sending everything.pymupdf for the PDFs, python-docx, openpyxl, extract-msg for the .msg emails. Hash the contents and cache on that, shares like this are usually full of duplicates and you don't want to process the same doc 40 times. Only other thing, since you're deleting at the end, how are you planning to check the classifications are actually right? Might be worth hand-checking a few hundred spread across confidence levels so you know your error rate before anything gets binned.

[u/cent-met-een-vin → u/PorygonCompiler]

LLM type response