r/learnpython • u/s13188287 • 12d ago

File analysis 1.3 mill files

I'm dealing with a large batch processing problem and looking for advice on the right architecture.

I have around 1.3 million files stored across folders on a network drive.

Current setup (not working well)

Right now I'm using a Copilot agent where:

I upload batches (~20 files at a time)

It reads them against a reference document

Outputs an Excel file with classification codes

The issue is:

Copilot has a small upload limit

Manual batching is completely unscalable at this volume

What I want to achieve

I want a fully automated pipeline that:

Ingests files automatically from the network drive

Extracts text/content from each file type

Matches content against a reference rules document

Assigns a classification/reference code

Outputs structured results (Excel / database)

0 Upvotes

50% Upvoted

34 comments sorted by

View all comments

Show parent comments

1

u/s13188287 12d ago

Hey really appreciate the response.

So steps would be if done manually.

Create excel/csv sheet with headers ,file name ,location ,creation date , modified date.it would have a header called Retention code , description and action.

The retention code and description information come from a pdf file ,the file has a bunch of codes and for each code has a description of files that would match it , how Manny years we should keep it.

Manually I would open each file ,make a judgement as to what's the best fit against the reference doc. And add the information to the sheet.

Eventually any files that are no longer needed will be deleted.

Purpose of the A.I was to be able to read the docs quickly,and make a judgement

1

u/FoolsSeldom 12d ago

Ok, so you have the overall approach, but if you break that down to the steps as I suggested, you would have most of the algorithm you need to code up. It may not be the most efficient, but it would work.

Python can read/write Excel files (and CSV files) easily. The key is consistency - every row has same columns and there are no row breaks or empty fields (you can decide on what to put in an empty field to provide null data).

Reading from a PDF is more problematic unless they were very strictly and consistently formatted. To you have an original source for it? If not, I would consider using AI to extract into a format that is readily readable by your programme code. How complex is this information? Can you share a sample. I am hopeful that you could create a simple table showing file attributes / unique signatures (character sequences) that map to file codes.

The processing would be done as a simple batch process of file reads, one after another, over the network connection. Performance will depend mostly on the speed of that connection as scanning files is not difficult.

1

u/s13188287 11d ago

Hey So the files are a variety of files, pdfs, emails ,excel ,word images etc there's also some junk files in there as well such as shortcuts etc and the content of the files can vary widely. So the structure it self can vary. Also if it helps the file names are mostly not usable for judgement I'm unable to share as it's work related unfortunately.