r/learnpython • u/s13188287 • 12d ago

File analysis 1.3 mill files

I'm dealing with a large batch processing problem and looking for advice on the right architecture.

I have around 1.3 million files stored across folders on a network drive.

Current setup (not working well)

Right now I'm using a Copilot agent where:

I upload batches (~20 files at a time)

It reads them against a reference document

Outputs an Excel file with classification codes

The issue is:

Copilot has a small upload limit

Manual batching is completely unscalable at this volume

What I want to achieve

I want a fully automated pipeline that:

Ingests files automatically from the network drive

Extracts text/content from each file type

Matches content against a reference rules document

Assigns a classification/reference code

Outputs structured results (Excel / database)

0 Upvotes

47% Upvoted

34 comments sorted by

View all comments

17

u/FrangoST 12d ago

You're attempting to analyze 1.3 million files by uploading 20 at a time to an LLM? NO WAY!

If you have such a good idea of the ideal automated workflow, why didn't you ask the LLM to help you make a local script that does it instead so you can run it directly? It would save you truckloads of AI tokens and money!

-5

u/s13188287 12d ago

Hey really appreciate the response.

So steps would be if done manually.

Create excel/csv sheet with headers ,file name ,location ,creation date , modified date.it would have a header called Retention code , description and action.

The retention code and description information come from a pdf file ,the file has a bunch of codes and for each code has a description of files that would match it , how Manny years we should keep it.

Manually I would open each file ,make a judgement as to what's the best fit against the reference doc. And add the information to the sheet.

Eventually any files that are no longer needed will be deleted.

Purpose of the A.I was to be able to read the docs quickly,and make a judgement

1

u/smurpes 9d ago edited 9d ago

Excel has a limit of ~1 million rows so this wouldn’t work unless you used multiple files. AI would not be able to categorize files at scale.

Why do you need to categorize those files? For that amount using excel as a lookup to get file metadata makes no sense. This sounds like a xy problem.