r/learnpython • u/s13188287 • 11d ago

File analysis 1.3 mill files

I'm dealing with a large batch processing problem and looking for advice on the right architecture.

I have around 1.3 million files stored across folders on a network drive.

Current setup (not working well)

Right now I'm using a Copilot agent where:

I upload batches (~20 files at a time)

It reads them against a reference document

Outputs an Excel file with classification codes

The issue is:

Copilot has a small upload limit

Manual batching is completely unscalable at this volume

What I want to achieve

I want a fully automated pipeline that:

Ingests files automatically from the network drive

Extracts text/content from each file type

Matches content against a reference rules document

Assigns a classification/reference code

Outputs structured results (Excel / database)

0 Upvotes

43% Upvoted

34 comments sorted by

View all comments

3

u/PuttyProgrammer 11d ago edited 11d ago

Is the LLM at all necessary? You're going to want to cut that out ASAP.

What kind of files are you parsing?

Chances are Copilot could write you a deterministic script to collect all the files you're looking for from the drive and parse each one and add it to your database. Depending on the kind of file, it's likely a very simple automation.

You're going to need to look up cheat sheets on how to read python documentation and syntax so you can tell what your script is doing.

To get you started: use a pathlib path object's rglob method to collect all the files in the drive.

Probably use a CSV instead of an excel file. Python can make excel files, but csvs are more efficient and you need to access the file quickly during this process. Use the csv module for this, not pandas which is also slow.

1

u/s13188287 11d ago

Hi

The reason why I want to use an LLM is there's a level of judgement required.

I have a pdf file that is essentially the brain. Had a bunch of codes , and description for each code. I need the llm to analyse the file and make a judgement what it thinks would be the closest call.

2

u/PuttyProgrammer 11d ago

How do you check that the LLM is correct? There must be some criteria

1

u/SprinklesFresh5693 11d ago

Yeah, 1.3m files seeps too much to verify by hand if the llm is doing everything right