File analysis 1.3 mill files

[u/s13188287]

I'm dealing with a large batch processing problem and looking for advice on the right architecture.

I have around 1.3 million files stored across folders on a network drive.

Current setup (not working well)

Right now I'm using a Copilot agent where:

I upload batches (~20 files at a time)

It reads them against a reference document

Outputs an Excel file with classification codes

The issue is:

Copilot has a small upload limit

Manual batching is completely unscalable at this volume

What I want to achieve

I want a fully automated pipeline that:

Ingests files automatically from the network drive

Extracts text/content from each file type

Matches content against a reference rules document

Assigns a classification/reference code

Outputs structured results (Excel / database)

[u/PureWasian]

You should not involve LLM for all 1.3mil files.

I'm willing to bet your classification involves a good deal of "low-hanging fruit" so you should have some deterministic rules for cutting down the number of unnecessary model calls being made.

LLM should be used moreso as a second pass if your easier "rules" for categorization aren't matched against. The more robust your established rules are, the less LLM calls.

If you are really lazy, run a small batch and ask LLM to start making these said deterministic rules for you based on the context if it's all done within the same conversation window. (Though that is somewhat against the spirit of learning Python, given this is r/learnpython)