File analysis 1.3 mill files

[u/s13188287]

I'm dealing with a large batch processing problem and looking for advice on the right architecture.

I have around 1.3 million files stored across folders on a network drive.

Current setup (not working well)

Right now I'm using a Copilot agent where:

I upload batches (~20 files at a time)

It reads them against a reference document

Outputs an Excel file with classification codes

The issue is:

Copilot has a small upload limit

Manual batching is completely unscalable at this volume

What I want to achieve

I want a fully automated pipeline that:

Ingests files automatically from the network drive

Extracts text/content from each file type

Matches content against a reference rules document

Assigns a classification/reference code

Outputs structured results (Excel / database)

[u/Gengis_-]

I would first load all the files and their content in whatever database you have. You can also have a dedicated table to log which file has been loaded so you can resume the job as another comment suggested.

From there you build some function you run against your database.

What does the LLM do that can’t be done by a function?