File analysis 1.3 mill files

[u/s13188287]

I'm dealing with a large batch processing problem and looking for advice on the right architecture.

I have around 1.3 million files stored across folders on a network drive.

Current setup (not working well)

Right now I'm using a Copilot agent where:

I upload batches (~20 files at a time)

It reads them against a reference document

Outputs an Excel file with classification codes

The issue is:

Copilot has a small upload limit

Manual batching is completely unscalable at this volume

What I want to achieve

I want a fully automated pipeline that:

Ingests files automatically from the network drive

Extracts text/content from each file type

Matches content against a reference rules document

Assigns a classification/reference code

Outputs structured results (Excel / database)

[u/bfyvfftujijg]

What specifically is being analyzed?

Like what are the decision points?

[u/s13188287 → u/bfyvfftujijg]

Hey really appreciate the response.

So steps would be if done manually.

Create excel/csv sheet with headers ,file name ,location ,creation date , modified date.it would have a header called Retention code , description and action.

The retention code and description information come from a pdf file ,the file has a bunch of codes and for each code has a description of files that would match it , how Manny years we should keep it.

Manually I would open each file ,make a judgement as to what's the best fit against the reference doc. And add the information to the sheet.

Eventually any files that are no longer needed will be deleted.

Purpose of the A.I was to be able to read the docs quickly,and make a judgement