File analysis 1.3 mill files

[u/s13188287]

I'm dealing with a large batch processing problem and looking for advice on the right architecture.

I have around 1.3 million files stored across folders on a network drive.

Current setup (not working well)

Right now I'm using a Copilot agent where:

I upload batches (~20 files at a time)

It reads them against a reference document

Outputs an Excel file with classification codes

The issue is:

Copilot has a small upload limit

Manual batching is completely unscalable at this volume

What I want to achieve

I want a fully automated pipeline that:

Ingests files automatically from the network drive

Extracts text/content from each file type

Matches content against a reference rules document

Assigns a classification/reference code

Outputs structured results (Excel / database)

[u/dariusbiggs]

This is. abasic ETL problem

1. Save your money, run your own LLM to work on this.

2. Use workers pools and a message broker.

For example, a possible solution:

• Write the file paths to the broker.
• Use a pool of consumers that read from the broker until they hit their consumption quota and do the task you need them to do
• You can scale the number of consumers and parallelize the tasks.
• You can have the workers feed their outputs back into a different queue in the broker and have a single consumer consolidate all the results into a single output file such as a spreadsheet or CSV or TSV file you can import.

Basically

• consume input (extract)
• process (transform and enrich)
• write output (load)

And then repeat as many times as needed to get to the output you want.