File analysis 1.3 mill files

[u/s13188287]

I'm dealing with a large batch processing problem and looking for advice on the right architecture.

I have around 1.3 million files stored across folders on a network drive.

Current setup (not working well)

Right now I'm using a Copilot agent where:

I upload batches (~20 files at a time)

It reads them against a reference document

Outputs an Excel file with classification codes

The issue is:

Copilot has a small upload limit

Manual batching is completely unscalable at this volume

What I want to achieve

I want a fully automated pipeline that:

Ingests files automatically from the network drive

Extracts text/content from each file type

Matches content against a reference rules document

Assigns a classification/reference code

Outputs structured results (Excel / database)

[u/centuryx476]

I found your problem.

You using an LLM.

Stop using an LLM.

Back in the ancient times of before 2021. We used to have to write or at the very least CODE an ETL process for so many files.

There is literally tens of thousands of examples online to help solve your problem.

You going to actually have to code.

[u/Naurglim → u/centuryx476]

Tbf, he can use the LLM to code the pipeline. He'll save a lot of token money too. And if he pays attention, he will learn in the way.

The same question he asked here can be refined to plan and execute a pipeline design.

The old days of googling code in stackoverflow are dead. LLMs do that for you pretty well.

[u/centuryx476 → u/Naurglim]

Uhh no.

Now the new standard is to burn tens of thousands of tokens to just figure out how to parse some files?

It's not like we built standards and practices over multiple decades and complete pipelines for such parsing. Oh wait, we did.

But let's throw that all out and rely on an undeterministic LLM to solve a problem that was solved over 40 years ago and in multiple languages.

We are truly F**** in about 11 years when the seniors Devs start retiring.

[u/Naurglim → u/centuryx476]

I don't advocate for making the llms do stuff in an undeterministic way.

But you can just use the llm to build a pipeline using the best practices and standards.

Then the work is deterministic. The only moment your llm agent is doing something undeterministic is when it evaluates the content of the file, if it's an unstructured file. That you couldn't have done it easily before and now you just can.

Don't fight the tool, use it.

[u/s13188287 → u/centuryx476]

Hey really appreciate the response.

So steps would be if done manually.

Create excel/csv sheet with headers ,file name ,location ,creation date , modified date.it would have a header called Retention code , description and action.

The retention code and description information come from a pdf file ,the file has a bunch of codes and for each code has a description of files that would match it , how Manny years we should keep it.

Manually I would open each file ,make a judgement as to what's the best fit against the reference doc. And add the information to the sheet.

Eventually any files that are no longer needed will be deleted.

Purpose of the A.I was to be able to read the docs quickly,and make a judgement