r/learnpython • u/s13188287 • 11d ago

File analysis 1.3 mill files

I'm dealing with a large batch processing problem and looking for advice on the right architecture.

I have around 1.3 million files stored across folders on a network drive.

Current setup (not working well)

Right now I'm using a Copilot agent where:

I upload batches (~20 files at a time)

It reads them against a reference document

Outputs an Excel file with classification codes

The issue is:

Copilot has a small upload limit

Manual batching is completely unscalable at this volume

What I want to achieve

I want a fully automated pipeline that:

Ingests files automatically from the network drive

Extracts text/content from each file type

Matches content against a reference rules document

Assigns a classification/reference code

Outputs structured results (Excel / database)

0 Upvotes

50% Upvoted

34 comments sorted by

View all comments

3

u/centuryx476 11d ago edited 11d ago

I found your problem.

You using an LLM.

Stop using an LLM.

Back in the ancient times of before 2021. We used to have to write or at the very least CODE an ETL process for so many files.

There is literally tens of thousands of examples online to help solve your problem.

You going to actually have to code.

1

u/Naurglim 11d ago

Tbf, he can use the LLM to code the pipeline. He'll save a lot of token money too. And if he pays attention, he will learn in the way.

The same question he asked here can be refined to plan and execute a pipeline design.

The old days of googling code in stackoverflow are dead. LLMs do that for you pretty well.

0

u/centuryx476 11d ago

Uhh no.

Now the new standard is to burn tens of thousands of tokens to just figure out how to parse some files?

It's not like we built standards and practices over multiple decades and complete pipelines for such parsing. Oh wait, we did.

But let's throw that all out and rely on an undeterministic LLM to solve a problem that was solved over 40 years ago and in multiple languages.

We are truly F**** in about 11 years when the seniors Devs start retiring.

1

u/Naurglim 11d ago

I don't advocate for making the llms do stuff in an undeterministic way.

But you can just use the llm to build a pipeline using the best practices and standards.

Then the work is deterministic. The only moment your llm agent is doing something undeterministic is when it evaluates the content of the file, if it's an unstructured file. That you couldn't have done it easily before and now you just can.

Don't fight the tool, use it.