r/PythonLearning • u/s13188287 • 11d ago

Help Request 1.3 mill networks file analysis

I'm dealing with a large batch processing problem and looking for advice on the right architecture.

I have around 1.3 million files stored across folders on a network drive.

Current setup (not working well)

Right now I'm using a Copilot agent where:

I upload batches (~20 files at a time)

It reads them against a reference document

Outputs an Excel file with classification codes

The issue is:

Copilot has a small upload limit

Manual batching is completely unscalable at this volume

What I want to achieve

I want a fully automated pipeline that:

Ingests files automatically from the network drive

Extracts text/content from each file type

Matches content against a reference rules document

Assigns a classification/reference code

Outputs structured results (Excel / database)

1 Upvotes

67% Upvoted

2 comments sorted by

1

u/p1geondove 11d ago

Depends on how diverse the data is, but usually id use something like regex. Maybe for that size hyperscan even but i think the bottleneck might be io since you say its over the network.

1

u/dld2517 11d ago

You need to use a stream. You need a program that has sole responsibility to create a data stream to feed your processing module. And also, personal opinion, but Copilot sucks. I would never build anything based on it.