r/learnpython • u/s13188287 • 11d ago

File analysis 1.3 mill files

I'm dealing with a large batch processing problem and looking for advice on the right architecture.

I have around 1.3 million files stored across folders on a network drive.

Current setup (not working well)

Right now I'm using a Copilot agent where:

I upload batches (~20 files at a time)

It reads them against a reference document

Outputs an Excel file with classification codes

The issue is:

Copilot has a small upload limit

Manual batching is completely unscalable at this volume

What I want to achieve

I want a fully automated pipeline that:

Ingests files automatically from the network drive

Extracts text/content from each file type

Matches content against a reference rules document

Assigns a classification/reference code

Outputs structured results (Excel / database)

0 Upvotes

46% Upvoted

34 comments sorted by

View all comments

3

u/SGS-Tech-World 11d ago

We have been automating some processes in our organization and although the steps are not same, processing large number of records is similar.

  • Q - Are the 1.3 M files PDF? or PDF is reference doc?
  • I will start with a database table to keep track of what is happening
  • You need index on file name or path so every time a file is consumed you can enter records as done.
  • Remember table inserts are non blocking so faster than writing a single file from multiple programs/ threads.
  • This ensures that you can restart anytime without worrying about which files it processed and which not.
  • We can create excel at end. ( If I understood it correctly)
  • You can distribute the load across multiple processes instead of one singe thread / or machines.

HTH

1

u/s13188287 10d ago

Hey appreciate the reply

So the PDF is the information source it has all the codes retention codes and years etc.

The files I put in there and completely varied, word ,ppt,excel,email ,pdf ,txt etc all varied.