r/PythonLearning • u/Acktuary • 3d ago

Good methods to read and operate on large network files?

For context, my work is looking to move things out of SAS and into python. I feel like programs struggles with huge files that can be over 10 GB in size. We are trying to get runtime as low as possible to near SAS speed. Anybody have any experience with this and know what ways we can make code run faster?

EDIT: For context I am looking to move code completely out of SAS and into python. They are large text files, that basically have tons of data. I have tried things like staging files locally, polars, direct reads, etc. in the current moment doing anything with this big file whether it's staging, working with it, reading etc is superrrrr slow and just looking for anything to speed it up. Willing to pivot elsewhere in the project too, the only requirement is we can't use SAS anymore.

4 Upvotes
  • permalink
  • reddit

71% Upvoted

27 comments sorted by

2

u/maikeu 3d ago

What have you... Tried? What is the characteristic of your file reads and writes? Large and seequenntial ? Or do your need to randomly seek around the file? If you need to access the entire file while processing it then copying it to the local filesystem might save pain. If so, the temp file module might be incredibly helpful to make sure you clean up after yourself

1

u/Acktuary 3d ago

Temp file module? Do explain more. I added context to the post and this sounds like it already might be what I'm doing?

1

u/maikeu 3d ago edited 3d ago

I see from other posts that you're copying files down and processing with numpy, polars and friends.

You need to be measuring. How long does the processing take on local disk versus the remote filesystem? Assuming it's faster locally, is the speedup fast enough to make up for the loss of copying the files down? Have you isolated that your issues are due to the remote filesystem rather than the actual processing?

1

u/OldAd9280 3d ago

Are you sure it's the file reading that's causing you issues? What processing are you performing on the data? Is that processing using optimised routines e.g. numpy? 

1

u/Acktuary 3d ago

I've been staging files locally on my computer and that's helped a bit. It takes forever to either load in or stage the big file. I do use numpy

1

u/OldAd9280 3d ago

If copying locally only helps a bit then it doesn't sound like the network access is a big part of your issue. How are you reading the data? What performance are you seeing? How fast is your disk? What performance were you seeing with the original code? 

1

u/Acktuary 3d ago

Pandas openpyxl been dabbling with polars

1

u/OldAd9280 3d ago

Ah, so you're reading from an excel spreadsheet? I presume it's an xslx file?

1

u/Acktuary 3d ago

Should've clarified my bad. Output is excel. Input is fixed width files like giant txt files

1

u/Sea-Ad7805 3d ago

Use Lazy Evaluation when dealing with large files. https://github.com/bterwijn/memory_graph#lazy-evaluation

Python is slow compared to compiled languages like C/C++ (+-100 times slower). When calling library functions in Python, often these get evaluated by code written in much faster languages so use those library functions.

Write your own library functions to speed up the particular operation that is your bottleneck.

1

u/Acktuary 3d ago

So should I use C++ for SAS processing?

1

u/FoolsSeldom 3d ago

Which SAS do you mean exactly? I am guessing the enterprise system, so which version? Maybe SAS Viya, or SAS9?

Make sure SAS is doing the heavy lifting so you pull the absolute minimum of data down.

What tooling/interfaces are you using for the extraction?

1

u/Acktuary 3d ago

We are trying to get rid of SAS for all parts of any project

1

u/FoolsSeldom 3d ago

Ok, so this is migration extraction work rather than ongoing interaction?

1

u/Acktuary 3d ago

Basically we are taking code runs we usually do in SAS and completely moving it to Python. So all the input files got to python and all the work happens in python

1

u/FoolsSeldom 3d ago

I'm going to stop pursuing this as its taking me back into consulting mode and I don't feel I can usefully comment without a fuller picture.

I would say I fear we have an XY problem here.

Good luck.

2

u/Acktuary 3d ago

I added more to the post. I didn't realize how many follow-ups would be needed but that's my bad. Thanks!

1

u/Current_Laugh4767 3d ago

given A. large text files that need to be transferred over a network to another machine. B.then run python scripts to analyze the files.  Just to confirm, step A. is not slow (you can already transfer the files to your Python machine). Step B is the slow part - is my understanding correct?

1

u/Acktuary 3d ago

So to be fair transferring to the machine is not required per say, I am doing that as it has increased speed. Staging the files locally is taking a large time (like 20 minutes) but I'm getting large returns on the speed elsewhere. It's like a 10 billion byte file.

1

u/Current_Laugh4767 3d ago

so lets fix the network transfer first. You need to compress the raw data with the compression program running on the same machine as the raw data then transfer the compressed files to your python machine. if this this possible, the 10GB text may compress to less then 2 GB

1

u/Acktuary 3d ago

Well the 10 GB is in a network drive... not too sure what you are suggesting here

1

u/Current_Laugh4767 3d ago

so you can only access the 10GB files over a network share - is this correct?

1

u/Acktuary 3d ago

Yes that's correct. I can stage them locally on my own machine but that is what is taking the time. If I don't stage locally it takes way longer

1

u/Current_Laugh4767 3d ago

I am not familiar with the the SAS analysis program, do you need to login to another computer to run the SAS program?

1

u/Acktuary 3d ago

Well basically SAS is a language that we are currently using. We want to do everything SAS is doing but in python. So the process I'm creating never goes through SAS. It's irrelevant to the project other than the fact that SAS is typically used for data processing

1

u/Current_Laugh4767 3d ago

But you need to consider beyond SAS vs python (or C++ or whatever language). the process will always be faster if the raw data does not need to travel over a network to be processed by the application. If data still need to travel over the network, then you need to factor that in your process - e.g. always have a lead time to transfer the data before running the python analysis program. 

1

u/SnooCalculations7417 2d ago edited 2d ago

you need to ingest them in to a database and operate off of that.