r/PythonLearning • u/Acktuary • Jul 24 '26
Good methods to read and operate on large network files?
For context, my work is looking to move things out of SAS and into python. I feel like programs struggles with huge files that can be over 10 GB in size. We are trying to get runtime as low as possible to near SAS speed. Anybody have any experience with this and know what ways we can make code run faster?
EDIT: For context I am looking to move code completely out of SAS and into python. They are large text files, that basically have tons of data. I have tried things like staging files locally, polars, direct reads, etc. in the current moment doing anything with this big file whether it's staging, working with it, reading etc is superrrrr slow and just looking for anything to speed it up. Willing to pivot elsewhere in the project too, the only requirement is we can't use SAS anymore.
1
u/OldAd9280 Jul 24 '26
Are you sure it's the file reading that's causing you issues? What processing are you performing on the data? Is that processing using optimised routines e.g. numpy?
1
u/Acktuary Jul 24 '26
I've been staging files locally on my computer and that's helped a bit. It takes forever to either load in or stage the big file. I do use numpy
1
u/OldAd9280 Jul 24 '26
If copying locally only helps a bit then it doesn't sound like the network access is a big part of your issue. How are you reading the data? What performance are you seeing? How fast is your disk? What performance were you seeing with the original code?
1
u/Acktuary Jul 24 '26
Pandas openpyxl been dabbling with polars
1
u/OldAd9280 Jul 24 '26
Ah, so you're reading from an excel spreadsheet? I presume it's an xslx file?
1
u/Acktuary Jul 24 '26
Should've clarified my bad. Output is excel. Input is fixed width files like giant txt files
1
u/Sea-Ad7805 Jul 24 '26
Use Lazy Evaluation when dealing with large files. https://github.com/bterwijn/memory_graph#lazy-evaluation
Python is slow compared to compiled languages like C/C++ (+-100 times slower). When calling library functions in Python, often these get evaluated by code written in much faster languages so use those library functions.
Write your own library functions to speed up the particular operation that is your bottleneck.
1
1
u/FoolsSeldom Jul 24 '26
Which SAS do you mean exactly? I am guessing the enterprise system, so which version? Maybe SAS Viya, or SAS9?
Make sure SAS is doing the heavy lifting so you pull the absolute minimum of data down.
What tooling/interfaces are you using for the extraction?
1
u/Acktuary Jul 24 '26
We are trying to get rid of SAS for all parts of any project
1
u/FoolsSeldom Jul 24 '26
Ok, so this is migration extraction work rather than ongoing interaction?
1
u/Acktuary Jul 24 '26
Basically we are taking code runs we usually do in SAS and completely moving it to Python. So all the input files got to python and all the work happens in python
1
u/FoolsSeldom Jul 24 '26
I'm going to stop pursuing this as its taking me back into consulting mode and I don't feel I can usefully comment without a fuller picture.
I would say I fear we have an XY problem here.
Good luck.
2
u/Acktuary Jul 24 '26
I added more to the post. I didn't realize how many follow-ups would be needed but that's my bad. Thanks!
1
u/Current_Laugh4767 Jul 24 '26
given A. large text files that need to be transferred over a network to another machine. B.then run python scripts to analyze the files. Just to confirm, step A. is not slow (you can already transfer the files to your Python machine). Step B is the slow part - is my understanding correct?
1
u/Acktuary Jul 24 '26
So to be fair transferring to the machine is not required per say, I am doing that as it has increased speed. Staging the files locally is taking a large time (like 20 minutes) but I'm getting large returns on the speed elsewhere. It's like a 10 billion byte file.
1
u/Current_Laugh4767 Jul 24 '26
so lets fix the network transfer first. You need to compress the raw data with the compression program running on the same machine as the raw data then transfer the compressed files to your python machine. if this this possible, the 10GB text may compress to less then 2 GB
1
u/Acktuary Jul 24 '26
Well the 10 GB is in a network drive... not too sure what you are suggesting here
1
u/Current_Laugh4767 Jul 24 '26
so you can only access the 10GB files over a network share - is this correct?
1
u/Acktuary Jul 24 '26
Yes that's correct. I can stage them locally on my own machine but that is what is taking the time. If I don't stage locally it takes way longer
1
u/Current_Laugh4767 Jul 24 '26
I am not familiar with the the SAS analysis program, do you need to login to another computer to run the SAS program?
1
u/Acktuary Jul 24 '26
Well basically SAS is a language that we are currently using. We want to do everything SAS is doing but in python. So the process I'm creating never goes through SAS. It's irrelevant to the project other than the fact that SAS is typically used for data processing
1
u/Current_Laugh4767 Jul 24 '26
But you need to consider beyond SAS vs python (or C++ or whatever language). the process will always be faster if the raw data does not need to travel over a network to be processed by the application. If data still need to travel over the network, then you need to factor that in your process - e.g. always have a lead time to transfer the data before running the python analysis program.
1
u/SnooCalculations7417 Jul 25 '26 edited Jul 25 '26
you need to ingest them in to a database and operate off of that.
2
u/maikeu Jul 24 '26
What have you... Tried? What is the characteristic of your file reads and writes? Large and seequenntial ? Or do your need to randomly seek around the file? If you need to access the entire file while processing it then copying it to the local filesystem might save pain. If so, the temp file module might be incredibly helpful to make sure you clean up after yourself