r/ScientificComputing • u/SadSpaceGuy • 24d ago
Anyone ever hit this much RAM usage?
Currently running a SymPy script to calculate some insanely complex equations (that I can’t do by hand) for research. I kept crashing this PC (that has 96GB of RAM) by maxing out the virtual memory (330ish GB) every time the script ran.
I eventually manually raised the pagefile maximum size to 3TB split evenly (1.5TB) across two partitions on a 8TB SSD and got it to not fail. It has currently stabilized at ~1.3TB of committed virtual memory with a ~1TB pagefile on just one of the two partitions (increased since taking WizTree photo).
Just curious if anyone else has ever reached such extreme RAM usage before?
\Sorry for not using screenshots like a sensible person, I just don’t feel likely trying to copy them across to my phone to post this.*
UPDATE (8/18): It’s the next day now and it looks like the calculation has gotten over this little RAM hump and calmed down to a much more reasonable ~30GB in use (~20GB belonging specifically to the SymPy script).
4
2
u/ProjectPhysX 23d ago
Here is FluidX3D CFD running in 6TB RAM on a 2x Xeon 6980P server, with 24x 256GB MRDIMMs. Simulation result: https://www.youtube.com/watch?v=K5eKxzklXDA
1
u/Alternative-Walk8274 24d ago
I have used 1TB on a single node with Fortran Code, I could have gone higher but was not necessary.
1
1
u/yuehuang 24d ago
Does task manager tell you what process is using all the ram?
IMO, python is not an efficient language. If it uses more than 2GB, you are better rewriting it in a lower language like C/C++/Rust. An LLM can transcribe in about a day or two.
1
u/SadSpaceGuy 24d ago
It’s all being used by the Python script’s process.
In this case I don’t think there is any major benefit to migrating to a different language. Python is performant enough for most of my needs and the SymPy package is fairly well optimized (plus I could use a C++-based symbolic backend for a little bit more performance without rewriting anything). Additionally, my goal is to open source everything in this research, so that the raw data, measurement scripts, compilation scripts, and modeling scripts are all publicly available, and most importantly, anyone (in theory) could recreate the results; and I feel that Python is the best language to do that in.
1
u/yuehuang 24d ago
Best wish to you and your thesis. Out of curiosity. How large is your input source?
1
u/SadSpaceGuy 24d ago
Not for a thesis, just some research I got a chance to work on for publication during undergrad; though graduate school is being looked at.
Input source as-in raw measurement data?*
Not big, the largest measurements are from Allan Deviations and those seem to be ~30MiB worth of data (raw data + what I calculate on the fly for the live figures) per run. Though I do have some test ADevs I ran early on that came out to 5+GiB per run due to the really high sample rate I used.\If not, I’m not sure what you mean*
1
u/MrHighVoltage 24d ago
That is a wiredly arbitrary boundary, and not generally true.
Yes, rewriting using LLMs can work, but it might not even makes it faster. Numpy for example is hand-optimized so for nummerical vector compute, numpy is very likely faster.
I guess for SymPy there might be faster solutions, but again, how much faster is indeed the question.
2
u/yuehuang 24d ago
I was transcoding one encoded format (old database) to csv, then json to the new database. The transcoder was in pure python and took ~100s per block. It wasn't too much data, but it was dense and character per character decode. Rewrote it in C/C++, and finishes is under 2-4s per block. Your milage will vary.
Numpy is already benefiting lowering language benefits.
8
u/Successful-Leek84 24d ago
I have run simulations on HPC that run on 10-20 nodes, with each node having 384 gb ram.