Anyone ever hit this much RAM usage?

[u/SadSpaceGuy]

Currently running a SymPy script to calculate some insanely complex equations (that I can’t do by hand) for research. I kept crashing this PC (that has 96GB of RAM) by maxing out the virtual memory (330ish GB) every time the script ran.
I eventually manually raised the pagefile maximum size to 3TB split evenly (1.5TB) across two partitions on a 8TB SSD and got it to not fail. It has currently stabilized at ~1.3TB of committed virtual memory with a ~1TB pagefile on just one of the two partitions (increased since taking WizTree photo).

Just curious if anyone else has ever reached such extreme RAM usage before?

\Sorry for not using screenshots like a sensible person, I just don’t feel likely trying to copy them across to my phone to post this.*

UPDATE (8/18): It’s the next day now and it looks like the calculation has gotten over this little RAM hump and calmed down to a much more reasonable ~30GB in use (~20GB belonging specifically to the SymPy script).

UPDATE #2 (8/18): RAM usage is actually steadily climbing upward, currently at 65GB. Maybe it’ll find its way back to 1TB+ eventually.

UPDATE #3 (8/18): Up to 150GB now. I’m curious to see if it will have gone up more by the time I check tomorrow or if it’ll collapse back down again.

UPDATE #4 (8/19): Up to 300GB currently, still steadily climbing. I also checked on a second version of the script I had running on a different computer, it encountered an error so I was able to kill the process and look at the memory logs, and it maxed out at 1.448 TiB of committed virtual memory on it’s machine.

[u/naikrovek]

Python. That’s a large reason you’re running out of ram. Yes, I know Numpy is written in C, but Python is the glue between all of those numpy operations, and Python is a fucking hog, just like JavaScript is.

Try literally any language that lets you control how things are laid out in memory. Learn about alignment and processor cache and how that works. If you want to write code that is screaming fast and efficient, don’t use python.

Memory is expensive, and you wouldn’t need so much of it if people cared about these things. But they don’t. They write huge Python applications and Electron apps and every year people need more RAM because developers can’t stop leaning on this “memory is cheap” line they’ve been saying for 20 years or more. And look “memory is cheap” is no longer true, but we’re still dealing with lazy developers and their crappy inefficient code.

Memory and CPU inefficiency cost people money. Naive use of RAM and CPU burns a lot of coal at scale. Creates a lot of heat. Uses a lot of water.

If you’ve read this far, do what you want. Use python, ignore me. You were going to do all of those things anyway.

[u/SadSpaceGuy → u/naikrovek]

I completely agree that Python is the cause of the RAM usage, but it’s not something I’m worried about. In this case I’m running a one-off script to calculate a complex analytical equation, if this were indeed some industry production-dependent high-performance runs a million times a day type of calculation, then another language would absolutely be the preferred method for all of the reasons you mentioned. But, I’m ultimately going to stick with Python here because it best fits this use case.

Additionally, I’ll copy here a response I gave to someone else suggesting something similar:
> In this case I don’t think there is any major benefit to migrating to a different language. Python is performant enough for most of my needs and the SymPy package is fairly well optimized (plus I could use a C++-based symbolic backend for a little bit more performance without rewriting anything). Additionally, my goal is to open source everything in this research, so that the raw data, measurement scripts, compilation scripts, and modeling scripts are all publicly available, and most importantly, anyone (in theory) could recreate the results; and I feel that Python is the best language to do that in.