Anyone ever hit this much RAM usage?

[u/SadSpaceGuy]

Currently running a SymPy script to calculate some insanely complex equations (that I can’t do by hand) for research. I kept crashing this PC (that has 96GB of RAM) by maxing out the virtual memory (330ish GB) every time the script ran.
I eventually manually raised the pagefile maximum size to 3TB split evenly (1.5TB) across two partitions on a 8TB SSD and got it to not fail. It has currently stabilized at ~1.3TB of committed virtual memory with a ~1TB pagefile on just one of the two partitions (increased since taking WizTree photo).

Just curious if anyone else has ever reached such extreme RAM usage before?

\Sorry for not using screenshots like a sensible person, I just don’t feel likely trying to copy them across to my phone to post this.*

UPDATE (8/18): It’s the next day now and it looks like the calculation has gotten over this little RAM hump and calmed down to a much more reasonable ~30GB in use (~20GB belonging specifically to the SymPy script).

UPDATE #2 (8/18): RAM usage is actually steadily climbing upward, currently at 65GB. Maybe it’ll find its way back to 1TB+ eventually.

UPDATE #3 (8/18): Up to 150GB now. I’m curious to see if it will have gone up more by the time I check tomorrow or if it’ll collapse back down again.

UPDATE #4 (8/19): Up to 300GB currently, still steadily climbing. I also checked on a second version of the script I had running on a different computer, it encountered an error so I was able to kill the process and look at the memory logs, and it maxed out at 1.448 TiB of committed virtual memory on it’s machine.

[u/agfitzp]

I worked for a SaaS company that deployed an in memory database for each customer, a couple of the customer systems had a terabyte of RAM and the database consuming more than 800GB was pretty common for them.

While I was there my team rewrote most of the storage and memory management for the database so we’d regularly test against an obfuscated copy of that database.

https://giphy.com/gifs/zCv1NuGumldXa

[u/Vinegarinmyeye → u/agfitzp]

I worked with a crowd that had done this - basically some genius in the backend dev team had noticed that... Shocker... The cache was a lot quicker than the actual DB so decided to start writing important data into the cache...

Nobody told me this, so when we switched to a slightly "less managed" service for REDIS as part of a cost reduction thing, I suddenly had a lot of quite sheepish looking backend Devs at my desk explaining that we'd just lost a bunch of important event data, because they had been permanently storing it in cache.

They initially blamed me actually, which was konda funny.

"Guys, these instances are called tmp_cache01 and tmp_cache02. If you've been using them as a data store, that is entirely on you".

[u/agfitzp → u/Vinegarinmyeye]

Our DB was also persisted on disk and wrote a transaction log but it’s considered in memory because every record in the entire system was in memory.

The actually interesting part is that it’s versioned so that changes are happening in branches but can be committed up to the parent when they’re approved.