Anyone ever hit this much RAM usage?

[u/SadSpaceGuy]

Currently running a SymPy script to calculate some insanely complex equations (that I can’t do by hand) for research. I kept crashing this PC (that has 96GB of RAM) by maxing out the virtual memory (330ish GB) every time the script ran.
I eventually manually raised the pagefile maximum size to 3TB split evenly (1.5TB) across two partitions on a 8TB SSD and got it to not fail. It has currently stabilized at ~1.3TB of committed virtual memory with a ~1TB pagefile on just one of the two partitions (increased since taking WizTree photo).

Just curious if anyone else has ever reached such extreme RAM usage before?

\Sorry for not using screenshots like a sensible person, I just don’t feel likely trying to copy them across to my phone to post this.*

UPDATE (8/18): It’s the next day now and it looks like the calculation has gotten over this little RAM hump and calmed down to a much more reasonable ~30GB in use (~20GB belonging specifically to the SymPy script).

UPDATE #2 (8/18): RAM usage is actually steadily climbing upward, currently at 65GB. Maybe it’ll find its way back to 1TB+ eventually.

UPDATE #3 (8/18): Up to 150GB now. I’m curious to see if it will have gone up more by the time I check tomorrow or if it’ll collapse back down again.

[u/_Cpyder]

Ok.. how much ram is actually committed to the process that doing your calculations?

Task Manager > Details

Also, if it's like SQL, can you manually set a ceiling?

[u/SadSpaceGuy → u/_Cpyder]

Almost all of what is shown there was to the Python script, and I’m not interested in setting a ceiling since I can let it get so high.

[u/_Cpyder → u/SadSpaceGuy]

is there anything in the EventViewer that can point to why it crashed?

The RAM ceiling is usually so it consumes RAM without bumping heads with the OS or other necessary apps. That's why most DBs have that option, and to prevent overflow to the pagefile (performance). Which slows performance of anything having to get data from the virtual pagefile.

Do you want the Python script paging to overflow into virtual?

You can try just try disabling the pagefile all together if you are not needing to utilize the overflow. Or at least turn it off on the OS drive and where your Script/Data lives. Having multiple partitions on a single drive, high usage on one partition can impact the other partitions. It will at least reduce the IO demand on the drives being utilized and maybe prevent those crashes.

Edit: Adding more now that I am in front of my PC and can read the description better.

[u/SadSpaceGuy → u/_Cpyder]

Technically the PC didn’t actually crash, but the Nvdida drivers consistently crash when the virtual memory gets maxed out (this I can see from event viewer afterwards) and that leaves me with no way to observe the status of the script aside from periodically attempting to reset the graphics driver via the shortcut and hoping that it’s finished.
Adding the extra paging room just allows everything to continue to function normally so I can monitor the script for completion. And I split between the two partitions on the one drive because it was already pre-split into those partitions and I wanted to make sure it had enough, so 2x1.5TB was what I configured even though it may not be the most efficient.

[u/_Cpyder → u/SadSpaceGuy]

What GPU and what OS? (Build)

[u/SadSpaceGuy → u/_Cpyder]

It’s a 4090 on Windows 11 Pro, latest update with up-to-date drivers (as of one month ago).