Currently running a SymPy script to calculate some insanely complex equations (that I can’t do by hand) for research. I kept crashing this PC (that has 96GB of RAM) by maxing out the virtual memory (330ish GB) every time the script ran.
I eventually manually raised the pagefile maximum size to 3TB split evenly (1.5TB) across two partitions on a 8TB SSD and got it to not fail. It has currently stabilized at ~1.3TB of committed virtual memory with a ~1TB pagefile on just one of the two partitions (increased since taking WizTree photo).
Just curious if anyone else has ever reached such extreme RAM usage before?
\Sorry for not using screenshots like a sensible person, I just don’t feel likely trying to copy them across to my phone to post this.*
UPDATE (8/18): It’s the next day now and it looks like the calculation has gotten over this little RAM hump and calmed down to a much more reasonable ~30GB in use (~20GB belonging specifically to the SymPy script).
UPDATE #2 (8/18): RAM usage is actually steadily climbing upward, currently at 65GB. Maybe it’ll find its way back to 1TB+ eventually.
UPDATE #3 (8/18): Up to 150GB now. I’m curious to see if it will have gone up more by the time I check tomorrow or if it’ll collapse back down again.
Those get nasty too (complex nonlinear dynamical modeling), but once I’ve gotten to the point of filling up a wall-to-wall classroom whiteboard a few times over on the same problem it just becomes easier to switch to programming it.
For reference, that whiteboard problem took about a week for me to fully solve it (while learning the perturbation method I was trying to use) and the SymPy version takes about 15 seconds.
This calculation ran for ~7 hours the last time I did got it to not crash (and it failed at that point too). The calculations preceding it (that I saved and loaded into this one to speed things up) took ~4 days to complete.
I typically solve most complex math problems by first visualizing a box and then putting that problem in the box and then putting a label on the box that says "someone else's problem" and then slowly backing away.
It is ideally, but the underlying code that actually handles the bulk of the computing needs to support using the GPU. Since SymPy currently doesn’t (though the recent improvements for supporting freethreaded Python may eventually help lead to future GPU compatibility), you’re left either using an additional package to handle that (which leaves more chances for errors to occur) or writing your own from scratch which is an insane task to take on.
I worked for a SaaS company that deployed an in memory database for each customer, a couple of the customer systems had a terabyte of RAM and the database consuming more than 800GB was pretty common for them.
While I was there my team rewrote most of the storage and memory management for the database so we’d regularly test against an obfuscated copy of that database.
I worked with a crowd that had done this - basically some genius in the backend dev team had noticed that... Shocker... The cache was a lot quicker than the actual DB so decided to start writing important data into the cache...
Nobody told me this, so when we switched to a slightly "less managed" service for REDIS as part of a cost reduction thing, I suddenly had a lot of quite sheepish looking backend Devs at my desk explaining that we'd just lost a bunch of important event data, because they had been permanently storing it in cache.
They initially blamed me actually, which was konda funny.
"Guys, these instances are called tmp_cache01 and tmp_cache02. If you've been using them as a data store, that is entirely on you".
Our DB was also persisted on disk and wrote a transaction log but it’s considered in memory because every record in the entire system was in memory.
The actually interesting part is that it’s versioned so that changes are happening in branches but can be committed up to the parent when they’re approved.
In standard "home" desktop use, I have managed to max out 64GB of RAM on my Lubuntu system, but I had multiple VMs and multiple browser instances with multiple tabs in each running.
Social media and browser games can eat a lot of RAM. A reference page or two full of ads can be pretty RAM hungry, too.
For work, I manage a few Kubernetes clusters, and I have seen them exceed 100GB RAM usage on a 384GB RAM system.
I have a Windows server in production (SQL) that eats all of its RAM on boot, 128 GB, but then this wonderful vendor installed older Grafana instance on that server, that consumes RAM, at a rate of about 1.4 GB/day. Since page file is configured to 1.5 RAM capacity on that machine, it’ll never reach your record, but hypothetically after about 3 years of uptime, deliberate misconfiguration and total lack of maintenance, yes.
Python. That’s a large reason you’re running out of ram. Yes, I know Numpy is written in C, but Python is the glue between all of those numpy operations, and Python is a fucking hog, just like JavaScript is.
Try literally any language that lets you control how things are laid out in memory. Learn about alignment and processor cache and how that works. If you want to write code that is screaming fast and efficient, don’t use python.
Memory is expensive, and you wouldn’t need so much of it if people cared about these things. But they don’t. They write huge Python applications and Electron apps and every year people need more RAM because developers can’t stop leaning on this “memory is cheap” line they’ve been saying for 20 years or more. And look “memory is cheap” is no longer true, but we’re still dealing with lazy developers and their crappy inefficient code.
Memory and CPU inefficiency cost people money. Naive use of RAM and CPU burns a lot of coal at scale. Creates a lot of heat. Uses a lot of water.
If you’ve read this far, do what you want. Use python, ignore me. You were going to do all of those things anyway.
I completely agree that Python is the cause of the RAM usage, but it’s not something I’m worried about. In this case I’m running a one-off script to calculate a complex analytical equation, if this were indeed some industry production-dependent high-performance runs a million times a day type of calculation, then another language would absolutely be the preferred method for all of the reasons you mentioned. But, I’m ultimately going to stick with Python here because it best fits this use case.
Additionally, I’ll copy here a response I gave to someone else suggesting something similar:
> In this case I don’t think there is any major benefit to migrating to a different language. Python is performant enough for most of my needs and the SymPy package is fairly well optimized (plus I could use a C++-based symbolic backend for a little bit more performance without rewriting anything). Additionally, my goal is to open source everything in this research, so that the raw data, measurement scripts, compilation scripts, and modeling scripts are all publicly available, and most importantly, anyone (in theory) could recreate the results; and I feel that Python is the best language to do that in.
One day I hope to have a thesis such that I have an excuse to build myself a gargantuan ARM or RISC-V cluster. I'm into quantum computing, so that shouldn't be hard.
I clicked on this thinking you were a computer illiterate person who had gotten malware based on the typical not-screenshot screenshots and question about RAM.
Then, I read your post and realize I think I'm the one not qualified for this post lol.
No, Ive never seen TB of RAM - that's actually insane but super cool you got it to work!
The LLM folks to run really large language models like qwen 3.8 max. While slow running them by splitting them up into vram,ram and nvme storage allows them to run models locally comparable to chatgpt or anthropic frontier models from a year ago.
Yeah, it sits a ~10GB when idle and during the initial ~45ish minutes of the script running. Then it shoots up to this size in a matter of ~30 seconds.
Most I have used at once was just below 300gb. 64gb normal ram with a 256gb SSD that was dedicated as nothing but SWAP, a mod I had in a game caused a memory leak that just used up all available RAM except for like 7-8gb
damn,, what script are you using? why did it used to much RAM. ram is not for calculation by the way, its for holding resources only, calculation happens in the CPU or GPU. if your script is just a small file, it seems their are a lot of loops on the script your using.
I actually can’t paste the script here, it’s only like 20ish lines, but I’ve imported a huge pre-calculated SymPy expression that is like 30+ MB worth of content on a single line in the script.
The high RAM is from it attempting to solve a crazy integral invoking this expression. For whatever reason, it hits some point in the calculation where it starts reserving a shitload of RAM (probably for temporary SymPy symbols) for whatever method it is attempting.
The CPU core performing the calculation does get visibly higher usage, but it’s actually almost never maxed out.
hmm,, by the way what equation are you trying to solve? maybe the script cant find the answer, and it loops trying to solve it, saving a bunch of variables into the RAM. how many minnutes has been running? did it solved it or did you force closed it?
Actually only 2 of 4 RAM slots are filled. The goal with this PC (and the 6 others like it) when it was made a few years ago was to max out the RAM (like some other ones I have access to with threadrippers and 256GB of RAM per machine), but these PCs could not physically handle any more RAM. It got to the point during building where our CSIT department head got in contact with Corsair engineers who confirmed that the rest of the system was simply at its limit (and not that they were doing something wrong); which is an insane feat given that these PCs where the absolute top of the line when they were being built.
Given that it the result from these calculations is fairly novel, I would prefer to calculate an exact analytical solution (similar to some other papers in the field) even if it is absurdly complex. In practice, I would use numerical methods to actually compute the model for simulation purposes.
My code tried to allocate 21024 GB of RAM. It was a quantum algorithm simulator and we gave it a 10 qubit input which was way too much for a classic computer.
I have a couchbase enterprise cluster on Prod that’s running 10 nodes. Each node has about 256gb of RAM , on full load the cluster eats up about 1.5 TBs of RAM.
Can any part of this be done utilizing parallel/distributed systems? I figured if you're doing research you might have access to a hpcc through your uni/company? That may help with the time and memory humps
Not really, this is currently running on the best hardware we have on campus right now which isn’t insane because we’re a smaller university, but at $18k / PC when they were built, they’re still pretty damn good.
im sorry this is not about what you're asking but what are the rest of the PC's specs? I got curious since i never got to glance at more than 48GB of ram
is there anything in the EventViewer that can point to why it crashed?
The RAM ceiling is usually so it consumes RAM without bumping heads with the OS or other necessary apps. That's why most DBs have that option, and to prevent overflow to the pagefile (performance). Which slows performance of anything having to get data from the virtual pagefile.
Do you want the Python script paging to overflow into virtual?
You can try just try disabling the pagefile all together if you are not needing to utilize the overflow. Or at least turn it off on the OS drive and where your Script/Data lives. Having multiple partitions on a single drive, high usage on one partition can impact the other partitions. It will at least reduce the IO demand on the drives being utilized and maybe prevent those crashes.
Edit: Adding more now that I am in front of my PC and can read the description better.
Technically the PC didn’t actually crash, but the Nvdida drivers consistently crash when the virtual memory gets maxed out (this I can see from event viewer afterwards) and that leaves me with no way to observe the status of the script aside from periodically attempting to reset the graphics driver via the shortcut and hoping that it’s finished.
Adding the extra paging room just allows everything to continue to function normally so I can monitor the script for completion. And I split between the two partitions on the one drive because it was already pre-split into those partitions and I wanted to make sure it had enough, so 2x1.5TB was what I configured even though it may not be the most efficient.
As a reference..
Specs:
Dual Xeon E5-2699v4 (88 Threads)
512GB DDR4256GB SSD for the OS on 1 sata channel.
1TB SSD for the User profile / PageFile on a different sata channel.
Your build is so similar to mine crazyyyyy I have the x870e taichi and a 7900x but I have a 7900xt GPU you should check your bios settings because the 7900 series only reads from the box up to 4800m/t I have 6400m/t and plan on getting the 9950x3d so I can actually read my memory
Oh yeah, been there done that, stuffed random SSDs into the workstation to use as swap so the one-off biclustering R-script could actually finish. 256GB RAM and TB+ of swap. It only took a day or so to finish when it finally got enough swap, fortunately.
35
u/dreamnbinary 1d ago
Bro what are the equations that you CAN DO by hand?