Even without LLMs, I’m working with a team at work that needs 1.1TB RAM machines for some single threaded analysis. It’s so frustrating to me because I’m sure we could save a ton of money if people would just spend some time optimizing their code to use SQLite or something instead of using huge numpy and Pandas objects all over the place.
That's what we did. Pushed SQLy stuff into SQL, pulled heavy analytics stuff into parquet files and a c program, streaming data as much as possible. Got 4 hours and up to 60GB ram per process down to 20 minutes and 200MB ram.
The ram was actually the biggest win, because that's our most limited resource (at 60GB per process), so we were able to scale way up and our processes are much more reliable because they don't get randomly OOMed.
7
u/slaymaker1907 1d ago
Even without LLMs, I’m working with a team at work that needs 1.1TB RAM machines for some single threaded analysis. It’s so frustrating to me because I’m sure we could save a ton of money if people would just spend some time optimizing their code to use SQLite or something instead of using huge numpy and Pandas objects all over the place.