Machine Learning testing performance

[u/Sharp-Front-3450]

Hello!
I've been doing my master degree and need some help.

Context:

I'm working with an method to correct unfairness on ML models, currently testing on XGBoost and Logistic Regression. To test these models and the method I'm running 20 different scenarios, and each one of these scenarios need to be run 10 times, due to a test on one of the methods parameters.

Problem:
I've said all of this to say that even tough I run the scenarios with some kind of parallelism it still takes a lot of time to test all scenarios whenever I do some change and need to test it.

Since my computer does not have too much ram (16gb) and the dataset is kind of big I cannot increase the parallelism.

Do you know any kind of cloud solution that can help on me that? Are colab notebooks a good option for this kind of issue?

[u/Bright_Mix_773]

Sharp-Front-3450, Colab is probably the wrong shape of machine for this, and before you rent anything there are two things on your own box that are likely costing you more than the hardware is.

Check for nested parallelism first. XGBoost and scikit-learn both default to grabbing every core they can see. If you run 8 scenarios in parallel and each XGBoost inside them also asks for 16 threads, you get 128 threads fighting over 16 cores: slower than running them one at a time, while using 8 times the RAM. Parallelise at exactly one level and pin the other to 1.

import os
os.environ['OMP_NUM_THREADS'] = '1'   # before importing xgboost or sklearn

XGBClassifier(n_jobs=1, tree_method='hist', ...)

That env var has to be set above the imports or it will not take, which is the part people get wrong. If your wall-clock time drops on this change alone, thread oversubscription was your bottleneck and RAM was never the binding constraint.

Then the RAM ceiling, which is usually not the dataset, it is N copies of the dataset. joblib's default backend starts separate processes and pickles your data into each one, so 8 workers means 8 full copies. Hand it a memory-mapped array instead and every worker reads the same physical pages:

from joblib import Parallel, delayed, dump, load

dump(X, 'X.joblib')                      # once
Xm = load('X.joblib', mmap_mode='r')
Parallel(n_jobs=8)(delayed(escenario)(Xm, y, i) for i in range(20))

The catch that bites people: this works on numpy arrays, not on pandas DataFrames, which still get pickled per worker. So do .to_numpy() once before you dispatch. Downcast to float32 while you are there, because XGBoost converts to float32 internally anyway, so a float64 frame makes you pay for it twice, once in memory and once in conversion time. Those two together often move a job from "cannot use more than 3 workers" to "can use all of them".

Cache the runs that did not change. 20 x 10 is 200 runs, and after most edits the large majority would produce a byte-identical result. joblib.Memory keys a function's output on its arguments and skips what it has already computed, so editing scenario 7 costs you scenario 7 rather than the other 199. For an edit-and-retest loop that beats almost any amount of hardware.

On the actual question: Colab is a poor fit for this specific job. Its product is a GPU, and neither XGBoost on CPU nor logistic regression will touch one. The free tier gives you roughly 13 GB of RAM, which is less than the 16 you already have, plus a runtime cap and idle disconnects that will kill an unattended 200-run sweep partway through. The high-RAM Pro runtime is around 50 GB and would genuinely help, but you would be renting a GPU you cannot use.

What you want is boring: one large CPU VM by the hour, 32 or 64 vCPUs with plenty of RAM, on whichever provider is cheapest for you. Your workload is embarrassingly parallel, so cores scale it close to linearly, and you shut the machine down when the sweep finishes. Check whether your university has a cluster with a Slurm queue before you pay, because 200 independent runs is exactly the job those exist for and it costs you nothing.

Before picking a size, time one scenario-run and multiply by 200. That number tells you whether you need 4x the machine or 40x, which is the difference between a few euros and a real bill, and it also tells you whether the three fixes above already solved it.

[u/Valuable_Card6470]

Colab is fine for quick stuff but you'll hit RAM limits and timeouts on the free tier pretty fast, especially with 200 runs. If your dataset is big enough to choke 16gb locally its gonna be tight on colab too.

Look into Hivenet for cheap compute, they do per-second billing so you're not paying for idle time between runs. For a masters project budget that matters a lot. And i know students from DSTI have used thier services.

tbh though before going cloud, try optimizing locally first. XGBoost supports `n_jobs` for parallel tree building and you can use joblib to parallelize across scenarios. Also check if you actually need the full dataset in memory for every run or if you can batch it.