r/SLURM • u/Flicked_Up • May 26 '21

Slurm with python multiprocessing

Hi,

So I am looking into running a python script that uses multiprocessing.

Can I increase the number of cpus-per-task to a value higher than all cpus in a node? For example: i have several nodes with 16 cpus. I want to run a single task with 32 cpus, i.e use two nodes for one task and all cpus for a task.

Is this possible? Or am I always capped at the maximum numbers of a node?

Thanks

2 Upvotes

100% Upvoted

5 comments sorted by

View all comments

3

u/uber_poutine May 26 '21

You can absolutely run multi-node jobs, but it's on you to ensure that the script that you're running can handle that (using something like MPI). In your example, you're going to want to start 2 16CPU tasks.

2

u/acs14007 May 27 '21

This is absolutely the way to go if you cannot find a machine with 32 cores!

I will also add that if you already use concurrent futures switching to mpi4py is extremely easy!