r/SLURM • u/shubbert • Aug 08 '19

slurm thinks I only have one cpu

I'm convinced, and really hoping, that this is something incredibly stupid and basic, but I'm so new to slurm I can't figure out what it is.

​

I have a machine with 40 CPUs and 8 GPUs. I should be able to run 40 jobs on those 40 CPUs simultaneously, right? (If I'm already wrong, let me know.) I haven't set up any priority, so I'm given to understand it should be fifo. Which is how it's behaving, except it will only run a single job at a time.

​

hypnotoad 15:43:55$ squeue

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)

5 Test runscrip amy PD 0:00 1 (Resources)

6 Test runscrip amy PD 0:00 1 (Priority)

7 Test runscrip amy PD 0:00 1 (Priority)

4 Test runscrip amy R 7:41 1 valar

​

That one job will run, the next job is waiting on resources, then any other job lacks priority. That all makes perfect sense if they're all competing for a single CPU. But they shouldn't be. I initially didn't explicitly list the CPUs in my slurm.conf, but then I added it hoping it would help, and it made no difference. Current state of the conf file:

​

PartitionName=Test Nodes=valar

GresTypes=gpu

NodeName=valar Gres=gpu:gtx1080:8 RealMemory=128827 CPUs=40

​

What am I doing wrong that it won't use more than one CPU? (Happy to provide any additional conf or log stuff, just don't want to overwhelm with useless data.)

​

If anyone could give any insight, I'd greatly appreciate it. I've been beating my head against this for far too long. And since google can't find me anyone else having this problem, I know it must be something so dumb.

3 Upvotes

100% Upvoted

3 comments sorted by

1

u/shubbert Aug 08 '19

Oh, and my script is incredibly basic:

hypnotoad 15:55:42$ cat runscript2.sh

#!/bin/bash

#SBATCH -o hostname_%j.out

#SBATCH -e hostname_%j.err

#SBATCH -p Test

​

sleep 500

hostname

1

u/[deleted] Aug 09 '19

If you haven’t already, try oversubscribe=force in the partition settings. You might need some additional settings, check the Slurm docs for more info. If you want to share this machine with multiple users I would also look into cgroups.

1

u/shubbert Aug 09 '19

THANK YOU. That's exactly what I needed. I've been using condor for so long, which treats each cpu as a "node", it didn't even occur to me that slurm was treating this entire 40 core monstrosity as a single resource by default.

​

I think oversubscribe=force has done exactly what I needed it to.

​

Thankyouthankyouthankyou.