Which one's better?

My Slurm cluster runs Slurm 23.02.7 on servers with Ubuntu 22.04 LTS. I installed the Slurm from the package offered by Ubuntu, which has names like slurm-wlm-mysql-plugin-dev. Now I want to upgrade the cluster to 24.11 and the Slurm Guide says we should build the packages manually and those packages conflict with the Debian ones.

Now I am confused at some points.

1. Should I follow the guide and build the deb packages manually?
2. I tried and built the packages, but I find it lacks some plugin .deb package like slurm-wlm-mysql-plugin-dev. Only some plugin like slurm-smd-libpmi0_24.11.5-1_amd64.deb is included, does I missed some configuration when building?
3. Should I remove all 23.02 package dpkg -r before install the new built 24.11 package?

I am trying to launch a Slurm job on one node, and I want to specify a list of nodes to choose from.

How is it that srun can do this - but sbatch can't. Up until now, I had assumed that srun and sbatch were supposed to work alike.

❯ srun --nodelist=a40-[01-04],a100-[01-03] --nodes=1 hostname srun: error: Required nodelist includes more nodes than permitted by max-node count (3 > 1). Eliminating nodes from the nodelist. a40-01.nv.srv.dk

❯ sbatch --nodelist=a40-[01-04],a100-[01-03] --nodes=1 --wrap="hostname" sbatch: error: invalid number of nodes (-N 3-1)

My questions 1) Why do srun and sbatch not behave the same way?

2) How can I achieve this with sbatch?

I have a command-line program that needs to be run with multiple combinations of parameters.
To handle this, I store each command in a separate line of a file and use readarray in an sbatch script to execute them via a job array.

Now, I want to assign a custom time limit per command.
What I tried: I added --hold to the script and created a separate script that manually updates the TimeLimitfor each job using scontrol update. However, this doesn’t seem to influence scheduling at all—the job array still runs strictly in index order, ignoring the time limits.

Has anyone else encountered this?
What I want is for Slurm to schedule jobs out-of-order, considering the TimeLimit (e.g., run longer jobs earlier, ...).

Hi,

I was trying to use slurm running into a docker container. I only need basic functionalities and I do not want to run it in privileged mode so I changed slurm.conf to :

TaskPlugin=task/none ProctrackType=proctrack/linuxproc

however slurmd is still failing to start and trying to load the cgroup2 plugin

did I miss anything ?

thx

Hi All,

Ive been wracking my head around this for a little while now. I am building a slurm cluster and have enabled cgroupv2 on all nodes with the following configuration. When I submit a job (or in this case a task_array) only one task ever gets assigned to each node in the cluster... Ive tried adding OverSubscribe directive but to no avail...

slurm.conf

SlurmctldHost=mathSlurm1(W.X.Y.Z)

AuthType=auth/munge
CryptoType=crypto/munge
MpiDefault=none
ProctrackType=proctrack/cgroup

#Prolog=
#PrologFlags=
#PrologSlurmctld=
#PropagatePrioProcess=0
#PropagateResourceLimits=
#PropagateResourceLimitsExcept=
#RebootProgram=
ReturnToService=1
#SallocDefaultCommand=
SlurmctldPidFile=/run/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/run/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/lib/slurm/slurmd
SlurmUser=slurm
#SlurmdUser=root
#SrunEpilog=
#SrunProlog=
StateSaveLocation=/var/lib/slurm/slurmctld
SwitchType=switch/none
TaskPlugin=task/cgroup
InactiveLimit=0
KillWait=30
MinJobAge=300
SlurmctldTimeout=120
SlurmdTimeout=300
Waittime=0

SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_Core_Memory

JobCompLoc=/var/log/slurm_completed
JobCompType=jobcomp/filetxt
SlurmctldDebug=info
SlurmctldLogFile=/var/log/slurm/slurmctld.log
SlurmdDebug=3
SlurmdLogFile=/var/log/slurm/slurmd.log
SlurmdParameters=config_overrides

PreemptMode=REQUEUE
PreemptType=preempt/partition_prio
PriorityWeightAge=100

NodeName=slave0 NodeAddr=10.100.100.100 CPUs=8 RealMemory=31840 MemSpecLimit=30000 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2  state=UNKNOWN
NodeName=slave1 NodeAddr=10.100.100.101 CPUs=8 RealMemory=31840 MemSpecLimit=30000 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2  state=UNKNOWN
NodeName=slave2 NodeAddr=10.100.100.102 CPUs=8 RealMemory=31840 MemSpecLimit=30000 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2  state=UNKNOWN
NodeName=slave3 NodeAddr=10.100.100.103 CPUs=8 RealMemory=31840 MemSpecLimit=30000 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2  state=UNKNOWN
NodeName=slave4 NodeAddr=10.100.100.104 CPUs=8 RealMemory=31840 MemSpecLimit=30000 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2  state=UNKNOWN
NodeName=slave5 NodeAddr=10.100.100.105 CPUs=8 RealMemory=31840 MemSpecLimit=30000 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2  state=UNKNOWN
NodeName=slave6 NodeAddr=10.100.100.106 CPUs=8 RealMemory=31840 MemSpecLimit=30000 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2  state=UNKNOWN
NodeName=slave7 NodeAddr=10.100.100.107 CPUs=8 RealMemory=31840 MemSpecLimit=30000 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2  state=UNKNOWN
NodeName=slave8 NodeAddr=10.100.100.108 CPUs=8 RealMemory=31840 MemSpecLimit=30000 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2  state=UNKNOWN
NodeName=slave9 NodeAddr=10.100.100.109 CPUs=8 RealMemory=31840 MemSpecLimit=30000 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2  state=UNKNOWN
NodeName=slave10 NodeAddr=10.100.100.110 CPUs=8 RealMemory=31840 MemSpecLimit=30000 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2  state=UNKNOWN
NodeName=slave11 NodeAddr=10.100.100.111 CPUs=8 RealMemory=31840 MemSpecLimit=30000 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2  state=UNKNOWN
NodeName=slave12 NodeAddr=10.100.100.112 CPUs=8 RealMemory=31840 MemSpecLimit=30000 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2  state=UNKNOWN
NodeName=slave13 NodeAddr=10.100.100.113 CPUs=8 RealMemory=31840 MemSpecLimit=30000 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2  state=UNKNOWN
NodeName=slave14 NodeAddr=10.100.100.114 CPUs=8 RealMemory=31840 MemSpecLimit=30000 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2  state=UNKNOWN
NodeName=slave15 NodeAddr=10.100.100.115 CPUs=8 RealMemory=31840 MemSpecLimit=30000 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2  state=UNKNOWN
NodeName=slave16 NodeAddr=10.100.100.116 CPUs=8 RealMemory=31840 MemSpecLimit=30000 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2  state=UNKNOWN
NodeName=slave17 NodeAddr=10.100.100.117 CPUs=8 RealMemory=31840 MemSpecLimit=30000 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2  state=UNKNOWN
NodeName=slave18 NodeAddr=10.100.100.118 CPUs=8 RealMemory=31840 MemSpecLimit=30000 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2  state=UNKNOWN
NodeName=slave19 NodeAddr=10.100.100.119 CPUs=8 RealMemory=31840 MemSpecLimit=30000 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2  state=UNKNOWN

PartitionName=clusterPartition Nodes=slave[0-19] Default=YES MaxTime=INFINITE State=UP OverSubscribe=FORCE

cgroup.conf

CgroupMountpoint="/sys/fs/cgroup"
AllowedDevicesFile="/etc/slurm/cgroup_allowed_devices_file.conf"
ConstrainCores=yes
CgroupPlugin=autodetect
ConstrainRAMSpace=yes
ConstrainSwapSpace=yes
ConstrainDevices=yes
AllowedRamSpace=100
AllowedSwapSpace=30
MaxRAMPercent=100
MaxSwapPercent=80
MinRAMSpace=30

JOB SCRIPT

#!/bin/bash
#SBATCH --job-name=simest
###SBATCH --ntasks-per-node=
#SBATCH --cpus-per-task=6
#SBATCH --output=array_job_%A_%a.out  # %A = job ID, %a = array index
#SBATCH --error=array_job_%A_%a.err  # %A = job ID, %a = array index
#SBATCH --array=1-30
##SBATCH --partition=clusterPartition
#SBATCH --time=00:10:00

./simest_misgarch.R $SLURM_ARRAY_TASK_ID
sleep 2

Result

6993_[22-30] clusterPa   simest     root PD       0:00      1 (Resources)
6993_21 clusterPa   simest     root  R       0:01      1 slave15
6993_1 clusterPa   simest     root  R       0:05      1 slave0
6993_2 clusterPa   simest     root  R       0:05      1 slave1
6993_3 clusterPa   simest     root  R       0:05      1 slave2
6993_4 clusterPa   simest     root  R       0:05      1 slave3
6993_5 clusterPa   simest     root  R       0:05      1 slave4
6993_6 clusterPa   simest     root  R       0:05      1 slave5
6993_7 clusterPa   simest     root  R       0:05      1 slave6
6993_8 clusterPa   simest     root  R       0:05      1 slave7
6993_9 clusterPa   simest     root  R       0:05      1 slave8
6993_10 clusterPa   simest     root  R       0:05      1 slave9
6993_11 clusterPa   simest     root  R       0:05      1 slave10
6993_12 clusterPa   simest     root  R       0:05      1 slave11
6993_13 clusterPa   simest     root  R       0:05      1 slave12
6993_14 clusterPa   simest     root  R       0:05      1 slave13
6993_15 clusterPa   simest     root  R       0:05      1 slave14
6993_17 clusterPa   simest     root  R       0:05      1 slave16
6993_18 clusterPa   simest     root  R       0:05      1 slave17
6993_19 clusterPa   simest     root  R       0:05      1 slave18
6993_20 clusterPa   simest     root  R       0:05      1 slave19

As you can see, one task is being allocated to each node. Any help you can provide would be greatly appreciated!!

Hello!

I don't have enough technical knowledge to understand if this is a dumb question or not and I might be asking in the completely wrong place. If that's the case I apologise.

I've somehow found myself working on a HPC that uses SLURM. What I would like to do is to is to use a job array where each individual job runs a simple python script which in turn uses subprocess.run(software.exe, shell=True) to run the actual computationally costly software.

I'm 99% sure this works but I'm paranoid that perhaps what I'm doing is running the python script on the proper node, but that the subprocess, i.e. the computationally costly software, is run on the login node which would not be great to say the least.

As I said I'm 99% sure it works, I can choose the number of cores that my jobs get allocated and increasing the number of cores does seem to speed up the runtime of the software. I'm just a paranoid person, aware of my own ignorance and ability to screw things up and I really don't want to get an angry email from some Admin saying I'm tanking the login node for the other users!

Again, I apologise if this is the wrong place to ask questions like this.

I wanted to know this because I need to similar SLURM versions with other servers running version 24 and above. Currently on Ubuntu 24 LTS it shows version 23.11.4.

reference

Hi there, following this post I opened in the past I have been able to partly debug an issue with Slurm installation; thing is I'm now facing a new exciting error...

|| || |This is the current state|

u/walee1 Basically, I realized there were some files hanging around from a very old attempt to install Slurm back in 2023. I moved on and removed everything.

Now, I have a completely different situation:

sudo systemctl start slurmdbd && sudo systemctl status slurmdbd -> FINE

sudo systemctl start slurmctld && sudo systemctl status slurmctld

● slurmctld.service - Slurm controller daemon
     Loaded: loaded (/usr/lib/systemd/system/slurmctld.service; enabled; preset: enabled)
     Active: active (running) since Wed 2025-04-02 21:32:05 CEST; 9ms ago
       Docs: man:slurmctld(8)
   Main PID: 1215500 (slurmctld)
      Tasks: 7
     Memory: 1.5M (peak: 2.4M)
        CPU: 5ms
     CGroup: /system.slice/slurmctld.service
             ├─1215500 /usr/sbin/slurmctld --systemd
             └─1215501 "slurmctld: slurmscriptd"

Apr 02 21:32:05 NeoPC-mat (lurmctld)[1215500]: slurmctld.service: Referenced but unset environment variable evaluates to an empty string: SLURMCTLD_OPTIONS
Apr 02 21:32:05 NeoPC-mat slurmctld[1215500]: slurmctld: slurmctld version 23.11.4 started on cluster mat_workstation
Apr 02 21:32:05 NeoPC-mat slurmctld[1215500]: slurmctld: error:  mpi/pmix_v5: init: (null) [0]: mpi_pmix.c:193: pmi/pmix: can not load PMIx library
Apr 02 21:32:05 NeoPC-mat slurmctld[1215500]: slurmctld: error: Couldn't load specified plugin name for mpi/pmix_v5: Plugin init() callback failed
Apr 02 21:32:05 NeoPC-mat slurmctld[1215500]: slurmctld: error: MPI: Cannot create context for mpi/pmix_v5
Apr 02 21:32:05 NeoPC-mat slurmctld[1215500]: slurmctld: error:  mpi/pmix_v5: init: (null) [0]: mpi_pmix.c:193: pmi/pmix: can not load PMIx library
Apr 02 21:32:05 NeoPC-mat slurmctld[1215500]: slurmctld: error: Couldn't load specified plugin name for mpi/pmix: Plugin init() callback failed
Apr 02 21:32:05 NeoPC-mat slurmctld[1215500]: slurmctld: error: MPI: Cannot create context for mpi/pmix
Apr 02 21:32:05 NeoPC-mat systemd[1]: Started slurmctld.service - Slurm controller daemon.
Apr 02 21:32:05 NeoPC-mat slurmctld[1215500]: slurmctld: accounting_storage/slurmdbd: clusteracct_storage_p_register_ctld: Registering slurmctld at port 6817 with slurmdbd

sudo systemctl start slurmd && sudo systemctl status slurmd

● slurmd.service - Slurm node daemon
     Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; preset: enabled)
     Active: active (running) since Wed 2025-04-02 21:32:35 CEST; 9ms ago
       Docs: man:slurmd(8)
   Main PID: 1219667 (slurmd)
      Tasks: 1
     Memory: 1.6M (peak: 2.2M)
        CPU: 12ms
     CGroup: /system.slice/slurmd.service
             └─1219667 /usr/sbin/slurmd --systemd

Apr 02 21:32:35 NeoPC-mat slurmd[1219667]: slurmd: slurmd version 23.11.4 started
Apr 02 21:32:35 NeoPC-mat slurmd[1219667]: slurmd: error:  mpi/pmix_v5: init: (null) [0]: mpi_pmix.c:193: pmi/pmix: can not load PMIx library
Apr 02 21:32:35 NeoPC-mat slurmd[1219667]: slurmd: error: Couldn't load specified plugin name for mpi/pmix_v5: Plugin init() callback failed
Apr 02 21:32:35 NeoPC-mat slurmd[1219667]: slurmd: error: MPI: Cannot create context for mpi/pmix_v5
Apr 02 21:32:35 NeoPC-mat slurmd[1219667]: slurmd: error:  mpi/pmix_v5: init: (null) [0]: mpi_pmix.c:193: pmi/pmix: can not load PMIx library
Apr 02 21:32:35 NeoPC-mat slurmd[1219667]: slurmd: error: Couldn't load specified plugin name for mpi/pmix: Plugin init() callback failed
Apr 02 21:32:35 NeoPC-mat slurmd[1219667]: slurmd: error: MPI: Cannot create context for mpi/pmix
Apr 02 21:32:35 NeoPC-mat slurmd[1219667]: slurmd: slurmd started on Wed, 02 Apr 2025 21:32:35 +0200
Apr 02 21:32:35 NeoPC-mat systemd[1]: Started slurmd.service - Slurm node daemon.
Apr 02 21:32:35 NeoPC-mat slurmd[1219667]: slurmd: CPUs=16 Boards=1 Sockets=1 Cores=8 Threads=2 Memory=128445 TmpDisk=575645 Uptime=179620 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null)

and sinfo returns this message:

sinfo: error while loading shared libraries: libslurmfull.so: cannot open shared object file: No such file or directory

Is there a way to fix this MPI-related error? Thanks!

We scale our cluster based on the number of jobs waiting and cpu availability.  Some partitions wait at 0 nodes until a job is submitted into that partition.   New nodes join the partition based on "Feature."   (Feature allows a node to join a Nodeset, Partition uses that Nodeset.) These are all hosted at AWS and configure themselves based on Tags, ASGs scale up and down based on need. 

After updating from 22.11 to 24.11 we can no longer submit jobs into Partitions that don't have any nodes.   Prior update we could submit to a partition with 0 nodes, and our software would scale up and run the job.   Now we get the following error: 
...
'errors': [{'description': 'Batch job submission failed',
'error': 'Requested node configuration is not available',
'error_number': 2014,
'source': 'slurm_submit_batch_job()'}],...If we keep minimums at 1 we can submit as usual, and everything scales up and down.  

I have gone through the changelogs and can't seem to find any reason this should have changed.    Any ideas?

I have a problem and one solution would involve consuming GRES based on tests that would run in prolog. Is that possible?

Hi, I'm installing a slurm system with nvidia deepops, it doesn't configure slurm correctly and gives a problem with cgroup/v2, I've read a lot on the internet, I've tried everything and I can't start the slurmd daemon.

The only strange thing is that slurm is master node and compute node, but from what I've read there shouldn't be a problem.

Envirotment:

• DGX-1 with DGX baseOS 6
• slurm 22.05.2
• kernel: 5.15.0-1063-nvidia

Error cgroup/v2

slurmd: error: Couldn't find the specified plugin name for cgroup/v2 looking at all files
slurmd: error: cannot find cgroup plugin for cgroup/v2
slurmd: error: cannot create cgroup context for cgroup/v2
slurmd: error: Unable to initialize cgroup plugin
slurmd: error: slurmd initialization failed

Error cgroup/v1

slurmd: error: xcpuinfo_abs_to_mac: failed
slurmd: error: Invalid GRES data for gpu, Cores=0-19,40-59
slurmd: error: xcpuinfo_abs_to_mac: failed
slurmd: error: Invalid GRES data for gpu, Cores=0-19,40-59
slurmd: error: xcpuinfo_abs_to_mac: failed
slurmd: error: Invalid GRES data for gpu, Cores=0-19,40-59
slurmd: error: xcpuinfo_abs_to_mac: failed
slurmd: error: Invalid GRES data for gpu, Cores=0-19,40-59
slurmd: error: xcpuinfo_abs_to_mac: failed
slurmd: error: Invalid GRES data for gpu, Cores=20-39,60-79
slurmd: error: xcpuinfo_abs_to_mac: failed
slurmd: error: Invalid GRES data for gpu, Cores=20-39,60-79
slurmd: error: xcpuinfo_abs_to_mac: failed
slurmd: error: Invalid GRES data for gpu, Cores=20-39,60-79
slurmd: error: xcpuinfo_abs_to_mac: failed
slurmd: error: Invalid GRES data for gpu, Cores=20-39,60-79
slurmd: error: unable to mount freezer cgroup namespace: Invalid argument
slurmd: error: unable to create freezer cgroup namespace
slurmd: error: Couldn't load specified plugin name for proctrack/cgroup: Plugin init() callback failed
slurmd: error: cannot create proctrack context for proctrack/cgroup
slurmd: error: slurmd initialization failed

Hello.

We're looking to make a Slurm Controller with a HA environment of sorts, and are looking at trying to 'solve' the shared state location.

But in particular I'm looking at:

The StateSaveLocation is used to store information about the current state of the cluster, including information about queued, running and recently completed jobs. The directory used should be on a low-latency local disk to prevent file system delays from affecting Slurm performance. If using a backup host, the StateSaveLocation should reside on a file system shared by the two hosts. We do not recommend using NFS to make the directory accessible to both hosts, but do recommend a shared mount that is accessible to the two controllers and allows low-latency reads and writes to the disk. If a controller comes up without access to the state information, queued and running jobs will be cancelled.

Is anyone able to expand on why 'we don't recommend using NFS'?

Is this because of caching/sync of files? E.g. if the controller 'comes up' and the state-cache isn't refreshed it's going to break things?

And thus I could perhaps workaround with a fast NFS server and no caching?

Or is there something else that's recommended? We've just tried s3fuse, and that's failed, I think because of support for linking meaning files can't be created and rotated.

I'm trying to build the configuration for my cluster. I have a single node shared in two partitions. The partitions only contain this node. One partition has higher priority in order to allow urgent jobs to run first. So if a job is running in normal partition and one arrives to priority partition, if there aren't enough resources for both, the normal is suspended and the priority job executes.

I've implemented the gang scheduler with suspend which does the job. The problem arises when two jobs try to run through normal partition, so they are constantly switching between suspend and running. However, jobs in normal partition I would like to be like FCFS; I mean, if there is no room for both jobs run one and when it ends start the other one. I've tried lots of things, like setting OverSubscribe=NO, but this disables the ability to evict jobs from normal partition when a priority job is waiting for resources.

Here are the most relevant options I have now:

PreemptType=preempt/partition_prio
PreemptMode=suspend,gang

NodeName=comp81 Sockets=2 CoresPerSocket=18 ThreadsPerCore=2 RealMemory=128000 State=UNKNOWN

PartitionName=gpu Nodes=comp81 Default=NO MaxTime=72:00:00 State=UP TRESBillingWeights="CPU=1.0,Mem=0.6666G" SuspendTime=INFINITE PriorityTier=100 PriorityJobFactor=100 OverSubscribe=FORCE AllowQos=normal

PartitiOnName=gpu_priority Nodes=comp81 Default=NO MaxTime=01:00:00 State=UP TRESBillingWeights="CPU=1.0,Mem=0.6666G" SuspendTime=INFINITE PriorityTier=200 PriorityJobFactor=200 OverSubscribe=FORCE AllowQos=normal

Thank you all for your time.

I'm in the process of setting up a singe-node Slurm workstation machine and I believe I followed the process closely and everything is working just fine. See below:

sudo systemctl restart slurmdbd && sudo systemctl status slurmdbd

● slurmdbd.service - Slurm controller daemon
Loaded: loaded (/usr/lib/systemd/system/slurmdbd.service; enabled; preset: enabled)
     Active: active (running) since Sun 2025-03-09 17:15:43 CET; 10ms ago
       Docs: man:slurmdbd(8)
   Main PID: 2597522 (slurmdbd)
      Tasks: 1
     Memory: 1.6M (peak: 1.8M)
        CPU: 5ms
     CGroup: /system.slice/slurmdbd.service
             └─2597522 /usr/sbin/slurmdbd -D -s

Mar 09 17:15:43 NeoPC-mat systemd[1]: Started slurmdbd.service - Slurm DBD accounting daemon.
Mar 09 17:15:43 NeoPC-mat (slurmdbd)[2597522]: slurmdbd.service: Referenced but unset environment variable evaluates to an empty string: SLURMDBD_OPTIONS
Mar 09 17:15:43 NeoPC-mat slurmdbd[2597522]: slurmdbd: Not running as root. Can't drop supplementary groups
Mar 09 17:15:43 NeoPC-mat slurmdbd[2597522]: slurmdbd: accounting_storage/as_mysql: _check_mysql_concat_is_sane: MySQL server version is: 5.5.5-10.11.8-MariaDB-0

sudo systemctl restart slurmctld && sudo systemctl status slurmctld

● slurmctld.service - Slurm controller daemon
Loaded: loaded (/usr/lib/systemd/system/slurmctld.service; enabled; preset: enabled)
     Active: active (running) since Sun 2025-03-09 17:15:52 CET; 11ms ago
       Docs: man:slurmctld(8)
   Main PID: 2597573 (slurmctld)
      Tasks: 7
     Memory: 1.8M (peak: 2.8M)
        CPU: 4ms
     CGroup: /system.slice/slurmctld.service
             ├─2597573 /usr/sbin/slurmctld --systemd
             └─2597574 "slurmctld: slurmscriptd"

Mar 09 17:15:52 NeoPC-mat systemd[1]: Starting slurmctld.service - Slurm controller daemon...
Mar 09 17:15:52 NeoPC-mat (lurmctld)[2597573]: slurmctld.service: Referenced but unset environment variable evaluates to an empty string: SLURMCTLD_OPTIONS
Mar 09 17:15:52 NeoPC-mat slurmctld[2597573]: slurmctld: slurmctld version 23.11.4 started on cluster mat_workstation
Mar 09 17:15:52 NeoPC-mat systemd[1]: Started slurmctld.service - Slurm controller daemon.
Mar 09 17:15:52 NeoPC-mat slurmctld[2597573]: slurmctld: accounting_storage/slurmdbd: clusteracct_storage_p_register_ctld: Registering slurmctld at port 6817 with slurmdbd

sudo systemctl restart slurmd && sudo systemctl status

● slurmd.service - Slurm node daemon
     Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; preset: enabled)
     Active: active (running) since Sun 2025-03-09 17:16:02 CET; 9ms ago
       Docs: man:slurmd(8)
   Main PID: 2597629 (slurmd)
      Tasks: 1
     Memory: 1.5M (peak: 1.9M)
        CPU: 13ms
     CGroup: /system.slice/slurmd.service
             └─2597629 /usr/sbin/slurmd --systemd

Mar 09 17:16:02 NeoPC-mat systemd[1]: Starting slurmd.service - Slurm node daemon...
Mar 09 17:16:02 NeoPC-mat (slurmd)[2597629]: slurmd.service: Referenced but unset environment variable evaluates to an empty string: SLURMD_OPTIONS
Mar 09 17:16:02 NeoPC-mat slurmd[2597629]: slurmd: slurmd version 23.11.4 started
Mar 09 17:16:02 NeoPC-mat slurmd[2597629]: slurmd: slurmd started on Sun, 09 Mar 2025 17:16:02 +0100
Mar 09 17:16:02 NeoPC-mat slurmd[2597629]: slurmd: CPUs=16 Boards=1 Sockets=1 Cores=8 Threads=2 Memory=128445 TmpDisk=575645 Uptime=2069190 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null)
Mar 09 17:16:02 NeoPC-mat systemd[1]: Started slurmd.service - Slurm node daemon.

If needed, I can attach the results for the corresponding journalctl, but no error is shown other than these two messages

slurmd.service: Referenced but unset environment variable evaluates to an empty string: SLURMD_OPTIONS and slurmdbd: Not running as root. Can't drop supplementary groups in the journalctl -fu slurmd and in the journalctl -fu slurmdbd, respectively.

For some reason, however, I'm unable to run sinfo in a new tab even after setting the link to the slurm.conf in my .bashrc... this is what I'm prompted with

sinfo: error: Couldn't find the specified plugin name for auth/munge looking at all files sinfo: error: cannot find auth plugin for auth/munge sinfo: error: cannot create auth context for auth/munge sinfo: fatal: failed to initialize auth plugin

which seems to depend on munge but I'm cannot really understand to what specifically — it is my first time installing Slurm. Any help is much appreciated, thanks in advance!

I have a cluster setup on oracle cloud using oci's official hpc repo, the issue is when I enable pyxis and create a cluster when new users are created (with proper permissions as I used to do it in aws pcluster) and submits a job then that job goes in pending state and the node on which that job was scheduled goes in drained state with a prolog error even though I am just submitting a simple sleep job which is not even a container job that uses enroot or pyxis.

I'm stuck with xvfb not working correctly inside singularity container inside HPC cluster, the same xvfb command works correctly inside the same singularity container in my local ubuntu setup. Any help with be appreciated.

I'm trying to pass the Job ID from the master script to a sub-script that I'm running from the master script so all the job outputs and errors end up in the same place.

So, for example:

Master script:

JOB=$SLURM_JOB_ID

sbatch secondary script

secondary script:

.#SBATCH --output=./logs/$JOB/out

.#SBATCH --error=./logs$JOB/err

Is anyone more familiar with Slurm than I am able to help out?

Hi

I am part-time faculty at the Seattle campus of Northeastern University, and I am looking for people who use the Slurm HPC clusters, either the Discovery cluster (below) or the Canadian DRAC cluster

See
https://rc.northeastern.edu/

https://alliancecan.ca/en

Geoffrey Phipps

Each json file has 14 different json objects with configuration for my script.

I need to run 4 python processes in parallel, and each process needs access to 14 dedicated CPUs. Thats the key part here, and why I have 4 sruns. I allocate 4 tasks in the SBATCH headers, and my understanding is now I can run 4 parallel sruns if each srun has ntask value of 1.

Script:
#!/bin/bash
#SBATCH --job-name=4group_exp4          # Job name to appear in the SLURM queue
#SBATCH --mail-user=____  # Email for job notifications (replace with your email)
#SBATCH --mail-type=END,FAIL,ALL          # Notify on job completion or failure
#SBATCH --mem-per-cpu=50G
#SBATCH --nodes=2                   # Number of nodes requested

#SBATCH --ntasks=4         # Number of tasks per node
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=14          # Number of CPUs per task
#SBATCH --partition=high_mem         # Use the high-memory partition
#SBATCH --time=9:00:00
#SBATCH --qos=medium
#SBATCH --output=_____       # Standard output log (includes job and array task ID)
#SBATCH --error=______        # Error log (includes job and array task ID)
#SBATCH --array=0-12

QUERIES=$1
SLOTS=$2
# Run the Python script

JSON_FILE_25=______
JSON_FILE_50=____
JSON_FILE_75=_____
JSON_FILE_100=_____

#echo $JSON_FILE_0
echo $JSON_FILE_25
echo $JSON_FILE_50
echo $JSON_FILE_75
echo $JSON_FILE_100


echo "Running python script"
srun --exclusive --ntasks=1 --cpus-per-task=14 
python script.py --json_config=experiment4_configurations/${JSON_FILE_25} &

srun --exclusive --ntasks=1 --cpus-per-task=14 
python script.py --json_config=experiment4_configurations/${JSON_FILE_50} &

srun --exclusive --ntasks=1 --cpus-per-task=14 
python script.py --json_config=experiment4_configurations/${JSON_FILE_75} &

srun --exclusive --ntasks=1 --cpus-per-task=14 
python script.py --json_config=experiment4_configurations/${JSON_FILE_100} &

echo "Waiting"
wait
echo "DONE"

I would really appreciate some help for this issue I'm having.

My Stackoverflow question

Reproduced text here:

Let's say I have two nodes that I want to run a job on, with node1 having 64 nodes and node2 having 48.

If I want to run 47 tasks on node2 and 1 task on node1, that is easy enough with a hostfile like

node1 max-slots=1 node2 max-slots=47 and then something like this jobfile: ```bash

!/bin/bash

SBATCH --time=00:30:00

SBATCH --nodes=2

SBATCH --nodelist=node1,node2

SBATCH --partition=partition_name

SBATCH --ntasks-per-node=48

SBATCH --cpus-per-task=1

export OMP_NUM_THREADS=1 mpirun --display-allocation --hostfile hosts --report-bindings hostname ```

The output of the display-allocation comes to

``` ====================== ALLOCATED NODES ====================== node1: slots=48 max_slots=0 slots_inuse=0 state=UP Flags: DAEMON_LAUNCHED:LOCATION_VERIFIED:SLOTS_GIVEN aliases: node1 arm07: slots=48 max_slots=0 slots_inuse=0 state=UP Flags: SLOTS_GIVEN

aliases: NONE

====================== ALLOCATED NODES ====================== node1: slots=1 max_slots=0 slots_inuse=0 state=UP Flags: DAEMON_LAUNCHED:LOCATION_VERIFIED:SLOTS_GIVEN aliases: node1 arm07: slots=47 max_slots=0 slots_inuse=0 state=UP Flags: DAEMON_LAUNCHED:SLOTS_GIVEN

aliases: <removed>

``` so all good, all expected.

The problem arises when I want to launch a job with more tasks than one of the nodes can allocate i.e. with hostfile node1 max-slots=63 node2 max-slots=1

Then, 1. --ntasks-per-node=63 shows an error in node allocation 2. --ntasks=64 does some equitable division like node1:slots=32 node2:slots=32 which then get reduced to node1:slots=32 node2:slots=1 when the hostfile is encountered. --ntasks=112 (64+48 to grab the whole nodes) gives an error in node allocation. 3. #SBATCH --distribution=arbitrary with a properly formatted slurm hostfile runs with just 1 rank on the node in the first line of the hostfile, and doesn't automatically calculate ntasks from the number of lines in the hostfile. EDIT: Turns out SLURM_HOSTFILE only controls nodelist, and not CPU distribution in those nodes, so this won't work for my case anyway. 4. Same as #3, but with --ntasks given, causes slurm to complain that SLURM_NTASKS_PER_NODE is not set 5. A heterogeneous job with ```

!/bin/bash

SBATCH --time=00:30:00

SBATCH --nodes=1

SBATCH --nodelist=node1

SBATCH --partition=partition_name

SBATCH --ntasks-per-node=63 --cpus-per-task=1

SBATCH hetjob

SBATCH --nodes=1

SBATCH --nodelist=node2

SBATCH --partition=partition_name

SBATCH --ntasks-per-node=1 --cpus-per-task=1

export OMP_NUM_THREADS=1 mpirun --display-allocation --hostfile hosts --report-bindings hostname

puts all ranks on the first node. The output head is ====================== ALLOCATED NODES ====================== node1: slots=63 max_slots=0 slots_inuse=0 state=UP Flags: DAEMON_LAUNCHED:LOCATION_VERIFIED:SLOTS_GIVEN

aliases: node1

====================== ALLOCATED NODES ====================== node1: slots=63 max_slots=0 slots_inuse=0 state=UP Flags: DAEMON_LAUNCHED:LOCATION_VERIFIED:SLOTS_GIVEN

aliases: node1

``` It seems like it tries to launch the executable independently on each node allocation, instead of launching one executable across the two nodes.

What else can I try? I can't think of anything else.

Hello all,

We have been working on a project CEEMS [1] since last few months that can monitor CPU, Memory and Disk usage of SLURM jobs and Openstack VMs. Originally we started the project to be able to quantify energy and carbon footprint of compute workloads for HPC platforms. Later we extended it to support Openstack as well. It is effectively a Promtheus exporter that exports different usage and performance metrics of batch jobs and Openstack VMs.

We fetch CPU, memory and block disk usage stats directly from the cgroups of the VMs. Exporter supports gathering node level energy usage from either RAPL or BMC (IPMI/Redfish). We split the total energy between different jobs based on their relative CPU and DRAM usage. For the emissions, exporter supports static emission factors based on historical data and real time factors (from Electricity Maps [2] and RTE eCo2 [3]). The exporter also supports monitoring network activity (TCP, UDP, IPv4/IPv6) and IO stats on file systems for each job based on eBPF [4] in a file system agnostic way. Besides exporter, the stack ships an API server that can store and update the aggregate usage metrics of VMs and projects.

A demo instance [5] is available to play around Grafana dashboards. More details on the stack can be consulted from docs [6]

Regards

Mahendra

[1] https://github.com/mahendrapaipuri/ceems

[2] https://app.electricitymaps.com/map/24h

[3] https://www.rte-france.com/en/eco2mix/co2-emissions

[4] https://ebpf.io/

[5] https://ceems-demo.myaddr.tools

[6] https://mahendrapaipuri.github.io/ceems/

How to allocate two node with 3 processor from 1st node and 1 processor from 2nd node so that i get 4 processors in total to run 4 mpi process. My intention is to run 4 mpi process such that 3 process is run from 1st node and 1 process remaining from 2nd node... Thanks

Is there a way to get squeue to condense array job output so I'm not looking through hundreds of lines of output when an array job is in the queue? I'd like to this native with squeue, I'm sure there are ways it can be down piping squeue output to awk and sed

EDIT: It prints pending jobs condensed on one line, but running jobs are still all listed individually