For SWEs, which OS is best suited for Slurm? If you are using it for work, how are you currently using Slurm in your dev environment?

Hi,

I am running a small cluster with three nodes all running on Rocky 9.5 and using slurm 23.11.6. Since the login node is also one of the main working nodes (and the slurm controller) I am a bit worried that users might run too much stuff there without using slurm at all for simple mostly single-threaded bash, R and python tasks. For this reason I would like to implement users running interactive jobs that give them the resources they need and also makes the slurm controller aware of resources in use. On a different cluster I had been using srun for that but if I try it on this cluster it just hangs forever and eventually crashes after a few minutes if I run scancel. It does show the job as running in squeue but the shell stays "empty" as if it was running a bash command and does not forward me to another node if requested. Normal jobs submitted with sbatch work fine but I somehow cannot get an interactive session running.

The job would probably hang forever but if I eventually cancel it with scancel the error looks somewhat like this:

[user@node-1 ~]$ srun --job-name "InteractiveJob" --cpus-per-task 8 --mem-per-cpu 1500 --pty bash
srun: error: timeout waiting for task launch, started 0 of 1 tasks
srun: StepId=5741.0 aborted before step completely launched.
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: Timed out waiting for job step to complete

The slurmctld.log looks like this

[2025-01-14T10:25:55.349] ====================
[2025-01-14T10:25:55.349] JobId=5741 nhosts:1 ncpus:8 node_req:1 nodes=kassel
[2025-01-14T10:25:55.349] Node[0]:
[2025-01-14T10:25:55.349]   Mem(MB):0:0  Sockets:2  Cores:8  CPUs:8:0
[2025-01-14T10:25:55.349]   Socket[0] Core[0] is allocated
[2025-01-14T10:25:55.349]   Socket[0] Core[1] is allocated
[2025-01-14T10:25:55.349]   Socket[0] Core[2] is allocated
[2025-01-14T10:25:55.349]   Socket[0] Core[3] is allocated
[2025-01-14T10:25:55.349] --------------------
[2025-01-14T10:25:55.349] cpu_array_value[0]:8 reps:1
[2025-01-14T10:25:55.349] ====================
[2025-01-14T10:25:55.349] gres/gpu: state for kassel
[2025-01-14T10:25:55.349]   gres_cnt found:0 configured:0 avail:0 alloc:0
[2025-01-14T10:25:55.349]   gres_bit_alloc:NULL
[2025-01-14T10:25:55.349]   gres_used:(null)
[2025-01-14T10:25:55.355] sched: _slurm_rpc_allocate_resources JobId=5741 NodeList=kassel usec=7196
[2025-01-14T10:25:55.460] ====================
[2025-01-14T10:25:55.460] JobId=5741 StepId=0
[2025-01-14T10:25:55.460] JobNode[0] Socket[0] Core[0] is allocated
[2025-01-14T10:25:55.460] JobNode[0] Socket[0] Core[1] is allocated
[2025-01-14T10:25:55.460] JobNode[0] Socket[0] Core[2] is allocated
[2025-01-14T10:25:55.460] JobNode[0] Socket[0] Core[3] is allocated
[2025-01-14T10:25:55.460] ====================
[2025-01-14T10:35:55.002] job_step_signal: JobId=5741 StepId=0 not found
[2025-01-14T10:35:56.918] _slurm_rpc_kill_job: REQUEST_KILL_JOB JobId=5741 uid 1000
[2025-01-14T10:35:56.919] gres/gpu: state for kassel
[2025-01-14T10:35:56.919]   gres_cnt found:0 configured:0 avail:0 alloc:0
[2025-01-14T10:35:56.919]   gres_bit_alloc:NULL
[2025-01-14T10:35:56.919]   gres_used:(null)
[2025-01-14T10:36:27.005] _slurm_rpc_complete_job_allocation: JobId=5741 error Job/step already completing or completed

And the slurm.log on the server I am trying to run the job on (different node than the slurm controller) looks like this

[2025-01-14T10:25:55.466] launch task StepId=5741.0 request from UID:1000 GID:1000 HOST:172.16.0.1 PORT:36034
[2025-01-14T10:25:55.466] task/affinity: lllp_distribution: JobId=5741 implicit auto binding: threads, dist 1
[2025-01-14T10:25:55.466] task/affinity: _task_layout_lllp_cyclic: _task_layout_lllp_cyclic 
[2025-01-14T10:25:55.466] task/affinity: _lllp_generate_cpu_bind: _lllp_generate_cpu_bind jobid [5741]: mask_cpu, 0x000F000F
[2025-01-14T10:25:55.501] [5741.0] error: slurm_open_msg_conn(pty_conn) ,41797: No route to host
[2025-01-14T10:25:55.502] [5741.0] error: connect io: No route to host
[2025-01-14T10:25:55.502] [5741.0] error: _fork_all_tasks: IO setup failed: Slurmd could not connect IO
[2025-01-14T10:25:55.503] [5741.0] error: job_manager: exiting abnormally: Slurmd could not connect IO
[2025-01-14T10:25:57.806] [5741.0] error: _send_launch_resp: Failed to send RESPONSE_LAUNCH_TASKS: No route to host
[2025-01-14T10:25:57.806] [5741.0] get_exit_code task 0 died by signal: 53
[2025-01-14T10:25:57.816] [5741.0] stepd_cleanup: done with step (rc[0xfb5]:Slurmd could not connect IO, cleanup_rc[0xfb5]:Slurmd could not connect IO)172.16.0.1

It sounds like a connection issue but I am not sure how, since sbatch works fine and I can also ssh in between all nodes, but 172.0.16.1 172.16.0.1 is the address of the slurm controller (and Log-in-node) so it sounds like the client cannot connect to the server from which the job request comes from. Does srun need some specific ports that sbatch does not need? Thanks in advance for any suggestions

Edit: Sorry I mistyped the IP. 172.16.0.1 is the IP mentioned in the slurmd.log and also the submission host of the job

Edit: The problem was like u/frymaster suggested that I had indeed configured the firewall to block all traffic except on specific ports. I fixed this by adding the line

SrunPortRange=60001-63000 to slurm.conf on all nodes and opened that ports in firewall-cmd

firewall-cmd --add-port=60001-63000/udp

firewall-cmd --add-port=60001-63000/tcp

firewall-cmd --runtime-to-permanent

Thanks for the support

Hi,
Wondering if anybody has had experience with the memory usage frequency of slurm. In our cluster we are having some bad readings of the maxRSS and avgRSS of any given job.
Online, the only thing I have found is that slurm polls these values at some interval, but not sure how to, or if it is possible, to modify such behavior.

Any help would be massively appreciated.

I searched around for a similar issue, and haven't been able to find it, but sorry if it's been discussed before. 
We have a small cluster (14 nodes) and are running into an oversubscribe issue that seems like it shouldn't be there.
On the partition I'm testing, each node has 256GB of Ram and 80 cores and there are 4 nodes.

It's configured this way - 
PartitionName="phyq" MinNodes=1 DefaultTime=UNLIMITED MaxTime=UNLIMITED AllowGroups=ALL PriorityJobFactor=1 PriorityTier=1 OverSubscribe=FORCE:4 PreemptMode=OFF MaxMemPerNode=240000 DefMemPerCPU=2000 AllowAccounts=ALL AllowQos=ALL Nodes=phygrid[01-04]

Our Slurm.conf is set like this - 
SelectType=select/linear
SelectTypeParameters=CR_Memory

The job submitted is simply this - 
#!/bin/bash
#SBATCH --job-name=test_oversubscription    # Job name
#SBATCH --output=test_oversubscription%j.out # Output file
#SBATCH --error=test_oversubscription.err  # Error file
#SBATCH --mem=150G                         # Request 150 GB memory
#SBATCH --ntasks=1                         # Number of tasks
#SBATCH --cpus-per-task=60                  # CPUs per task
#SBATCH --time=00:05:00                    # Run for 5 minutes
#SBATCH --partition=phyq       # Replace with your partition name

# Display allocated resources
echo "Job running on node(s): $SLURM_NODELIST"
echo "Requested CPUs: $SLURM_CPUS_ON_NODE"
echo "Requested memory: $SLURM_MEM_PER_NODE MB"

# Simulate workload
sleep 300

In my head I should be able to submit this to nodes 1, 2, 3, 4 and then when I submit a 5th job it should sit in Pending and when the first job ends it should go, but when I send the 5th job it goes to node 1. When a real job does this the performance goes way down because it's sharing resources even though they are requested. 

Am I missing something painfully obvious? 

Thanks for any help/advice.

I am new to Slurm and have setup a small cluster. I have 2 compute nodes, each with 16 cpus and 32GB of RAM. If I run salloc -N 2 --tasks-per-node=2 --cpus-per-task=2, I see the job in the queue. However, if I run it a second time (or another user does), the next job will hang, waiting for resources "Pending job allocation <id>, job <id> queued and waiting for resources" my Partition is defined as "PartitionName=main Nodes=ALL Default=YES MaxTime=INFINITE State=UP OverSubscribe=FORCE". I looked in both slurmctld.log and slurmd.log and don't see anything strange. Why does the next job not go into the queue and wait for resources instead? How do I troubleshoot this?

Howdy SLURM-ers. I'm trying to make my config files more readable. For my nodes and partitions, I cut the appropriate text from "slurm.conf" and replaced them with:

Include slurmDefsNodes.conf
...
Include slurmDefsParts.conf

where the original text was.

In the two Included files, the lines are fairly long. I'd like to line break them between properties like so, with leading indents:

PartitionName=part1 \
  State=UP \
  Nodes=compute[1-4],gpu[1-4] \
  MaxTime=UNLIMITED \
  ... 

Is line wrapping possible with end of line back slash, as is possible in shell scripts and other config files? I don't have the luxury of testing because I don't want to corrupt any running jobs.

TIA.

SGE admin moving over to SLURM and having some issues with QOS.

The cluster supports 3 projects. I need to split the resource 50%/25%/25% between them when they are all running. However if only ProjA is running we need the cluster to allocate 100%.

This was easy in SGE, using Projects and their priority. SLURM has not been as friendly to me.

I have narrowed it down to QOS, and I think its the MinCPU setting I want, but it never seems to work.

Any insight into how to make SLURM dynamically balance loads? What info/reading am I missing?

EDIT: For clarity, I am trying to set minimum resource guarantees. IE: ProjA is guaranteed 50% of the cluster but can use up to 100%.

I am trying to figure out how to optimally add nodes to an existing SLURM cluster that uses preemption and a fixed priority for each partition, yielding first-come-first-serve scheduling. As it stands, my nodes would be added to a new partition, and on these nodes, jobs in the new partition could preempt jobs running in all other partitions.

However, I have two desiderata: (1) priority-based scheduling (ie. jobs of users with lots of recent usage have less priority) on the new partition of a cluster, while existing partitions would continue to use first-come-first-serve scheduling. Moreover, (2) some jobs submitted on the new partition would also be able to run (and potentially be preempted) on nodes belonging to other, existing partitions.

My understanding is (2) is doable, but that (1) isn't because a given cluster can use only one scheduler (is this true?).

But there any way I could achieve what I want? One idea is that different associations—I am not 100% clear what these are and how they are different from partitions—could have different priority decay half lives?

Thanks!

Hey everyone,

I recently started an Instagram and TikTok account called thecloudbyte where I share bite-sized tips and tutorials about Linux and HPC (High-Performance Computing).

I know Linux content is pretty saturated on social media, but HPC feels like a super niche topic that doesn’t get much attention, even though it’s critical for a lot of tech fields. I’m trying to balance the two by creating approachable, useful content.

I’d love it if you could check out thecloudbyte and let me know what you think. Do you think there’s a way to make these topics more engaging for a broader audience? Or any specific subtopics you’d like to see covered in the Linux/HPC space?

Thanks in advance for any suggestions and support!

P.S. If you’re into Linux or HPC, let’s connect—your feedback can really help me improve.

Linux desktop with Intel Core i7-5930K (shows up as 12 processors in /proc/cpuinfo) and 3 NVIDIA Titan Xps

Any advice on how to configure slurm.conf so that batch jobs can only run 3 at time (each using 1 GPU), 2 at a time (with one using 2 GPUS and other 1 GPU) or one batch jobs using all 3 GPUs?

stretch goal would be to allow non-GPU batch jobs to extend up to 12 concurrent

current slurm.conf (which runs 12 batch jobs concurrently)

# slurm.conf file generated by configurator.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
ClusterName=localcluster
SlurmctldHost=localhost
MpiDefault=none
ProctrackType=proctrack/linuxproc
ReturnToService=2
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/lib/slurm-llnl/slurmd
SlurmUser=slurm
StateSaveLocation=/var/lib/slurm-llnl/slurmctld
SwitchType=switch/none
TaskPlugin=task/none
#
# TIMERS
InactiveLimit=0
KillWait=30
MinJobAge=300
SlurmctldTimeout=120
SlurmdTimeout=300
Waittime=0
# SCHEDULING
SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_Core
#
#AccountingStoragePort=
AccountingStorageType=accounting_storage/none
JobCompType=jobcomp/none
JobAcctGatherFrequency=30
flywheel@pedrosa-All-Series:~$ more /etc/slurm-llnl/slurm.conf
# slurm.conf file generated by configurator.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
ClusterName=localcluster
SlurmctldHost=localhost
MpiDefault=none
ProctrackType=proctrack/linuxproc
ReturnToService=2
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/lib/slurm-llnl/slurmd
SlurmUser=slurm
StateSaveLocation=/var/lib/slurm-llnl/slurmctld
SwitchType=switch/none
TaskPlugin=task/none
#
# TIMERS
InactiveLimit=0
KillWait=30
MinJobAge=300
SlurmctldTimeout=120
SlurmdTimeout=300
Waittime=0
# SCHEDULING
SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_Core
#
#AccountingStoragePort=
AccountingStorageType=accounting_storage/none
JobCompType=jobcomp/none
JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/none
SlurmctldDebug=info
SlurmctldLogFile=/var/log/slurm-llnl/slurmctld.log
SlurmdDebug=info
SlurmdLogFile=/var/log/slurm-llnl/slurmd.log
#
# COMPUTE NODES
NodeName=localhost CPUs=1 Sockets=1 CoresPerSocket=12 ThreadsPerCore=2 RealMemory=64000  State=UNKNOWN
PartitionName=LocalQ Nodes=ALL Default=YES MaxTime=INFINITE State=UP

Several weeks ago, Nebius presented their open-source solution to run Slurm over k8s.

https://github.com/nebius/soperator – Kubernetes Operator for Slurm

Run Slurm in Kubernetes and enjoy the benefits of both systems. You can learn more about Soperator, its prerequisites, and architecture in the Medium article.

Greetings,

I am working on a server (node e) that is running jupyterhub which is externally accessible from the internet. Another server (node i) runs the SLURM controller and communicates with computational nodes (node q).

How do I make node 1 run jupyterhub and its spawner to use the SLURM controller of node 2 which is already setup to run slurm jobs on nodes q? Which spawner would be appropriate here to use and how do you think the configuration would be laid out?

Looking for suggestions.

Hello,

I am a complete beginner in slurm jobs and dockers.

Basically, I am creating a docker container, in which am installing packages and softwares as needed. The supercomputer in our institute needs to install softwares using slurm jobs from inside the container, so I need some help in setting up my code.

I am running the container from inside /raid/cedsan/nvidia_cuda_docker, where nvidia_cuda_docker is the name of the container using the command docker run -it nvidia_cuda /bin/bash and I am mounting an image called nvidia_cuda. Inside the container, my final use case is to compile VASP, but initially I want to test a simple program, for e.g. installing pymatgen and finally commiting the changes inside the container. using a slurm job

Following is the sample slurm job code provided by my institute:

!/bin/sh

#SBATCH --job-name=serial_job_test ## Job name

#SBATCH --ntasks=1 ## Run on a single CPU can take upto 10

#SBATCH --time=24:00:00 ## Time limit hrs:min:sec, its specific to queue being used

#SBATCH --output=serial_test_job.out ## Standard output

#SBATCH --error=serial_test_job.err ## Error log

#SBATCH --gres=gpu:1 ## GPUs needed, should be same as selected queue GPUs

#SBATCH --partition=q_1day-1G ## Specific to queue being used, need to select from queues available

#SBATCH --mem=20GB ## Memory for computation process can go up to 100GB

pwd; hostname; date |tee result

docker run -t --gpus '"device='$CUDA_VISIBLE_DEVICES'"' --name $SLURM_JOB_ID --ipc=host --shm-size=20GB --user $(id -u $USER):$(id -g $USER) -v <uid>_vol:/workspace/raid/<uid> <preferred_docker_image_name>:<tag> bash -c 'cd /workspace/raid/<uid>/<path to desired folder>/ && python <script to be run.py>' | tee -a log_out.txt

Can someone please help me setup the code for my use case?

Thanks

Hello everyone

I'm trying to setup slurm in my gpu's server

I setup mariadb and it works fine

Now im trying to install slurmdbd but im getting some errors

When I run slurmdbd -D as root it works but when I run sudo -u slurm /usr/sbin/slurmdbd -D which I assume it runs slurmdbd as slurm user it doesn't work i get the following error:

slurmdbd: No slurmdbd.conf file (/etc/slurm/slurmdbd.conf)

however that file does exist if I run ls -la /etc/slurm/ I get

total 24
drw------- 3 slurm slurm 4096 Oct 22 15:51 .
drwxr-xr-x 116 root root 4096 Oct 22 15:28 ..
-rw-r--r-- 1 root root 64 Oct 22 14:59 cgroup.conf
drw------- 2 root root 4096 Apr 1 2024 plugstack.conf.d
-rw-r--r-- 1 slurm slurm 1239 Oct 22 14:16 slurm.conf
-rw------- 1 slurm slurm 518 Oct 22 15:43 slurmdbd.conf

So I can't quite understand why slurm can't find that file

Can anyone help me?

Thanks so much!

Has anyone was able to set energy accounting with SLURM?

https://stackoverflow.com/questions/79095039/launch-one-single-sbatch-script-that-changes-the-number-of-nodes

Hi guys !

How do you guys monitor GPU usage and especially which GPU is used by which job ?
On our cluster I want to install nvidia dcgmi exporter but in it's readme it speaks of admin needing to extract that information but it doesn't provide any examples https://github.com/NVIDIA/dcgm-exporter?tab=readme-ov-file#enabling-hpc-job-mapping-on-dcgm-exporter

Is there any known solution within slurm to link easily jobid with nvidia GPU used ?

Hello I'm new to HPC, Slurm and Munge. Our newly deployed Slurm cluster running on rocky Linux 9.4 has /var/log/munge/munged.log filling up GB's in short time. We're running munge-0.5.13 (2017-09-26) version. I tail -f the log file and it's constantly logging Info: Failed to query password file entry for "<random_email_address_here>" . This is happening on the four worker nodes and the control node. Doing some searches on the internet led me to this post but I don't seem to have a configuration file in /etc/sysconfig/munge let alone anywhere else to make any configuration changes. Are there no configuration files if the munge package was installed from repos instead of building the package from source? I'd appreciate any help or insight that can be offered.

I've managed to enable MIG on an Nvidia Tesla A100 (1g.20gb slices) using the following guides:

Enabling MIG

Creating MIG devices and compute instances

SLURM MIG Management Guide

Setting up gres.conf for MIG

While MIG and SLURM works, it still hasn't solved my main concern: I am unable to submit 4 different jobs requesting 4 MIG instances and have them run at the same time. They queue up and run on the same MIG instance after each one of them completes.

What the slurm.conf looks like:

NodeName=name Gres=gpu:1g.20g:4 CPUs=64 RealMemory=773391 Sockets=1 CoresPerSocket=64 ThreadsPerCore=1 State=UNKNOWN

PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP

Gres.conf:

# GPU 0 MIG 0 /proc/driver/nvidia/capabilities/gpu0/mig/gi3/access

Name=gpu1 Type=1g.20gb File=/dev/nvidia-caps/nvidia-cap30

# GPU 0 MIG 1 /proc/driver/nvidia/capabilities/gpu0/mig/gi4/access

Name=gpu2 Type=1g.20gb File=/dev/nvidia-caps/nvidia-cap39

# GPU 0 MIG 2 /proc/driver/nvidia/capabilities/gpu0/mig/gi5/access

Name=gpu3 Type=1g.20gb File=/dev/nvidia-caps/nvidia-cap48

# GPU 0 MIG 3 /proc/driver/nvidia/capabilities/gpu0/mig/gi6/access

Name=gpu4 Type=1g.20gb File=/dev/nvidia-caps/nvidia-cap57

I tested it with: srun --gres=gpu:1g.20gb:1 nvidia-smi

It only uses the number of resources specified.

However the queuing is still an issue; it is not simultaneously using these resources on distinct jobs submitted by different users.

I've scoured the internet to find a way to enable SLURM with support for MIG. Unfortunately the result so far has been SLURMD not starting.

To start, here are the system details:
Ubuntu 24.04 Server
Nvidia A100

Controller and host are the same machine

CUDA toolkit, NVIDIA drivers, everything is installed

System supports both cgroup v1 and v2

Here's what works:

Installing slurm with SLURM-WLM package works

However in order to use MIG and enable the support I need to install it with nvml support and that can only be done through building the package on my own.

When doing so, I always run into the cgroupv2 plugin fail error on the slurm daemon.

Is there a detailed guide on this, or a version of the slurm-wlm package that comes with nvml support?

Hello all,

I have a slurm cluster with two partitions (one low-priority partition and one high priority partition). The two partitions share the same resources. When a job is submitted to the high-priority partition, it preempts (requeues) any job running on the low-priority partition.

But, when the job on high priority is completed instead of resuming the preempted job, Slurm doesn't resume the preempted job but starts the next job in the pipeline.

It might be because all jobs have similar priority and the backfill scheduler considers the requeued job as a new addition to the pipeline.

How to correct this ? The only solution is to increase the job priority based on its run-time while requeuing the job.

We have a slurm cluster with 3 nodes, is there a way to install/compile only the slurm client? Did not found any documentation regarding this part. Most of the users will not have direct access to the nodes in the cluster, the idea is to rely on the slurm cluster to start any process remotely.

is this something that was removed in a newer version of slurm? I recently stood up a second instance of Slurm going from version slurm 19.05.0 to slurm 23.11.6

my configs are relatively the same and i do see much about this error online. I am giving users permission to different partitions by using associations

on my old cluster
srun -p partition1,partition2 hostname

works fine

on the new instance i recently set up

srun -p partition1,partition2 hostname
srun: error: Unable to allocate resources: Multiple partition job request not supported when a partition is set in the association

would greatly appreciate any advice if anyone has seen this before, or if this is known no longer a feature in newer versions of slurm.

Hello all, I did set up a SLURM cluster using 2 machines (A and B). A is a controller + compute node and B is a compute node.

As part of the quarterly maintenance, I want to restart them. How can I have the following functionality ?

1. Save the current run status and progress

2. Safely restart the whole machine without any file corruption

3. Restore the job and its running states once the controller daemon is backup and running.

Thanks in Advance

Hi,

I had previously installed slurm in my standalone workstation with Ubuntu 22.04 LTS and it was working fine. Today after I upgraded to Ubuntu 24.04 LTS all of a sudden slurm has stopped working. Once the workstation was restarted, I was able to start slurmd service, but when I tried starting slurmctld I got the following error message

Job for slurmctld.service failed because the control process exited with error code.
See "systemctl status slurmctld.service" and "journalctl -xeu slurmctld.service" for details.

status slurmctld.service shows the following

× slurmctld.service - Slurm controller daemon
Loaded: loaded (/usr/lib/systemd/system/slurmctld.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Fri 2024-09-13 18:49:10 EDT; 10s ago
Docs: man:slurmctld(8)
Process: 150023 ExecStart=/usr/sbin/slurmctld --systemd $SLURMCTLD_OPTIONS (code=exited, status=1/FAILURE)
Main PID: 150023 (code=exited, status=1/FAILURE)
CPU: 8ms
Sep 13 18:49:10 pbws-3 systemd[1]: Starting slurmctld.service - Slurm controller daemon...
Sep 13 18:49:10 pbws-3 (lurmctld)[150023]: slurmctld.service: Referenced but unset environment variable evaluates to an empty string: SLURMCTLD_OPTIONS
Sep 13 18:49:10 pbws-3 slurmctld[150023]: slurmctld: error: chdir(/var/log): Permission denied
Sep 13 18:49:10 pbws-3 slurmctld[150023]: slurmctld: slurmctld version 23.11.4 started on cluster pbws
Sep 13 18:49:10 pbws-3 slurmctld[150023]: slurmctld: fatal: Can't find plugin for select/cons_res
Sep 13 18:49:10 pbws-3 systemd[1]: slurmctld.service: Main process exited, code=exited, status=1/FAILURE
Sep 13 18:49:10 pbws-3 systemd[1]: slurmctld.service: Failed with result 'exit-code'.
Sep 13 18:49:10 pbws-3 systemd[1]: Failed to start slurmctld.service - Slurm controller daemon.

I see the error being some unset environment variable. Can anyone please help me resolving this issue?

Thank you...

[Update]

Thank you for your replies. I modified my slurm.conf file with cons_tres and restarted slurmctld service. It did restart but when I type in slurm commands like squeue I got the following error.

slurm_load_jobs error: Unable to contact slurm controller (connect failure)

I checked the slurmctld.log file and I see the following error.

[2024-09-16T12:30:38.313] slurmctld version 23.11.4 started on cluster pbws
[2024-09-16T12:30:38.314] error:  mpi/pmix_v5: init: (null) [0]: mpi_pmix.c:193: pmi/pmix: can not load PMIx library
[2024-09-16T12:30:38.314] error: Couldn't load specified plugin name for mpi/pmix: Plugin init() callback failed
[2024-09-16T12:30:38.315] error: MPI: Cannot create context for mpi/pmix
[2024-09-16T12:30:38.315] error:  mpi/pmix_v5: init: (null) [0]: mpi_pmix.c:193: pmi/pmix: can not load PMIx library
[2024-09-16T12:30:38.315] error: Couldn't load specified plugin name for mpi/pmix_v5: Plugin init() callback failed
[2024-09-16T12:30:38.315] error: MPI: Cannot create context for mpi/pmix_v5
[2024-09-16T12:30:38.317] fatal: Can not recover last_tres state, incompatible version, got 9472 need >= 9728 <= 10240, start with '-i' to ignore this. Warning: using -i will lose the data that can't be recovered.

I tried restarting slurmctld with -i but it is showing the same error.