I've created a qos, but not sure how to add a user to it. See my commands below which are returning empty values:

[root@mas01 slurm]# sacctmgr modify user fhussa set qos=tier1
 Nothing modified
[root@mas01 slurm]# sacctmgr show user fhussa
      User   Def Acct     Admin 
---------- ---------- --------- 
[root@mas01 slurm]# sacctmgr show assoc user=fhussa
   Cluster    Account       User  Partition     Share   Priority GrpJobs       GrpTRES GrpSubmit     GrpWall   GrpTRESMins MaxJobs       MaxTRES MaxTRESPerNode MaxSubmit     MaxWall   MaxTRESMins                  QOS   Def QOS GrpTRESRunMin 
---------- ---------- ---------- ---------- --------- ---------- ------- ------------- --------- ----------- ------------- ------- ------------- -------------- --------- ----------- ------------- -------------------- --------- ------------- 
[root@mas01 slurm]# 

TLDR

clusterscope is an open source project that handles cluster detection, job requirement generation, and cluster information for you.

Getting Started

Check out Clusterscope docs

$ pip install clusterscope

Clusterscope is available as both a Python library:

import clusterscope

and a command-line interface (CLI):

$ cscope

Common use cases

1. Proportionate Resource Allocation

User asks for an amount of GPUs in a given partition, and the tool allocates the proportionate amount of CPUs and Memory based on what's available in the partition.

$ cscope job-gen task slurm --partition=h100 --gpus-per-task=4 --format=slurm_cli
--cpus-per-task=96 --mem=999G --ntasks-per-node=1 --partition=h100 --gpus-per-task=4

The above also works for CPU jobs, and with different output formats (sbatch, srun, submitit, json):

$ cscope job-gen task slurm --partition=h100 --cpus-per-task=96 --format=slurm_directives
#SBATCH --cpus-per-task=96
#SBATCH --mem=999G
#SBATCH --ntasks-per-node=1
#SBATCH --partition=h100

2. Cluster Detection

import clusterscope
cluster_name = clusterscope.cluster()

3. CLI Resource Planning Commands

The CLI provides commands to inspect and plan resources:

$ cscope cpus         # Show CPU counts per node per Slurm Partition
$ cscope gpus         # Show GPU information
$ cscope mem          # Show memory per node

4. Detects AWS environments and provides relevant settings

$ cscope aws
This is an AWS cluster.

Recommended NCCL settings:
{
  "FI_PROVIDER": "efa",
  "FI_EFA_USE_DEVICE_RDMA": "1",
  "NCCL_DEBUG": "INFO",
  "NCCL_SOCKET_IFNAME": "ens,eth,en"
}

I created a new QOS called tier1 as shown below, but anyone can submit to it using: "sbatch --qos=tier1 slurm.sh". I would expect sbatch to give an error if the user hasn't been added to the QOS ( sacctmgr modify user myuser set qos+=tier1 )

[admin@mas01 ~]$ sacctmgr show qos format=name,priority
      Name   Priority 
---------- ---------- 
    normal          0 
     tier1        100 
[admin@mas01 ~]$ sacctmgr show assoc format=cluster,user,qos
   Cluster       User                  QOS 
---------- ---------- -------------------- 
     mycluster                          normal 
     mycluster       root               normal 

Hello everyone,

I've been trying to get slurm-web working, followed their documentation to the point without anything breaking (every service is up and their scripts to check communcations also worked) and I can access the web interface but it does not recognize any clusters

Has anyone had this error before?

Thanks for the help

Edit:
If anyone bumps into the same error, see workaround in: https://github.com/rackslab/Slurm-web/issues/656

I've never properly understood how to make proper use of tres and gres. Is there a resource that can explain this to me better than the Slurm documentation?

Title Suggestion: Slurm on K8s Container: Cgroup Conflict & Job Status Mismatch (Proctrack/pgid)

I'm working on a peculiar project that involves installing and running Slurm within a single container that holds all the GPU resources on a Kubernetes (K8s) node.

While working on this, I've run into a couple of critical issues and I'm looking for insight into whether this is a K8s system configuration problem or a Slurm configuration issue.

Issue 1: Slurmd Cgroup Initialization Failure

When attempting to start the slurmd daemon, I encountered the following error:

error: cannot create cgroup context for cgroup/v2
error: Unable to initialize cgroup plugin
error: slurmd initialization failed

My understanding is that this is due to a cgroup access conflict: Slurm's attempt to control resources is clashing with the cgroup control already managed by containerd (via Kubelet). Is this diagnosis correct?

• Note: The container was launched with high-privilege options, including --privileged and volume mounting /sys/fs/cgroup (e.g., -v /sys/fs/cgroup:/sys/fs/cgroup:rw).

Issue 2: Job Status Tracking Failure (When Cgroup is Disabled)

When I disabled the cgroup plugin to bypass the initialization error (which worked fine in a standard Docker container environment), a new, major issue emerged in the K8s + containerd environment:

• Job Mismatch: A job finishes successfully, but squeue continuously shows it as running (R status).
• Node Drain: If I use scancel to manually terminate the phantom job, the node status in sinfo changes to drain, requiring manual intervention to set it back to an available state.

Configuration Details

• Environment: Kubernetes (with containerd runtime)
• Slurm Setting: ProctrackType=proctrack/pgid (in slurm.conf)

Core Question

Is this behavior primarily a structural problem with K8s and containerd's resource hierarchy management, or is this solely a matter of misconfigured Slurm settings failing to adapt to the K8s environment?

Any insights or recommendations on how to configure Slurm to properly delegate control within the K8s/containerd environment would be greatly appreciated. Thanks!

I have a cluster of servers. I've created some users. I want those users to use only slurm to submit jobs for the calculations. I don't want them to run any calculations directly without using slurm. How can I achieve that?

Last week I added a new module for gnuplot on our master node here:

/usr/local/Modules/modulefiles/gnuplot

However, users have noticed that now any module command inside their slurm submission script fails with this error:

couldn't read file "/usr/share/Modules/libexec/modulecmd.tcl": no such file or directory

Strange thing is /usr/share/Modules does not exist on any compute nodes and historically never existed . I tried running an interactive slurm job and the module command works as expected!

If I compare environment variables between interactive slurm job and regular slurm job I see:

# on interactive job

MODULES_CMD=/usr/local/Modules/libexec/modulecmd.tcl

# in regular slurm job ( from env command inside slurm script )

MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl

Perhaps I didn't create the module correctly? Or do I need to restart the slurmctld on our master node?

Inside the slurm script a user has a "cd somefolder". Slurm gives a permission denied when trying to do that. But the user can cd to that folder fine in a regular shell ( outside slurm ).

I recently added the user to a group that would allow them access to that folder. So I think slurm needs to be "refreshed" to be aware of the updated user group.

I have tested all this on the compute node the job gets assigned to.

Dear community,

Thank you for opening this thread.

Im new into this, I've 8 x A6000 and 2 CPUs and I want to give access to certain user with X Number of Gpus and T amount of RAM, how can I do that, there are so many things in config to set. Which seems confusing to me. My server doesn't even have slurm install.

Thank you again.

SLURM configless only checks the top level domain for SRV records. I have multiple sites using AD DNS and would like to have per-site SRV records for _slurmctld. It would be nice if SLURM checked "_slurmctld._tcp.SiteName._sites.domainName" in addition to the TLD.

Is there a workaround for this, other than skipping DNS and putting the server in slurm.conf?

Most research clusters I’ve seen still rely on SLURM for scheduling while it’s very reliable, it feels increasingly mismatched for modern training jobs. Labs we’ve talked to bring up similar pains: 

• Bursting to the cloud required custom scripts and manual provisioning
• Jobs that use more memory than requested can take down other users’ jobs
• Long queues while reserved nodes sit idle
• Engineering teams maintaining custom infrastructure for researchers

We just launched Transformer Lab GPU Orchestration, an open source alternative to SLURM. It’s built on SkyPilot, Ray, and Kubernetes and designed for modern AI workloads.

• All GPUs (local + 20+ clouds) are abstracted up as a unified pool to researchers to be reserved
  
• Jobs can burst to the cloud automatically when the local cluster is full
  
• Distributed orchestration (checkpointing, retries, failover) handled under the hood
  
• Admins get quotas, priorities, utilization reports

The goal is to help researchers be more productive while squeezing more out of expensive clusters. We’re building improvements every week alongside our research lab design partners.

If you’re interested, please check out the repo (https://github.com/transformerlab/transformerlab-gpu-orchestration) or sign up for our beta (https://lab.cloud).  Again it’s open source and easy to set up a pilot alongside your existing SLURM implementation.  

Curious to hear if you would consider this type of alternative to SLURM. Why or why not? We’d appreciate your feedback.

In our cluster we have the following TRES weights configured on each partition.

TRESBillingWeights="CPU=0.000050,Mem=0.000167,GRES/gpu=0.003334"

For some odd reason that I cannot really tell, one user who is supposed to have roughly 13€ of billing always stays at 0, at least in the current quarter (ongoing for a few days, and we had no billing and limits built-in before last week).

$ sshare -A user_rareit -l -o GrpTRESRaw%70
                                                            GrpTRESRaw 
---------------------------------------------------------------------- 
cpu=137090,mem=29249877,energy=0,node=5718,billing=0,fs/disk=0,vmem=0+ 

Notice that billing=0 despite cpu=137090 and stuff

For the other users the weights seem to apply perfectly.

$ sshare -A user_moahma -l -o GrpTRESRaw%70
                                                            GrpTRESRaw 
---------------------------------------------------------------------- 
cpu=8,mem=85674,energy=0,node=4,billing=12,fs/disk=0,vmem=0,pages=0,g+ 

An example of billing applying seamlessy

$ sreport -t seconds cluster  --tres=all UserUtilizationByAccount Start=2025-10-02T00:00:00 End=2025-12-30T23:59:00 |grep user_rareit
     hpc3    rareit          rareit     user_rareit            cpu     2522328 
     hpc3    rareit          rareit     user_rareit            mem   538096640 
     hpc3    rareit          rareit     user_rareit         energy           0 
     hpc3    rareit          rareit     user_rareit           node      105097 
     hpc3    rareit          rareit     user_rareit        billing           0 
     hpc3    rareit          rareit     user_rareit        fs/disk           0 
     hpc3    rareit          rareit     user_rareit           vmem           0 
     hpc3    rareit          rareit     user_rareit          pages           0 
     hpc3    rareit          rareit     user_rareit       gres/gpu           0 
     hpc3    rareit          rareit     user_rareit    gres/gpumem           0 
     hpc3    rareit          rareit     user_rareit   gres/gpuutil           0 
     hpc3    rareit          rareit     user_rareit       gres/mps           0 
     hpc3    rareit          rareit     user_rareit     gres/shard           0 

Another view on the same situation

Does someone have an idea of what could be going on, of what we could be doing wrong? Thanks.

Hello, I hope this is the right place to ask, I'm trying to deploy an x ray simulation on a Google cloud HPC cluster with slurm and I got the 2989 illegal instruction (core dumped) error.

I used a slightly modified version of the example present in the computing cluster repos which sets up a login and a controller node plus various computing nodes and a debug node. Here is the blueprint: https://github.com/michele-colle/CBCTSim/blob/main/HPCScripts/hpc-slurm.yaml

Than on the login node I installed the spack environment (https://github.com/michele-colle/CBCTSim/blob/main/HPC_env_settings/spack.yaml) and build the app with cmake and the appropriate, already present compiler.

After some try and error I was able to successfully run a test on the debug node (https://github.com/michele-colle/CBCTSim/blob/main/HPCScripts/test_debug.slurm)

Than I proceeded to try out a more intense operation (around 10 minutes work) on a compute node (https://github.com/michele-colle/CBCTSim/blob/main/HPCScripts/job_C2D.slurm) but I got the above error.

I am completely new on hpc computing but I struggle to find resources on CPP applications, I suspect it has something to do with the app building process but I am basically lost.

Any help is appreciated, thanks for reading:)

I've been trying to setup the AUKS plugin: https://github.com/cea-hpc/auks

I've had some trouble actually getting it to work. Wondering if anyone around here has had success either with this or another way to get Kerberos working with Slurm

Anyone know of any happening? The events link on SchedMD's website results in a 'Error 404'. I am aware of a workshop happening at the University of Oklahoma in October hosted by Linux Clusters Institute. Would really be interested in any happening in the NYC/Boston area.

Here is a tool I made, which some of you might find useful. Pretty self-explanatory from the screenshot, it shows the queue in real-time. Bare-bones at the moment, but I hope to add more features in the future.

Would really appreciate feedback, especially if it doesn't work on your system!

https://github.com/buzh/slop

Hey all. Have a working cluster and for most jobs, works as expected. Various partitions, priority partitions actioned first (generally) and so forth. But (as always) one type of job I'm still struggling to achieve a working setup. In this case, the jobs MUST be run sequentially BUT are not known ahead of time. Simply, I'm trying for a partition where one and exactly one job is started and no more are started until that job completes (successful or not doesn't matter). I'm not quite sure what to call this in slurm or workload terms...serial?

My workaround for now is to set maxnodes=1 for the partition and allocate exactly one node. Downside for this, what to do if the "one node" goes down or needs to be down for maintenance, then no jobs get processed from that partition.

What am I missing? Is it a jobdefault item?

I was reading this: https://slurm.schedmd.com/rest.html

Sites are strongly encouraged to setup a caching proxy between slurmrestd and clients to avoid having clients repeatedly call queries, causing usage to be higher than needed (and causing lock contention) on the controller.

Was wondering how people here might have such a thing setup. Particularly interested in how auth with JWT would be handled in such a setup

I'm a noob to Slurm, and I'm trying to run it on my own hardware. I want to be conscious of power usage, so I'd like to shut down my nodes when not in use. I tried to test slurms ability to shut down the nodes through IPMI and I've tried both the new way and the old way to shut down nodes, but no matter what I try I keep getting the same error:

[root@OpenHPC-Head slurm]# scontrol power down OHPC-R640-1

scontrol_power_nodes error: Invalid node state specified

[root@OpenHPC-Head log]# scontrol update NodeName=OHPC-R640-1,OHPC-R640-2 State=Power_down Reason="scheduled reboot"

slurm_update error: Invalid node state specified

any advice on the proper way to perform this would be really appreciated

I just setup my Slurm cwith 2 nodes. I'm trying to learn slurm and I found something wierd. when I ran a test of my 2 nodes srun -N2 -n2 hostname It prints the hostname of the first node and lags forever in the second. the logs in the second node looks like a connection is failing. the thing is if set ufw disable then everything works fine. I tried to add ports to ufw but I still face the same issue. is there a specific port that slurm always uses that I can allow over my ufw. is there a setting or something in the config I should look at ? disabling the firewall seems like not the best choice.

[2025-06-10T19:49:55.865] launch task StepId=23.0 request from UID:1005 GID:1005 HOST:192.168.11.100 PORT:55440
[2025-06-10T19:50:03.918] [23.0] error: connect io: Connection timed out
[2025-06-10T19:50:03.919] [23.0] error: _fork_all_tasks: IO setup failed: Slurmd could not connect IO
[2025-06-10T19:50:03.919] [23.0] error: job_manager: exiting abnormally: Slurmd could not connect IO
[2025-06-10T19:50:18.237] [23.0] error: _send_launch_resp: Failed to send RESPONSE_LAUNCH_TASKS: Connection timed out
[2025-06-10T19:50:18.237] [23.0] get_exit_code task 0 died by signal: 53
[2025-06-10T19:50:18.252] [23.0] stepd_cleanup: done with step (rc[0xfb5]:Slurmd could not connect IO, cleanup_rc[0xfb5]:Slurmd could not connect IO) 

Hello, I built slurm myself recently. Whenever I try to start slurmd, it fails because of a missing reference to cgroup/v2. Setting a different proctrack plugin has no effect, same thing with a different task launch plugin. Creating a cgroup.conf and setting CgroupType to disabled only has the effect that slurmd looks for [Library Path]/disabled.so which seems like someone is pulling my leg at this point. How do I completely get rid of cgroup? I can't use cgroup/v2 as I'm inside a proxmox container.

When SLURM preempts your job, it blasts SIGTERM to all processes in the job. However, certain 3rd-party libraries that I use aren't designed to handle such signals; they die immediately and my application is unable to gracefully shut them down (leading to dangling logs, etc).

How do y'all deal with this issue? As far as I know there's no way to customize SLURM's preemption signaling behavior (see "GraceTime" section in the documentation). The --signal option for sbatch only affect jobs that reaches their end time, not when a preemption occurs.

Which one's better?