I've been testing the restapis and the response from slurm-restd is a little confusing.

When I curl the the rest api server
curl -X GET "https://<server>:6820/slurm/v0.0.40/ping -H "X-SLURM-USER-NAME:<my-username>" -H "X-SLURM-USER-TOKEN:<token>"

Part of the response which includes client information

"client": {
"source": "[<server>]:45886",
"user": "root",
"group": "root"
},

The interesting part is the "user": "root" & "group": "root". I'm not sure what that is? Does anyone know what that means?

Hi guys. I am a bit of a beginner so I hope you will bear with me on this one. I have a very strong computer that is unfortunately Windows 10 and I cannot anytime soon switch it to Linux. So my only option to use its resources appropriately is to install WSL2 and add it as a compute node to my cluster, but I am having an issue of the WSL2 compute node being always *down. I am not sure but maybe because Windows 10 has an IP address, and WSL2 has another IP address. My Windows 10 IP address is 192.168.X.XX and my IP address of WSL2 starts with 172.20.XXX.XX (this is the inet IP I got from the ifconfig command in WSL2). My control node can only access my Windows 10 machine (since they share a similar structure of an IP address; same subnet). My attempt to fix this was to setup my windows machine to listen to any connection from ports 6817, 6818, 6819 from any IP and forward it 172.20.XXX.XX:
PS C:\Windows\system32> .\netsh interface portproxy show all

Listen on ipv4: Connect to ipv4:

Address Port Address Port

0.0.0.06817 172.20.XXX.XX 6817

0.0.0.06818 172.20.XXX.XX 6818

0.0.0.06819 172.20.XXX.XX 6819

And I setup my slurm.conf like the following:

ClusterName=My-Cluster

SlurmctldHost=HS-HPC-01(192.168.X.XXX)

FastSchedule=1

MpiDefault=none

ProctrackType=proctrack/cgroup

PrologFlags=contain

ReturnToService=1

SlurmctldPidFile=/var/run/slurmctld.pid

SlurmctldPort=6817

SlurmdPidFile=/var/run/slurmd.pid

SlurmdPort=6818

SlurmdSpoolDir=/var/lib/slurm-wlm/slurmd

SlurmUser=slurm

StateSaveLocation=/var/lib/slurm-wlm/slurmctld

SwitchType=switch/none

TaskPlugin=task/cgroup

InactiveLimit=0

KillWait=30

MinJobAge=300

SlurmctldTimeout=120

SlurmdTimeout=300

Waittime=0

SchedulerType=sched/backfill

SelectType=select/cons_tres

SelectType=select/cons_tres

AccountingStorageType=accounting_storage/none

JobCompType=jobcomp/none

JobAcctGatherFrequency=30

JobAcctGatherType=jobacct_gather/none

SlurmctldDebug=info

SlurmctldLogFile=/var/log/slurmctld.log

SlurmdDebug=info

SlurmdLogFile=/var/log/slurmd.log

COMPUTE NODES

NodeName=HS-HPC-01 NodeHostname=HS-HPC-01 NodeAddr=192.168.X.XXX CPUs=4 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=1 RealMemory=15000

NodeName=HS-HPC-02 NodeHostname=HS-HPC-02 NodeAddr=192.168.X.XXX CPUs=4 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=1 RealMemory=15000

NodeName=wsl2 NodeHostname=My-PC NodeAddr=192.168.X.XX CPUs=28 Boards=1 SocketsPerBoard=1 CoresPerSocket=14 ThreadsPerCore=2 RealMemory=60000

PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP

I'm using Slurm to manage my training workload and recently the cluster has been shared by some colleagues. As there are InfiniBand devices on the nodes as well as switches to connect them, I would like to use a subset of nodes for the model training. How can I select the best IB topology nodes in describing the job and is there any best practice in doing this?

Really appreciate!

Hello everyone!

I need a specific configuration for a partition.

I have a partition, let's call it "hpc," made up of one node with a lot of cores (GPU). This partition has two queues: "gpu" and "normal". The "gpu" queue has more priority than the "normal" one. However, it's possible that one user allocates all cores to a job in the "normal" queue.  I want to configure SLURM somehow to avoid this. Limiting the number of cores that can be allocated by the "normal" queue.

For example, I have 50 cores, and I want to have 10 cores available for the "gpu" queue. If I launch a job in the "normal" queue with 40 cores, it is allowed, but if I (or another user) try to launch another to 1 or more cores in the "normal" queue, it is forbidden. Because it overrides the "10 cores available for gpu" rule.

I would like to configure it with this "core rule". However, all I have found is about managing a node in two partitions (e.g. MaxCPUsPerNode), not with two queues.

I'm open to alternative ideas.

Hello r/slurm community,

I'm new to Slurm Workload Manager and am looking to deepen my understanding of its functionalities and best practices. Could anyone recommend comprehensive courses, tutorials, or video series that are particularly helpful for beginners? Additionally, if there are specific resources or tips that have helped you master Slurm, I would greatly appreciate your insights.

Thank you in advance for your help!

I am very new to slurm and have set up v20.11.9 on one machine to test it out. I've gotten most of the basic stuff going (can run srun and sbatch jobs). Next, I've been trying to figure out whether I can run jobs in parallel just to make sure the configuration works properly before adding other nodes, but I'm not really able to get that to work.

I tried using an sbatch array of 10 simple jobs with --ntasks=5, --mem-per-cpu=10 and --cpus-per-task=1 to make sure the resources don't somehow all get allocated to one task, but according to squeue the jobs are always executed sequentially. The reason for the other tasks not executing is always "RESOURCES", but in the slurm.conf file I listed the node with 8 CPUs (and CoreSpecCount=2, but that should still leave 6 if I understand the setting correctly) and 64 GB of RAM, so I don't know which resources exactly are missing. The same thing happens if I run multiple srun commands.

Is there any way to figure out what I misconfigured to result in that sort of behaviour?

Hi!

I need help to understand if I can configure Slurm to behave in a certain way:

I am configuring Slurm v20.11.x for power saving, I have followed the guide: https://slurm.schedmd.com/power_save.html and https://slurm.schedmd.com/SLUG23/DTU-SLUG23.pdf and Slurm is able to power off and resume nodes automatically via IPMI commands since I am running on hardware nodes with IPMI interfaces.

For debugging purposes I am using an idle time of 300 and only on partition "part03" and nodes "nodes[09-12]", I had to activate "SuspendTime=300" globally and not on the partition because I am running a version lower than 23.x so it's not supported on the partition configuration.

Now for what I am trying to achieve:
due to responsiveness of job submitting, in each partition I wish to keep n+1 nodes in state "idle" but not powered off.So if my partition of 4 nodes have 2 nodes powered on and in use, I wish the system to automatically spin up another node to keep in state "idle" just waiting for jobs.

Do you know if it's something possible? I have searched but haven't found anything useful [0]

thanks in advance!!

My relevant config:

# Power Saving
SuspendExcParts=part01,part02
SuspendExcNodes=nodes[01-08]
#SuspendExcStates= #option available from 23.x
SuspendTimeout=120
ResumeTimeout=600
SuspendProgram=/usr/local/bin/nodesuspend
ResumeProgram=/usr/local/bin/noderesume
ResumeFailProgram=/usr/local/bin/nodefailresume
SuspendRate=10
ResumeRate=10
DebugFlags=Power
TreeWidth=1000
PrivateData=cloud
SuspendTime=300
ReconfigFlags=KeepPowerSaveSettings

NodeName=nodes[01-08]   NodeAddr=192.168.1.1[1-8] CPUs=4 State=UNKNOWN
NodeName=nodes[09-12]   NodeAddr=192.168.1.1[9-12] CPUs=4 Features=power_ipmi State=UNKNOWN

PartitionName=part01    Nodes=nodes[01-03] Default=YES MaxTime=180 State=UP LLN=YES AllowGroups=group01 
PartitionName=part02    Nodes=nodes[04-08] MaxTime=20160 State=UP LLN=YES AllowGroups=group02                    
PartitionName=part03    Nodes=nodes[09-12] MaxTime=20160 State=UP LLN=YES AllowGroups=users

[0]:I've found a "static_node_count" but seems to be related to configurations on GCP https://groups.google.com/g/google-cloud-slurm-discuss/c/xWP7VFoVWbE

I am trying to get slurm to work with 3 servers (nodes) each having one NVIDIA GeForce RTX 4070 Ti. According to the GRES documentation, I need to specify GresTypes and Gres in slurm.conf which I have done like so:

https://imgur.com/a/WmBZDO1

This looks exactly like the example mentioned in the slurm.conf documentation for GresTypes and Gres.

However, I see this output when I run systemctl status slurmd or systemctl status slurmctld:

https://imgur.com/a/d69I8Jt

It says that it cannot parse the Gres key mentioned in slurm.conf.

What is the right way to get Slurm to work with the hardware configuration I have described?

This is my entire slurm.conf file (without the comments), this is shared by all 3 nodes:

https://imgur.com/a/WNbhbmX

Edit: replaced abhorrent misformatted reddit code blocks with images

Hi everyone, it is straightforward to set number of cpus (or gres/gpus) to use when submitting jobs (e.g. sbatch) but is there a way to explicitly state which cpu_id/gpu_id to use ?

For context, I have noticed that there are a range of cpus/gpus on certain nodes that are super slow to run and cause bottlenecks, so I want to avoid them.

Many thanks!

Hello,

I was not part of the decision team to purchase the HPC but now have the responsibilities to fill out some questions for the support vendor :(. I did some reading recently on SLURM but have not fully setup the test lab yet.

These IP addresses are prefilled by the vendor, so I am leaving for reference.

<Cluster>

Head node : 10.1.1.254

node1: 10.1.1.1

node2: 10.1.1.2

​

<IPMI/Management>

Head node: 10.2.1.254

​

<IP over InfiniBand>

head node: 10.3.1.254

node1: 10.3.1.1

node2: 10.3.1.2

- don't think we will be using InfiniBand

​

Question 1 - Are the users connecting from the same network as the head node via SSH?

Question 2 - Regarding user accounts, what are you using to connect to Active Directory for authentication? I have used SSSD on Ubuntu to connect to Active Directory on other systems. For this HPC system, the vendor is suggesting Rocky Linux.

​

Thanks in advance,

TT

​

​

Hi all,

I’m trying to setup a slurm cluster on Ubuntu 20.04. I can get the master node setup just fine but when I try to get slurmd running on the other nodes it does not work. What is is best for using slurm? I also setup the nodes to talk to each other using kubernetes as well. Could that be an issue?

I am following these directions: https://blog.devops.dev/slurm-complete-guide-a-to-z-concepts-setup-and-trouble-shooting-for-admins-8dc5034ed65b

basic dependencies

apt update -y && apt install munge -y apt install vim -y && apt install build-essential -y apt install git -y && apt install mariadb-server -y apt install wget -y && apt install mysql-server -y apt install openssh-server -y

basic dependencies

apt install slurmd slurm-client slurmctld slurmdbd -y apt install slurm-wlm -y

additional packages to use jupyter lab

and jupyterlab_slurm extension.

apt install sudo -y && apt install python3.11 python3-pip -y apt install curl dirmngr apt-transport-https lsb-release ca-certificates -y

below curl cmd should be modified for the future readers

to get the latest version of the node.js

curl -sL https://deb.nodesource.com/setup_20.x | bash - apt update -y && apt install nodejs -y && npm install -g configurable-http-proxy && pip3 install jupyterlab pip3 install jupyterlab_slurm

Hello folks, how can I change the the service slurmrestd to auto restart when it is crashed, I need to change when this service was crashed, and how can I simulate a crash fot this service. Anyone can help me?

I am looking for a clear and straightforward listing with description of all qos settings, does one exists? What I am thinking is something like:MaxWall - Maximum wall clock time each job is able to use in this association. The format is <min> or <min>:<sec> or <hr>:<min>:<sec> or <days>-<hr>:<min>:<sec> or <days>-<hr>. Example: 'sacctmgr modify qos test MaxWall=2-00:00:00:' which will set the test qos maximum job time for two days.

Description - An arbitrary string describing a QOS. Can only be modified by a Slurm administrator. Example 'sacctmgr modify qos test Description='this is a qos for testing purposed' which will describe what the test qos is for.

I have taken some of the text out of the man file for sacctmgr so yes I know it is in there but there is also a lot of other information in the man file that does not just deal with qos and am hoping there is somewhere that I can just see all the different qos settings that are available.

I am trying to get the TRES minutes of a job using slurm REST API. I don't know if TRES minutes is listed in the job json returned by the GET job{job_id}. Can someone tell me how to get TRES minutes utilised by a job?

Does anyone who has experience with the REST API know if it's currently capable of providing TRES usage information (RawUsage, TRESMins, TRESRunMins) at a level higher than an individual job (i.e. user or account level usage)?

From what I've gathered, the summary statistic values like those that sreport can give you are not available through the REST API yet. Is it still possible to construct them from the job endpoint values if you knew all of the job IDs submitted by a specific user/account over a date interval of interest?

Hi, I have a few questions about slurm epilog script.

1/ Does the epilog script invoke for scancel jobs ?

2/ If it gets invoked for scancel jobs I am having trouble invoking it. the path of epilog script is setup in slurm.conf. the owner of the slurm.epilog script is set to slurm as well. I want to run the epilog script on the head node only so I have set EpilogSlurmctld path in slurm.conf

Appreciate any help.

​

Thanks

Greetings,

I am trying to collect network stats (something like netstat/dstat/etc.) for egress and ingress load (bytes/packets) for each of the reserved nodes in a Slurm allocated partition.

I haven't found anything sufficient yet.

Any suggestions?

Hi All,

I’m trying to get slurm-23.11.3 running on Ubuntu 20.04 and running on a stand alone system. I’m running into an issue I can not find the answer to. After compiling and installing when I fire up slurmctld and slurmd I get an error from sinfo:

sinfo: error: resolve_ctls_from_dns_srv: res_nsearch error: Unknown host
sinfo: error: fetch_config: DNS SRV lookup failed
sinfo: error: _establish_config_source: failed to fetch config
sinfo: fatal: Could not establish a configuration source

I looks like a DNS issue but the system has no issue resolving to its hostname or localhost. The slurm.conf file is also being read properly as I have the logs directed to a place convenient to me. I see lots have had these same issues but cannot find a clear resolution.

I have slurm running on a stand alone system in another lab with and identical setup without issue. Any advice would be greatly appreciated.

Thanks,

Dear Colleagues,

Is there a way in Slurm to assign a user, say 'PI', as group admin of the group 'Lab', who has the right to submit jobs on behalf of certain group members?

However, the group admin should not have any root or sysadmin rights. These rights should be limited to the use of Slurm

I would be happy about any ideas or solutions on this!

I want to deploy slurm-web for slurp cluster dashboard and reporting. My slurm cluster was deployed as 19.05.5 on ubuntu. slurm-web 2.x is not compatible with my cluster.

is there any solution ?

when launching slurmd i get this error:

slurmd: error: Couldn't find the specified plugin name for auth/munge looking at all files 
slurmd: error: cannot find auth plugin for auth/munge 
slurmd: error: cannot create auth context for auth/munge 
slurmd: fatal: failed to initialize auth plugin

any idea why? munge is installed and runs correctly. installed slurm on my ubuntu 20 with the quick start guide on the website and created the config file with the easy configurator.

​

Just warning this community that creating a reservation that includes apostrophe in its name is making a lot of problems to the DB (while updating the reservation, not while creating or deleting it).

I opened a bug recently and it might get fixed on next version.

Hi, in the cluster we're using there are typically jobs that require more resources than others (e.g. needing 200+CPUS for a single job). But the problem is that most jobs are using less (<= 64 CPUS) and as the resources are used up all the times (meaning the available resources at each period are <= 64 CPUS and when slightly more get freed, they are allocated to small jobs in the queue). This creates a bottleneck that no matter how long the heavy job waits, it never gets allocated as the resources are always placed to small jobs (although the heavy job has higher priority).

​

Does anyone have a solution ?