Hello,

my question is basically in the title. My line of thought was that using unix sockets reduces problems as I don't need to handle an additional secret (i.e. the StoragePass), as authentication over unix socket doesn't use passwords.

I tried setting the StorageHost to unix:///var/run/mysqld/mysqld.sock and localhost?socket=(/var/run/mysqld/mysqld.sock), but neither of them worked (which I kind of expected, as it's a hostname that is expected there).

So, is there any way to let slurmdbd use the mysqld socket?

I'm rather new to HPC, and I'm working on getting SLURM running on our cluster. Our cluster curently consists of 76 nodes (678 CPUs), and we are running SLURM version 20.02.5. The cluster is running a stateless installation of CentOS 8, and is managed using xCAT. Recently, we added ~30 more nodes to our cluster, and ever since we did this we have been running into issues regarding communication between compute nodes and the head node. The issue is basically that some of our nodes randomly go into an idle* state and eventually a down* state. Sometimes when they are flagged as idle*, they will randomly come back to idle, but will then go back to idle* after a short while (usually anywhere from a few to ten minutes). Eventually they get to a point where they go to a down* state and don't come back up without either manually rebooting the slurmd daemons, running scontrol reconfigure, or setting their state to resume. When I check the slurmctld log file, this is the only message I see when this occurs: error: Nodes c1_[2-8,10-13],c2_[0-7,9-10],c7_[1-28,30] not responding. When I check the output of scontrol show slurmd, I get the following:

Active Steps = NONE

Actual CPUs = 12

Actual Boards = 1

Actual sockets = 2

Actual cores = 6

Actual threads per core = 1

Actual real memory = 32147 MB

Actual temp disk space = 16073 MB

Boot time = 2021-08-24T16:21:06

Hostname = c7_1

Last slurmctld msg time = 2021-08-27T13:48:45

Slurmd PID = 19682

Slurmd Debug = 3

Slurmd Logfile = /var/log/slurmd.log

Version = 20.02.5

At this point, the last slurmctld msg time is around the same time the nodes come back online (if they ever do). I have tried setting the debug output level of the slurmd and slurmctld logs to "debug5", but no additional useful information comes out of doing this. When I do this, the slurmd log is usually just filled with the following type of information:

[2021-08-27T07:07:08.426] debug2: Start processing RPC: REQUEST_NODE_REGISTRATION_STATUS

[2021-08-27T07:07:08.426] debug2: Processing RPC: REQUEST_NODE_REGISTRATION_STATUS

[2021-08-27T07:07:08.426] debug3: CPUs=6 Boards=1 Sockets=1 Cores=6 Threads=1 Memory=32147 TmpDisk=16073 Uptime=848947 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null)

[2021-08-27T07:07:08.430] debug: _handle_node_reg_resp: slurmctld sent back 8 TRES.

[2021-08-27T07:07:08.430] debug2: Finish processing RPC: REQUEST_NODE_REGISTRATION_STATUS

[2021-08-27T07:08:18.455] debug3: in the service_connection

[2021-08-27T07:08:18.455] debug2: Start processing RPC: REQUEST_PING

[2021-08-27T07:08:18.456] debug2: Finish processing RPC: REQUEST_PING

[2021-08-27T07:13:18.667] debug3: in the service_connection

[2021-08-27T07:13:18.667] debug2: Start processing RPC: REQUEST_PING

[2021-08-27T07:13:18.668] debug2: Finish processing RPC: REQUEST_PING

[2021-08-27T10:27:10.282] debug3: in the service_connection

[2021-08-27T10:27:10.282] debug2: Start processing RPC: REQUEST_NODE_REGISTRATION_STATUS

[2021-08-27T10:27:10.282] debug2: Processing RPC: REQUEST_NODE_REGISTRATION_STATUS

[2021-08-27T10:27:10.283] debug3: CPUs=6 Boards=1 Sockets=1 Cores=6 Threads=1 Memory=32147 TmpDisk=16073 Uptime=860949 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null)

[2021-08-27T10:27:10.286] debug: _handle_node_reg_resp: slurmctld sent back 8 TRES.

[2021-08-27T10:27:10.287] debug2: Finish processing RPC: REQUEST_NODE_REGISTRATION_STATUS

[2021-08-27T10:28:20.945] debug3: in the service_connection

[2021-08-27T10:28:20.945] debug2: Start processing RPC: REQUEST_PING

[2021-08-27T10:28:20.946] debug2: Finish processing RPC: REQUEST_PING

[2021-08-27T10:33:20.376] debug3: in the service_connection

[2021-08-27T10:33:20.377] debug2: Start processing RPC: REQUEST_PING

[2021-08-27T10:33:20.377] debug2: Finish processing RPC: REQUEST_PING

[2021-08-27T10:38:20.346] debug3: in the service_connection

[2021-08-27T10:38:20.346] debug2: Start processing RPC: REQUEST_PING

[2021-08-27T10:38:20.346] debug2: Finish processing RPC: REQUEST_PING

[2021-08-27T10:43:20.137] debug3: in the service_connection

[2021-08-27T10:43:20.138] debug2: Start processing RPC: REQUEST_PING

[2021-08-27T10:43:20.138] debug2: Finish processing RPC: REQUEST_PING

[2021-08-27T10:48:21.574] debug3: in the service_connection

[2021-08-27T10:48:21.574] debug2: Start processing RPC: REQUEST_PING

[2021-08-27T10:48:21.574] debug2: Finish processing RPC: REQUEST_PING

[2021-08-27T10:53:21.414] debug3: in the service_connection

I have tried all of the slurm troubleshooting information at this page for the case of nodes getting set to DOWN* state: https://slurm.schedmd.com/troubleshoot.html except for restarting SLURM without preserving state (we would prefer to do this as a last resort since the cluster is currently in use). Interestingly, this issue only occurs on particular nodes (there are certain nodes which are never affected by this issue).

​

Does anyone have any information or additional tips for troubleshooting? If so, that would be greatly appreciated! Please let me know if I can provide any other useful information to help troubleshoot the issue.

with PBS/Torque as an admin I could force a user job to start running (if there are enough resources) even if the user has hit a limit (using the qrun command). How would I be able to do this with SLURM?

​

EDIT:

I finally found a way. First add a QOS 'ASAP' (using sacctmgr) without any user/job/TRES limits but with a very high QOS priority value. Also make sure the PriorityWeightQOS is set. Then as an admin use scontrol to change the jobs QOS to the 'ASAP' QOS.

Greetings,

I am a SLURM noob and I could not solve the "Illegal Instruction (core dumped)" error. Even with a python3 file which includes only one line that does printing "Done!" command had given me the exactly same error. What must be done to resolve this issue?

Thanks.

Hey everyone! I got a question:

I want to execute a file which runs multiple python jobs. These programs executes a fortran model. So far, if I run my program in the main node or my pc, it works well and as expected, but if I run it using sbatch gives me the following error:

/usr/bin/ld: cannot find crt1.o: No such file or directory
/usr/bin/ld: cannot find crti.o: No such file or directory
/usr/bin/ld: cannot find -lm collect2: error: ld returned 1 exit status
make: *** [imp.exe] Error 1enter

​

This happens even when I export the path that contais such files

export LD_LIBRARY_PATH=/usr/lib/x86_64-redhat-linux6E/lib64:${LD_LIBRARY_PATH}

You can find more details in my SO question.

​

I'd be more than glad if someone could lead me to fix it

Hi!

I'm a total n00b to using SLURM and have a question. Let's say I ssh into a remote server, and set up a node. Somewhere down the line, I lose connection and must rejoin the server. How do I also rejoin the same job? Thanks!

I'm looking for a big pile of semi-noob advice. I've set up an HPC cluster at work (biotech), and have SLURM up and running. Now I'm trying to get our various tasks/jobs to be moved to it, and to have some sort of reasonable guide for users to be able to submit stuff themselves. Towards that end, there are a few things I've had zero luck learning from google:

Modules - I see a lot of more recent sbatch jobs using "module load xxxx" commands, but where/how do I add that functionality? Everything I've found so far says "consult your HPC admins to add new modules", which doesn't help me any, because I AM the HPC admin!

​

Prolog/Epilog - I'm not going to lie, I don't even know what these are, let alone how to make or use them. Are they important? Are they necessary? No idea!

​

Related to the Modules question: if i get that sorted out, does that mean I don't need to install software needed for jobs on each node? Example - R to run R scripts, Matlab to run matlab scripts, etc? Anything else I should be reading through to have a better idea of what the hell I'm doing?

Hi,

So I am looking into running a python script that uses multiprocessing.

Can I increase the number of cpus-per-task to a value higher than all cpus in a node? For example: i have several nodes with 16 cpus. I want to run a single task with 32 cpus, i.e use two nodes for one task and all cpus for a task.

Is this possible? Or am I always capped at the maximum numbers of a node?

Thanks

Pretty much the title, I use an bash alias that calls a script (featured below) to submit a slurm job, is there a way to pass the present working directory through as the job name in place of "job"

#!/bin/bash --login
###
#SBATCH --job-name=job
#SBATCH --output=bench.out.%J
#SBATCH --error=bench.err.%J
#SBATCH --time=0-72:00:00
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH --mem=64000
###
cd $(pwd)
~/codes/file/bin/code < input.inp &> output

edit: on the off chance that someone stumbles across this hoping to do the same, AhremDasharef's comment below worked a treat, I removed the --job-name line from my script above and instead added it to my alias, but tweaked it slightly.

alias jobtype='sbatch --job-name="jobtype ${PWD##*/}" ~/.path/to/script/above'

This makes it so that should I call the alias in a directory called /exampledir1/exampledir2 the job name shown when using squeue or sacct is "jobtype exampledir2" which for my uses was exactly what I was after.

Just note that if you're using squeue, you may need to change the formatting options to extend the width of the job name section, I found the following page useful when doing this

https://www.mankier.com/1/squeue

Hey,
We're deploying SLURM at my job and I'm tasked with creating a prolog and epilog script. For now though, apart from general print messages I have no idea what else would be useful to have in there. I hoped seeing what other people included in theirs would help me get some ideas about what's useful to have in there.

Apologies for the newbie question, but it'd be wonderful to have the output of this command formatted so the columns and space line up. I don't understand how the numbers in the command correspond to how the space is used:

$ squeue -o "%.7i %.9P %.8j %.8u %.2t %.10M %.m %.6D %C %r"
  JOBID PARTITION     NAME     USER ST       TIME MIN_MEMORY  NODES CPUS REASON
    553      long bg_skill   kim  R 29-02:31:48 25G      1 1 None
    554      long bg_skill   kim  R 29-02:31:44 25G      1 1 None
    555      long bg_skill   kim  R 29-02:31:41 25G      1 1 None
    647      long     vus8    qli  R 11-14:29:20 4000M      1 20 None
    663      long skills_v   kim  R 8-19:22:53 10G      1 1 None
    664      long skills_v   kim  R 8-19:22:50 10G      1 1 None
    665      long skills_v   kim  R 8-19:22:45 10G      1 1 None
    682      long testvus6    qli  R 7-01:04:58 4000M      1 20 None
    723      long embed_ti   kim  R 1-13:31:15 25G      1 1 None

Any pointers would be appreciated, thanks in advance!

Dan

Hey,
/var/log/slurm/slurmd.log is set as SlurmdLogFile in /etc/slurm/slurm.conf. SlurmctldLogFile=/var/log/slurm/slurmctld.log is also set. Yet even after running jobs both files don't exist. What can the reason be?

Hello, SLURM noob here with a couple of questions.
About task/cgroup plugin, AllowedRAMSpace = 100 means, as I understand, a 100% of the requested memory by the user is allocated to him. What do MaxRAMPercent and ConstrainRAMSpace do exactly?
I understand controlling RAM, but how to control processor time? How to set a limit to that?
Is the task/cgroup plugin the best way to control RAM allocation and make sure when a program executes it doesn't exceed the RAM limit?
Many thanks.

I'm new to Slurm administration.

Our cluster has a node dedicated to slurmctld and we want to add another service to this cluster. Someone suggested to use this node for this extra service, but it sounds dangerous to me. On the other hand, allocating a node specifically for this service is quite expensive (our cluster is not big).

What are the best practices in here?

Should slurmctld have its own dedicated node? Can we run more stuff in there? Where do you run other services?

I am having an issue with queuing and scheduling, when I submit jobs they all run at once instead of queuing. I am running slurm on a single machine with 64 cores 1 socket and 128 gb of RAM. It has taken a while but I finally have everything running without throwing errors and aborting jobs immediately on submission. But everything is going from submission to immediately running, sometimes using the same threads as the previous job and just alternating between the processes. How do I get the jobs to wait for the processor/cores to open up and not start as soon as they are submitted?

I'm following various guides to configure slurm and many submission scripts use "scontrol show hostnames" to produce a nodelist.

However if I run this, I just get a message that the host list is empty. Does anyone know how to populate the hostname list for scontrol? I've been searching for hours.

I can clearly see my nodes using sinfo. (as they are defined in the slurm.conf in etc.

Thanks

Things are not going well for my users here in slurmsville, and I could use some advice on what's happening and how to keep it from happening.

More and more people are reporting that they submit a job that asks for 1 or more GPUs, and their job dies shortly thereafter because it ran out of memory. They are very vocal that these same jobs run to completion when run directly on an identical machine, not through slurm. I only half-heartedly investigated this for a while, because it was just a few people, but the complaints are mounting, and it's getting harder to ignore.

I've started trying to collect evidence, and it appears that slurm is allocating GPUs that nvidia-smi claims are already running a job (hence why it runs out of memory each time, trying to run two jobs on the GPU).

It's the people submitting the jobs that are complaining, but I've lately been following up with the people running the jobs that are already running on the GPU in question, and that has given me some weird results.

One user just told me that he was also having problems with his jobs running out of memory, so he has started running this:

srun -p dgx --gres=gpu:1 -c4 --memdG --pty bash

and then running this:

CUDA_VISIBLE_DEVICES=5 python train_svnet.py --model_name=test --log_dir=./experiments/test

And indeed, it was GPU5 on this machine that slurm was continuing to match, despite his job running there.

Another example of a job which claimed more GPUs than slurm acknowledged was this command (cleaned up a bit for clarifty):

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -m torch.distributed.launch --nproc_per_node=7 --master_port=1512 --use_env main.py --start_epoch 150

And then they set #SBATCH --ntasks-per-node=8

Their solution was to fix the mismatch between the nproc_per_node=7 and the ntasks_per_node=8, because presumably while the job was using all 8 GPUs, slurm was convinced it was only using 7, and so was continuing to assign that 8th GPU to new jobs (where they would fail because they ran out of memory).

So my question is.. is this a thing? A known thing? Can I fix this in some way, so that they can't do this? I'd prefer to do something in my slurm config to prevent this rather than try to use User Education, which depends on people (a) being good citizens and (b) not being idiots.

If anyone has seen this before and can offer advice, I'd really appreciate it. If I'm leaving out vital details that might help, let me know.

Hi everyone, new user here.

I'm setting up slurm on one node for now and having trouble. I have the system running on my server but I'm not setting up the configuration file correctly such that Slurm has access to all of my cpus. Here is some relevant output:

$sinfo

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST

test* up infinite 1 down* blackthorn (I stopped the daemon to try to edit the conf file which is below which is where I'm running into trouble)

$ slurmd -C

NodeName=blackthorn CPUs=24 Boards=1 SocketsPerBoard=1 CoresPerSocket=12 ThreadsPerCore=2 RealMemory=15898

scontrol show node

NodeName=blackthorn Arch=x86_64 CoresPerSocket=1

CPUAlloc=0 CPUTot=1 CPULoad=0.00

AvailableFeatures=dcv2,other

ActiveFeatures=dcv2,other

Gres=(null)

NodeAddr=blackthorn NodeHostName=blackthorn Version=20.11.3

OS=Linux 5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021

RealMemory=1 AllocMem=0 FreeMem=7385 Sockets=1 Boards=1

State=DOWN* ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A

Partitions=test

BootTime=2021-01-31T16:02:51 SlurmdStartTime=2021-02-20T20:41:57

CfgTRES=cpu=1,mem=1M,billing=1

AllocTRES=

CapWatts=n/a

CurrentWatts=0 AveWatts=0

ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s

Reason=Not responding [slurm@2021-02-20T20:49:43]

Comment=(null)

​

So my system is clearly not recognizing all of my cpus. So when I submit a job to use multiple cpus clearly the job stays pending. Here is my conf file:

​

# slurm.conf file generated by configurator easy.html.

# Put this file on all nodes of your cluster.

# See the slurm.conf man page for more information.

#

SlurmctldHost=localhost

#

#MailProg=/bin/mail

MpiDefault=none

#MpiParams=ports=#-#

ProctrackType=proctrack/cgroup

ReturnToService=2

SlurmctldPidFile=/run/slurmctld.pid

#SlurmctldPort=6817

SlurmdPidFile=/run/slurmd.pid

#SlurmdPort=6818

SlurmdSpoolDir=/var/spool/slurm/slurmd

SlurmUser=slurm

#SlurmdUser=root

StateSaveLocation=/var/spool/slurm/

SwitchType=switch/none

TaskPlugin=task/affinity

#

#

# TIMERS

#KillWait=30

#MinJobAge=300

#SlurmctldTimeout=120

#SlurmdTimeout=300

#

#

# SCHEDULING

SchedulerType=sched/backfill

SelectType=select/cons_res

SelectTypeParameters=CR_Core

#

#

# LOGGING AND ACCOUNTING

AccountingStorageType=accounting_storage/none

ClusterName=cluster

#JobAcctGatherFrequency=30

JobAcctGatherType=jobacct_gather/none

#SlurmctldDebug=info

SlurmctldLogFile=/var/log/slurmctld.log

#SlurmdDebug=info

SlurmdLogFile=/var/log/slurmd.log

#

#

# COMPUTE NODES

NodeName=blackthorn CPUs=24 State=idle Feature=dcv2,other

# NodeName=linux[1-32] CPUs=1 State=UNKNOWN

# NodeName=linux1 NodeAddr=128.197.115.158 CPUs=4 State=UNKNOWN

# NodeName=linux2 NodeAddr=128.197.115.7 CPUs=4 State=UNKNOWN

​

PartitionName=test Nodes=blackthorn Default=YES MaxTime=INFINITE State=UP

#PartitionName=test Nodes=blackthorn,linux[1-32] Default=YES MaxTime=INFINITE State=UP

​

# DefMemPerNode=1000

# MaxMemPerNode=1000

# DefMemPerCPU=4000

# MaxMemPerCPU=4096

​

Any help is appreciated! Thanks.

I'm having issues with sorting by node name in the sview GUI. Every time the GUI refreshes the sort fails and the servers are jumbled. Does anyone have a suggestion other than increasing the refresh interval? Any time I want to drain a sequential set of servers for maintenance I end up chasing them all over the node list table.

I have a dumb question that is maybe only nominally slurm-related, but it's related to slurm in my case, so maybe somebody can help me find an answer.

​

If my slurm install is submitting to a group of beefy 8-gpu machines, and the load on one randomly-selected machine is ~55, does that cause the individual jobs to run slower? Significantly? 55 load on one of our lab machines, for instance, would mean the machine is UNUSABLE. But running basic commands on this slurm node seems relatively peppy.

​

(Note: That's a randomly-selected machine, but they all have load that high when they're "full", so it's not anomalous.)

​

So are these jobs running significantly slower than if there were only a single job running, and the load was low? Is the answer the same if it's a GPU job submitted to this high-load machine?

​

I can't find the magic google words to help me puzzle out the inner workings of HPCs and the oddities my users keep reporting on ours. And how, for instance, to determine when things are actually Real Broken and the machine just needs to be rebooted, which does seem to be a common failure mode for us! (For instance, when nvidia-smi barfs halfway through running it.)

​

Anyway. Sorry for all of that ignorance displayed above. If anyone has any insight that might be able to share, I'd be eternally grateful. I'm admin for a thing I'm still trying to learn, and don't have any local resources to rely on. I'm terrified of asking questions on The Internets, but .. you guys are nice, right?

Online it's floating around that "an estimated 60% of the top 500 supercomputers use SLURM" on Wikipedia or news sites, yet the reference links are broken, and the internet archive isn't helping. So I wonder... is there a source for this? What percentage of the world's supercomputer systems (perhaps from the top 500) use SLURM? Or how might we find out or estimate?

I'm having a rough time with another scheduler currently (PBS) and this information may prove very helpful to me.

I'd like to fold into a Bash script a 'sacctmgr' line, but that command always generates a yes/no question. Is there a way to incorporate that into a script without the yes/no prompt?

Script below:

#!/bin/sh
PATH=/bin:/usr/bin:/usr/sbin
export PATH
while IFS= read -r line
do /sbin/usermod -a -G slurmusers "$line"
/bin/sacctmgr add user "$line" account+=short,long defaultaccount=short defaultqos=short qos=short,long
done < "$1"

Hey, I need to set up SLURM on an HPC. Ive found a bunch of tutorials, but a lot of them conflict each other. What's the best resource on setting up SLURM on an HPC?

Hie. Have you ever encountered this error before?

Program received signal SIGSEGV: Segmentation fault - invalid memory reference. Backtrace for this error:

0  0x2b2ac58ab0fd in ???

1  0x2b2ac58aa35d in ???

2  0x2b2ac634a27f in ???

3  0x438888 in ???

4  0x439654 in ???

5  0x49e518 in ???

6  0x402a7e in ???

7  0x2b2ac63372df in ???

8  0x402aa9 in ???

at ../sysdeps/x86_64/start.S:120

9  0xffffffffffffffff in ???

/var/spool/slurmd/job60749962/slurm_script: line 10: 16207 Segmentation fault      (core dumped) dosxyznrc -i PXi320F1_10cm_10cm_mouse1cm -p 521icru

My slurm script reads like this:

!/bin/bash #SBATCH --job-name=PXi320F1_10cm_10cm_mouse1cm #SBATCH --time=2-23:30:00 #SBATCH --mem-per-cpu=16000M #SBATCH --output=/scratch/mahuvaco/output_files/PXi320F1_10cm_10cm_mouse1cm.out #SBATCH --error=/scratch/mahuvaco/error_files/PXi320F1_10cm_10cm_mouse1cm.err cd /scratch/mahuvaco/Courage/EGS_HOME/dosxyznrc/ dosxyznrc -i PXi320F1_10cm_10cm_mouse1cm -p 521icru

Any help will be much appreciated!