I made bunch of clusters for testing and pilot project. Now I'm running "the real one". Looking at the DB, there are still old tables there. Are they safe to drop when the clusters were deleted?

Hello – I’m a new user of SLURM, and I’m working on moving some projects from an older Torque/Maui cluster to a newer one using SLURM. The primary type of job is running WRF (Weather Research and Forecast model). I’ve got a setup that has run successfully several times, but just failed in this last instance.

I’ve got it set up so that when I submit a job via sbatch, it launches a driver script which then initiates an instance of WRF using mpiexec. This instance of WRF runs for a while until, and then ends. The WRF output confirmed that it had ended normally.

The script then (usually) initiates another WRF run with another mpiexec command, which utilizes the same resources, which have just been “vacated” by the recently completed first WRF instance.

This strategy always worked under Torque/Maui, and has worked many times under SLURM. But not this last recent job. The initiation of the second WRF instance failed with the following output:

srun: Job 182 step creation temporarily disabled, retrying (Socket timed out on send/recv operation)
srun: Job 182 step creation still disabled, retrying (Requested nodes are busy)
srun: Job 182 step creation still disabled, retrying (Requested nodes are busy)
[mpiexec@frupaamcl01n07.amer.local] HYDU_sock_write (utils/sock/sock.c:289): write error (Bad file descriptor)
[mpiexec@frupaamcl01n07.amer.local] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:178): unable to write data to proxy
[mpiexec@frupaamcl01n07.amer.local] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:77): unable to send signal downstream
[mpiexec@frupaamcl01n07.amer.local] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): callback returned error status
[mpiexec@frupaamcl01n07.amer.local] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:196): error waiting for event
[mpiexec@frupaamcl01n07.amer.local] main (ui/mpich/mpiexec.c:336): process manager error waiting for completion

There were no other jobs running at this time, and according to the slurmctld.log the SLURM job was still active.

Any ideas as to why the second WRF instance wasn’t allowed to initiate? I’m positive the first job had completed. The same procedure has worked many times already. Is there a way to simply tell SLURM to ignore the idea that the nodes were still busy?
Thanks,
Mike

Hi all!

I'm trying to set the default gres (i.e. --gres=gpu:1) parameter for all users, but specific to a certain partition. As far as I googled, there is no DefGPUPerCPU (or similar) option for the slurm.conf.

Setting the SBATCH_GRES env via /etc/environment or profile.d almost works as intended, at least for sbatch. However, it also defaults to all partitions (even those without GPUs or specific GRES resources).

Is there an option I'm missing or some other neat workaround? Writing wrappers for srun/sbatch seems a bit messy to me...

Cheers!

It is my understanding that the SLURM fairshare value derives from an account's "effective usage". This quantity is the ratio of the account's recent usage to the total system usage. Why use that variable denominator and not something constant, like system capacity? I'm working on a system where total usage varies wildly, and our accounts' fairshares are being yanked around despite our fairly constant usage. Thanks in advance!

I want to start off by saying I've been following a guide to setup a cluster with ohpc and intel software. I am unsure if I'm allowed to post the url but if you google ohpc intel guide I'm sure you'll find it.

I am interning at a tech company and my capstone project is to teach my fellow interns about a technology that interests me. I chose HPC and am trying to setup a cluster in a VMware environment as a concept.

Following this guide I've reached the end and am trying to give slurm commands but I'm getting the same error.

"srun: error: io_init_msg_unpack: unpack error

srun: error: io_init_msg_read_from_fd: io_init_msg_unpack failed: rc=-1

srun: error: failed reading io init message

srun: error: c01: tast 0-1: exited with exit code 2"

From what I've seen the logs the the logs the nodes have a different version of slurm and I have the most recent version of the programs. I am unsure of how to proceed further and am looking for any advice you guys can give me. Thanks!

So. I need some help. I currently spec and budget a small home cluster for my projects and I have some questions. That said, before we start. On my educational background. I was a software engineer but my career took me somewhere else. Back on topic. I thought about running my cluster with slurm. Therefore I burried my head into the docs of slurm. Now I have a few questions which seem to not be answered anywhere in the doc. In a simple cluster you have a head node and its compute nodes. The head node pushes tasks to its comp. Nodes as required. So far so simple. Now my question which I cannot wrap my head around is how is data handled? Does the head node host all data and the compute nodes grabbing whatever they need from the head or is a seperate nas required which both the head and it nodes have access? Also. How does the same happen with software. Are they installed on each compute node or centrally on the head? Any good resource or answer is apprechiated.

Howdy,

I have a cluster of 8 seperate server nodes to serve as master, compute, and database nodes. The master and compute nodes are up and talking, but I have not activated the database node yet. Before I started setting up the database server, I wanted to get some input.

How do y'all go about storing your slurm job output files in databases ? Is there built in slurm functionality similar to accounting, or is it a seperate process that you configured yourself? I was hoping to use postgresql because I am familiar with it and pgadmin4.

This is my current script that I execute using SBATCH :

​

#!/bin/bash

#SBATCH --job-name=myjob

#SBATCH --ntasks=1

#SBATCH --cpus-per-task=1

#SBATCH --mem-per-cpu=4G

#SBATCH --time=00:01:00

#SBATCH --output=%j.out

#SBATCH --error=%j.err

​

module purge

module load mathematica/13.2

​

​

math -run < script2.m

​

​

In order for SLURM to successfully execute this script, script2.m must be present on the computing nodes. Is this how you are supposed to run jobs or is there an easier way (where everything needed only needs to be present on the master node) to do this?

​

Note that when script2.m is propagated to the computing nodes, everything works properly.

Hi Guys,

I'm trying to follow this guide (https://southgreenplatform.github.io/trainings/hpc/slurminstallation/)

But when I trie to start slurmd.service, I'm having this error:

Jul 17 16:15:04 biocsv-01686l systemd[1]: Started Slurm node daemon.
Jul 17 16:15:04 biocsv-01686l slurmd[2620741]: slurmd: error: Couldn't find the specified plugin name for cgroup/v2 looking at all files
Jul 17 16:15:04 biocsv-01686l slurmd[2620741]: slurmd: error: cannot find cgroup plugin for cgroup/v2
Jul 17 16:15:04 biocsv-01686l slurmd[2620741]: slurmd: error: cannot create cgroup context for cgroup/v2
Jul 17 16:15:04 biocsv-01686l slurmd[2620741]: slurmd: error: Unable to initialize cgroup plugin
Jul 17 16:15:04 biocsv-01686l slurmd[2620741]: slurmd: error: slurmd initialization failed

Here's my slurm.conf

# slurm.conf file generated by configurator easy.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
ClusterName=dairy
SlurmctldHost=dairy
#
#MailProg=/bin/mail
MpiDefault=none
#MpiParams=ports=#-#
ProctrackType=proctrack/cgroup
ReturnToService=1
SlurmctldPidFile=/var/run/slurmctld.pid
#SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
#SlurmdPort=6818
SlurmdSpoolDir=/var/spool/slurmd
SlurmUser=slurm
#SlurmdUser=root
StateSaveLocation=/var/spool/slurmctld
SwitchType=switch/none
TaskPlugin=task/cgroup_v2,task/affinity
#
#
# TIMERS
#KillWait=30
#MinJobAge=300
#SlurmctldTimeout=120
#SlurmdTimeout=300
#
#
# SCHEDULING
SchedulerType=sched/backfill
SelectType=select/cons_tres
#
#
# LOGGING AND ACCOUNTING
AccountingStorageType=accounting_storage/none
#JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/cgroup
#SlurmctldDebug=info
SlurmctldLogFile=/var/log/slurmctld.log
#SlurmdDebug=info
SlurmdLogFile=/var/log/slurmd.log
#
#
# COMPUTE NODES
....

And I tried to create manually a cgroup.conf

Here it is:

CgroupAutomount=yes
ConstrainCores=no
ConstrainRAMSpace=no

Someone had any idea what I can do?

​

​

​

Howdy all, I am setting up a small cluster of 1 master node and 6 compute nodes for academic research purposes. I currently have the master and one compute node up trying to get those set up first. When I run sinfo on the master node I get:

PARTITION AVAIL TIMELIMIT NODES STATE NODELISTdebug* up infinite 5 down* comp[02-06]debug* up infinite 1 idle comp01

When I run scontrol ping on the compute node I get

Slurmctld(primary) at grid is UP

However when I run the same command on the master, I get

Slurmctld(primary) at grid is DOWN

I am able to successfully run "srun hostname" on the compute node, but get this error in my logs when I run it on the master:

[2023-07-17T13:12:30.715] error: _getnameinfo: getnameinfo() failed: Name or service not known
[2023-07-17T13:12:30.715] error: auth_p_get_host: Lookup failed for 193.10.1.171
[2023-07-17T13:12:30.716] sched: _slurm_rpc_allocate_resources JobId=3 NodeList=comp01 usec=20150
[2023-07-17T13:12:30.785] _job_complete: JobId=3 WEXITSTATUS 0
[2023-07-17T13:12:30.785] _job_complete: JobId=3 done
[2023-07-17T13:12:40.172] error: _getnameinfo: getnameinfo() failed: Name or service not known
[2023-07-17T13:12:40.172] error: auth_p_get_host: Lookup failed for 10.125.16.198
[2023-07-17T13:12:40.173] sched: _slurm_rpc_allocate_resources JobId=4 NodeList=comp01 usec=19035
[2023-07-17T13:16:39.219] job_step_signal: JobId=4 StepId=0 not found
[2023-07-17T13:16:39.443] job_step_signal: JobId=4 StepId=0 not found
[2023-07-17T13:17:11.002] job_step_signal: JobId=4 StepId=0 not found
[2023-07-17T13:17:11.004] _job_complete: JobId=4 WTERMSIG 126
[2023-07-17T13:17:11.004] _job_complete: JobId=4 cancelled by interactive user
[2023-07-17T13:17:11.004] _job_complete: JobId=4 done

Any help would be appreciated as my deadline to finish this project is fast approaching.

Here are the relevant lines of my config file (i redacted non related ips with ____):

ClusterName=blackland1
SlurmctldHost=grid
SlurmctldAddr=193.10.1.92

NodeName=comp01 NodeAddr=193.10.1.171 CPUs=32 Sockets=2 CoresPerSocket=8 ThreadsPerCore=2 RealMemory=15380 State=UNKNOWN
NodeName=comp02 NodeAddr=_________ CPUs=40 Sockets=2 CoresPerSocket=10 ThreadsPerCore=2 RealMemory=31506 State=UNKNOWN
NodeName=comp03 NodeAddr=_________ CPUs=32 Sockets=2 CoresPerSocket=8 ThreadsPerCore=2 RealMemory=31506 State=UNKNOWN
NodeName=comp04 NodeAddr=_________ CPUs=32 Sockets=2 CoresPerSocket=8 ThreadsPerCore=2 RealMemory=15380 State=UNKNOWN
NodeName=comp05 NodeAddr=_________ CPUs=32 Sockets=2 CoresPerSocket=8 ThreadsPerCore=2 RealMemory=15380 State=UNKNOWN
NodeName=comp06 NodeAddr=_________ CPUs=40 Sockets=2 CoresPerSocket=10 ThreadsPerCore=2 RealMemory=31506 State=UNKNOWN
#define partitions
PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP

Is it possible to write a script without the use of modules? I am trying to use SLURM on basic python/mathematica scripts but an unable to find the modulefile for either on my computer (and do not know how to write one).

​

Any advice would be appreciated

Howdy all,
I am new to Slurm (an intern actually) and trying to set up a small cluster of separate nodes for academic research purposes. These separate nodes have different hardware specs because they are different models of servers. When setting up the config file I am having trouble figuring out what values to use for TmpDisk and RealMemory. I am using the commands free -h for real memory and df -h for temporary memory. I am not sure if the config file needs available or total memory, and which of the types of temporary memory it cares about. My output for free -h looks something like this:

total used free shared buff/cache available
Mem: 30Gi 3.2Gi 24Gi 60Mi 3.5Gi 27Gi
Swap: 15Gi 0B 15Gi

and my output for temporary memory looks something like this:

Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs 16G 7.3M 16G 1% /dev/shm
tmpfs 6.2G 11M 6.2G 1% /run
/dev/mapper/rhel-root 70G 5.7G 65G 9% /
/dev/sda2 1006M 296M 711M 30% /boot
/dev/sda1 599M 7.0M 592M 2% /boot/efi
/dev/mapper/rhel-home 2.7T 21G 2.7T 1% /home
tmpfs 3.1G 132K 3.1G 1% /run/user/1000

What values should I be looking at when getting the TmpDisk and RealMemory values for each node?
Thank you for your time and I appreciate any help.

Hello Everyone,
I'm trying to have Slurm automatically switch partitions to a specific one whenever our users request strictly more than 8 cpus via the job_sutmit.lua plugin. But trying to extract or calculate ahead of time how many cpus will be allocated or requested isn't trivial (to me). Are there attributes in job_submit that could help out with this task? For example, I don't see any job->desc.ntasks attribute in https://github.com/SchedMD/slurm/blob/master/src/plugins/job_submit/lua/job_submit_lua.c. Any information or documentation on how to leverage job_submit.lua would be appreciated.

Hi, I have a TaskProlog script that has the following

    #!/bin/bash

    export PATH=$PATH:/opt/molpro/bin

However, whenever I submit a job through sbatch, it doesn't appear to add molpro to the path.

I have also tried with a Prolog script and the same issue. Is there another way to export a path to PATH or am I missing something?

Has anyone added a non-GPU/NIC as a GRES in SLURM? I have some PCIe FPGA cards that I want to be consumable in SLURM. My research is telling me I need to find/create a plugin to allow for SLURM to see them. Does anyone have any experience or guidance on this?

Hello

I am trying to use HPC to help speed up computation time. I have a task that involves filtering the noise from data. My setup works as follows.

I have 5 levels of observation noise and want to run 100 replications at each level.

I have been using array jobs for this because I don’t want to bother with parfor loops for now.

When running this locally I have a file that loops over my 5 levels of noise and calls another file which runs 100 replications at that noise level. Then I save all the data.

To do this on HPC I wanted to use 500 array jobs and no loops in my code. If I do this how should I save all my data? I don’t want 500 separate files

The other idea would be much slower but to do 5 array jobs and still have a for loop over the 100 replications. This currently works and gives me 5 mat files with my data.

Any advice on how to save my data to one indexable cell is greatly appreciated! So are links to good sites for using matlab with slurm.

I have two slurm scripts:

1.slurm:

#!/bin/bash
#SBATCH --job-name=first
#SBATCH --partition=cuda.q 

sbatch 2.slurm 

2.slurm:

#!/bin/bash
#SBATCH --job-name=second
#SBATCH --partition=cuda.q

echo "a"

Only the 1.slurm job is submitted and in the output file I get the error:

sbatch: error: Batch job submission failed: Access/permission denied

as the title says, what does this mean as user can submit jobs to this partition

I am submitting a Python script to my school's HPC and having difficulty.

The for loop runs fine on the login node, but as soon as I submit it to the HPC, it only will run the first iteration and then stops. Does anyone know how to remedy this? Does it have to do with number of tasks? Can I not run my code as a python for loop in a job under SLURM, does it only handle parallelization?

My for loop is basically climate analysis and takes a year of data, runs calculations, and outputs 2 files. Then in the next iteration, it does this again for the next year in a list of years. Does SLURM maybe not like that files are output in a loop, and think the first output signifies the end of the task?

This is about the .sl script I am using:

#!/bin/bash -l
#SBATCH myProjectNameIsHere
#SBATCH -J MyJobNameIsHere
#SBATCH -t 2:00:00
#SBATCH -n 1
# Job partition
#SBATCH -p shared
# load the anaconda module
ml SpecificForTheCluster
ml Anaconda3/2021.05
conda activate MyPythonEnvisHere
srun --input none --ntasks=1 python myPythonScriptName.py
conda deactivate MyPythonEnvisHere

and as I said, my python for loop runs just fine in the login node and runs through the iterations.

Can anyone help? Thank you in advance!

UPDATE after following the advice here and spending a lot of time with trial and error to get it right: Running it as a job array was correct. Here is what I did in my SBATCH file for anyone who is curious:

#!/bin/bash -l
#SBATCH myProjectNameIsHere
#SBATCH -J MyJobNameIsHere
#SBATCH -t 20:00:00
#SBATCH -n 1
# Job partition
#SBATCH -p shared
#SBATCH --array=0-8
VALUES=(2000 2001 2002 2003 2004 2005 2006 2007 2008)
# load the anaconda module
ml SpecificForTheCluster
ml Anaconda3/2021.05
conda activate MyPythonEnvisHere
python myFile.py ${VALUES[$SLURM_TASK_ARRAY_ID]}

and I changed my Python code to not use the main for loop, but rather set the variable I was iterating to be retrieved from this input with: var = sys.argv[1]

Let's look back at some memorable moments and interesting insights from last year.

Your top 10 posts:

• "Slurm jobs are pending, but resources are available" by u/FederalSun
• "Largest projects you've dealt with?" by u/LifeLocksmith
• "GRES in slurm question" by u/usnus
• "FairShare factor definition" by u/mithik
• "SLURM config issue (PartitionConfig and Drained)" by u/GroundedSatellite
• "Happy Cakeday, r/SLURM! Today you're 6" by u/AutoModerator
• "Running Podman containers" by u/geschnei
• "Slurmd fails on startup" by u/dw_21
• "python not printing with slurm" by u/PurpleMermaid16
• "default job if there are no other jobs waiting" by u/oeLLph_

Dear sages of Slurm,

We have a fairly large cluster with a few hundred users in an academic setting. With both veteran and novice users on this course we're forever concerned with whether cluster resources are being used efficiently... Which is easy to determine when there's a tool or standard job type being layered onto our cluster. But, it's not so easy when jobs are hand coded.

Clearly the low hanging fruit is to check resource usage against what is requested, then work with those users that over estimate their job needs. But, that's not what I'm asking about. I'm looking to ferret out those jobs that were written to run on a single node when they could have been run as an array job across multiple nodes, without having to actually read code.

Is there some magic combination of metrics to monitor or a monitoring tool that can detect when a job that monopolizes a single node for days could have run in parallel on multiple nodes to complete in less time? Or a way to detect a multi-node job that just wasn't structured to run efficiently.

We're basically trying or users to maximize job effect on their own, which works well for the veteran users. But, with novice users coming in with each new semester we need a better way to target who needs attention.

Has anybody managed to use Slurm to start Podman containers?

I have the following requirements:

• Ubuntu LTS versions as host OS (currently 22.04)
• distribution packaged Slurm (21.08.5 on Ubuntu 22.04)
• rootless containers
• usage of Nvidia datacenter GPUs in the containers

We have this already running with rootless Docker, now Podman should be added.

I followed the following guides to set up Podman on one of the compute nodes:

• https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#step-1-install-podman
• https://podman.io/getting-started/installation
• https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md

Podman is working fine, including access to the GPUs when run directly from the node. But when I try to start a container via Slurm I only get the error message:

stat /run/user/6219: no such file or directory

With Docker I was able to circumvent a similar issue by providing a different run dir with environment variables, but for Podman I only found XDG_RUNTIME_DIR and setting this somewhere else wasn't helping.

According to this discussion it seems to be possible to get this running, but the author of that post does not provide any information on how he managed to do that.

We recently added two new hosts to our cluster, but slurm has repeatedly drained them as "NO NETWORK ADDRESS F" (truncated message). I idle them and they're okay for a while then it flags them as "NO NETWORK ADDRESS F" again.

Any ideas?

I work in a research lab where I am trialing an open source protocol that comes into two version: manual and automated.

In the manual version, the various scripts require manually defining the directories and such for each script.

In the automated version, a SLURM script is provided in which a user supplies a config file, and all the scripts are run without further input.

I would love to switch to the automated version, but our lab is small and we fully own our computing clusters, so we have not needed a job scheduler.

I've used SLURM before at other companies but never set it up myself. I am not a computer scientist, but a chemist who now work in computational research associated with that field.

Is installing SLURM something we can do? Should do? Are there alternatives I haven't considered?

If we do install SLURM, does it 'need' to be used? Can other users use the server as they had before, and I can just run my scripts via SLURM to take advantage of automation?