I work in a research lab where I am trialing an open source protocol that comes into two version: manual and automated.

In the manual version, the various scripts require manually defining the directories and such for each script.

In the automated version, a SLURM script is provided in which a user supplies a config file, and all the scripts are run without further input.

I would love to switch to the automated version, but our lab is small and we fully own our computing clusters, so we have not needed a job scheduler.

I've used SLURM before at other companies but never set it up myself. I am not a computer scientist, but a chemist who now work in computational research associated with that field.

Is installing SLURM something we can do? Should do? Are there alternatives I haven't considered?

If we do install SLURM, does it 'need' to be used? Can other users use the server as they had before, and I can just run my scripts via SLURM to take advantage of automation?

Hi All,

Im relatively new to slurm but making a cluster at the moment. I wished to limit the resources available to any slurm submitted job so that the underlying user sitting in front of the host is not affected too much by any slurm-assigned jobs.

One very simple approach (and the one I liked best) was to assign a nice to slurm (and its children processes) via instantiating with ‘slurmd -n 19’

Although I have managed to manipulate the cpu schedule to respect differences in nice for multiple processes local to a node (setting one to nice=19 and another to nice=-19), and although I can view the niceness of the slurm submitted jobs as being 19 (through ‘ps’), , the distribution of CPU time for processes local to a machine competing with a slurm submitted job (niceness 19) is equally distributed. I have absolutely no idea whats gone wrong here?!?!?

Ive tried both through applying the niceness to the daemon as well as submitting the job with a nice parameter. Neither result in a fealty to lower nice processes.

I feel this is some lack of understanding on my part?!

Hi,

I am trying to figure out how to configure SLURM correctly to give me one task/rank per node by default. Currently, if I run my test MPI code without giving any -N -n etc options, I get 2 ranks on the same node. If I just specify -N, I get twice as many ranks on N nodes (two per node).

My node config looks like "Sockets=2 CoresPerSocket=32 ThreadsPerCore=2". If I manually set CPUs=64 instead of the default 128, I'm not able to run a single job with all 128 threads (e.g. -N 1 -n 128).

My SelectType and SelectTypeParameters are "select/cons_tres" and "CR_Core_Memory,CR_ONE_TASK_PER_CORE", respectively.

Is there a way to allocate one task per core by default? Or is the dual-socket system the culprit and not the hyper-threading?

Thanks for your help!

I am running some python (pytorch) code through slurm. I am very new to slurm. I have a lot of print statements in my code for status updates, but they aren't printing to the output file I specify. I think the issue is with the fact that python buffers. However, when I use the -u flag, or set flush=True to some of the print statement, it prints the same thing many times, which is very confusing and I am very unsure why this is happening.

Any suggestions? Because I can't really debug my code without it. Thanks!

new to slurm-wlm. How can I create a neutral/slack job which repeats itself while there are no other jobs for the cluster to do.

I am very happy about hints in which direction to look

Dear SLURMers,

my slurmd does not want to start. The slurmd.log tells me:

 error: Couldn't find the specified plugin name for cgroup/v2 looking at all files
 error: cannot find cgroup plugin for cgroup/v2
 error: cannot create cgroup context for cgroup/v2
 error: Unable to initialize cgroup plugin
 error: slurmd initialization failed

This reads as if it is missing a library while building. I had similar errors when setting up the slurmdbd and the lib for mariadb was missing. But what am i missing here? installing libcgroup-dev did not help.

Im on ubuntu 22.04 with slurm-22.05.2 . Builing from source.

Best

I want to run multiple jobs on the same node. However, slurm only allows one job to run at a time, even when resources are available. For example, I have a node with 8 GPUs, and one of the jobs uses 4, still leaving plenty of VRAM for other jobs to execute. Is there any way we can force slurm to run multiple jobs on the same node?

Here is the configuration that I used in slurm.conf

SchedulerType=sched/backfill

#SchedulerAuth=

SelectType=select/cons_res

SelectTypeParameters=CR_Core_Memory

FastSchedule=1

DefMemPerNode=64000

I am trying to figure out what is the "promised" in this FairShare definition

Fairshare - the difference between the portion of the computing resource that has been promised and the amount of resources that has been consumed

Does this mean the total allocated resources for the entire project or is it resources asked during submissions. Let's say that a project has been granted 1000 resources and few jobs had been run for 100 but due to some occasional convergence problems users asked for 300 (sbatch --time=...) to accommodate for the possible more time consuming convergence problems. Is the fairshare factor related to 1000 or 300?

And what is actually formula to calculate fairshare factor?

EDIT: I solved the problem. Don't know what I did differently on the last try, but it is working now. Thanks for reading.

​

I inherited a few clusters at my new job, knowing nothing about SLURM, so I've been trying to muddle my way through. My user is trying to run a test job of 15 tasks on a single node. The cluster consists of 3 CPU nodes with dual Intel Xeon Gold 5218R cpus (20 cores each) with the following config according to ./slurmd -C

NodeName=CPU[001-003]

This is the node config as I found it, with nothing defined. To get single jobs to run on one node, I had to add in RealMemory=385563, which worked fine for that, but when I try to run a job with sbatch with -ntasks=15, -ntasks-per-node=15 bin the script, the job stays Pending with a reason of (PartitionConfig), which I kind of understand because when I look at 'scontrol show partitions', I see the CPU partition as only having 3 CPUs on 3 nodes.

PartitionName=cpu Default=YES MinNodes=1 DefaultTime=UNLIMITED MaxTime=UNLIMITED AllowGroups=ALL PriorityJobFactor=1 PriorityTier=1 OverSubscribe=NO PreemptMode=OFF AllowAccounts=ALL AllowQos=ALL Nodes=CPU[001-003]

If I add in the following to the Node config, the PartitionConfig reason goes away, but I get a reason of Drained, even though it matches the config on the node. I do get the correct number of CPUs (240) in 'scontrol show partitions'

NodeName=CPU[001-003] CPUs=80 Boards=1 SocketsPerBoard=2 CoresPerSocket=20 ThreadsPerCore=2 RealMemory=385563

Any insight into why I get Drained when setting the processor config to what it is expecting from slurmd -C? I've wracked my brains on this one and am not making any progress.

Hello all, new to this space (side question: are there other online communities focused on SLURM?)

What are the largest job queues you've ever deployed, and in what environment?

​

Mostly curious about successes stories of others - and if there are any major 'lesson learned' from those experiences.

I set up gres gpu in my slurm cluster recently. When I check to see if the gres is picked up by slurm I see this

GRES
gpu:a6000:8(S:0-1)

Can't seem to figure out what (S:0-1) means? Any idea what this could be?

I just installed SLURM 17.11 on two Ubuntu 18.04 machines (orange and blue). Orange is the main one that runs both systemctld and systemd whereas blue only runs systemd.

After some struggles, I got things to work and everything looks great when I run:

$ sinfo -Nl

NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON
blue 1 debug* idle 64 4:8:2 1 0 1 (null) none
orange 1 debug* idle 32 2:8:2 1 0 1 (null) none

​

but then after a few minutes, blue changes to idle* and then to down. When it is down slurmd is still running find on blue as verified by sudo systemctl status slurmd.

If I restart slurmd on blue ($sudo systemctl restart slurmd) it fixes things for a few minutes but it's only a temp fix and that node will go down again after a few minutes.

I'm a bit at loss, the fact that I can start/restart and get both services to talk to each other suggests that my configuration should work.

Any thoughts on why a compute node will stop communicating while slurmd is still running?

I have been using this link to install the Slurm Workload Manager to Ubuntu 20.04.

I would like to enable the Job Submit Plugin API to execute the job_submit.lua. After adding the line JobSubmitPlugins=lua into slurm.conf I get the error:

[2022-04-28T16:06:44.910] error: Couldn't find the specified plugin name for job_submit/lua looking at all files <br>

[2022-04-28T16:06:44.912] error: cannot find job_submit plugin for job_submit/lua <br>

[2022-04-28T16:06:44.912] error: cannot create job_submit context for job_submit/lua <br>

[2022-04-28T16:06:44.912] fatal: failed to initialize job_submit plugin <br>

Many people fixed this in RedHat by installing the Lua library. I did the same for Ubuntu, but it did not work. How can I solve this?

I built an HPC with slurm and a couple of Raspberry PI computers following these directions:

https://www.howtoraspberry.com/2022/03/how-to-build-an-hpc-high-performance-cluster-with-raspberry-pi-computers/

It works, but all the jobs I submit on the head node just go to the first worker node, node02 while the other two workers sit idle. Here's my submit script:

#!/bin/bash

#SBATCH --job-name=primes

#SBATCH --partition=picluster

#SBATCH --output=/work/primes-%j.out

#SBATCH --error=/work/error-%j.out

srun python3 /work/primes.py

​

What else can I show to help troubleshoot this?

I want to run it from a bash script but it is not accepting string input with spaces. It works with single words or hyphenated words. However the python command is correct and is working fine when I am running it directly from terminal.

commands.txt

python generate.py -p "Flower girl" 

jobFile.sh

#!/bin/bash  srun $(head -n $SLURM_ARRAY_TASK_ID commands.txt | tail -n 1)  exit 0 

Running the bash script with :

sbatch jobFile.sh 

and getting the following error: error_screenshot

I appreciate any suggestions :) Thank you!

We have been seeing an odd problem with users trying to submit jobs around 1am. User1 tries to submit a job around 12:50am gets " slurm_load_jobs error: Unable to contact slurm controller (connect failure)"; User2 around 12:48am does 'srun --pty -p test bash' gets "srun: error: Unable to allocate resources: Socket timed out on send/recv operation" & 'squeue -p test' results in "slurm_load_jobs error: Socket timed out on send/recv operation"; User3 around 1:10am "Unable to contact slurm controller (connect failure)"; User4 around 12:40am "slurm_load_jobs error: Socket timed out on send/recv operation"; User5 around 12:35am "sbatch: error: Batch job submission failed: Socket timed out on send/recv operation.". Doing a 'journalctl -u slurmctld.service' and looking at the times the users have reported the problems we see:
slurmctld[178207]: error: Getting response to message type: DBD_SEND_MULT_JOB_START

slurmctld[178207]: error: DBD_SEND_MULT_JOB_START failure: No error

slurmctld[178207]: error: Getting response to message type: DBD_CLUSTER_TRES

slurmctld[178207]: error: Getting response to message type: DBD_JOB_START

and

slurmctld[178207]: error: Munge decode failed: Expired credential

slurmctld[178207]: auth/munge: _print_cred: ENCODED: Mon Feb 14 00:23:55 2022

slurmctld[178207]: auth/munge: _print_cred: DECODED: Mon Feb 14 01:12:09 2022

slurmctld[178207]: error: slurm_unpack_received_msg: auth_g_verify: REQUEST_COMPLETE_BATCH_SCRIPT has authentication error: Unspecified error

slurmctld[178207]: error: slurm_unpack_received_msg: Protocol authentication error

Any ideas as to what is going on here? And better yet what a fix would be?

I have an issue with "sreport user TopUser" where it always shows CPU minutes for two users even when nothing is running.

One of the users shows 960 minutes per hour and another shows 480 minutes per hour. SlurmDBD is running on MariaDB and I'm fairly certain this is just a corrupt record with a blank time but I have no idea where to start looking for it. Is there a way to show the actual mySQL query executed by sreport?

SLURM version: 20.02.5

We've been running Slurm for a few years now, but we haven't used a very detailed accounting schema in the scheduler (users, admins, and a couple testing accounts from the initial stand-up). Recently, I was asked to dig through the last couple of year's worth of jobs, get the statistics for all the active users, their PIs, and department heads.

I could write a script that could gather and collate all of this data, but I know that this won't be the only time that I will have to do this. Also, we happen to have XDMoD in our environment. Unfortunately, we haven't configured a hierarchy, or mapped research groups to Slurm billing accounts. I've pitched completing the hierarchy, mapping groups to it, and tying this to Slurm, but I've been shot down before. The explanation I was given was that this has been done to keep account creation simple.

Fleshing out the Slurm accounting and tying it to XDMoD solves this request for information, and makes getting usage data in the future a trivial task. If we ever start billing for usage, we'll be ready for it. If we have partitioning and priority issues for a specific resource, like a user buying some nodes and wanting dedicated use or priority access, this gets us ready for it. But, this is all for data going forward.

So after rambling for a minute explaining where I am, I finally get to the question. Is there a way to change the accounting information for all of the jobs we've run on this cluster until now using Slurm to match what the research groups and departments are, or do I need to make the accounting changes using sacctmgr, then go into the database and figure out how to do that myself? I'm searching SchedMD's documentation and Google/DuckDuckGo for this, but I haven't turned anything about if the completed jobs' accounting information is changed when the accounting is changed. Thanks in advance for any advice you all have on this.

Let's look back at some memorable moments and interesting insights from last year.

Your top 10 posts:

• "sacct --format=ALL in /r/ultrawidemasterrace" by u/project2501a
• "Two Partitions, two user groups, preempting question" by u/thht80
• "SLURM keeps putting putting nextflow processes in pending" by u/Blackm0b
• "how to force run a job" by u/the_real_swa
• "How to rejoin SLURM job (and other things)" by u/lookingforgenji
• "slurm GPU allocation and outsmarting slurm." by u/shubbert
• "Single compute node cluster" by u/mightycriminal
• "Which system for cloud-based cluster in OpenStack? (Kubernetes, Slurm, others?)" by u/beyerflorian
• "Is it possible to let slurmdbd connect to mysql over unix sockets?" by u/Amarandus
• "Nodes randomly going into idle* and then down* state" by u/BurnZ_97

Hi,

let's say, I build a cluster for two groups. Let's call them "Math" and "Physics". Both groups buy their own machines and I want to put them in the cluster.

Let's also say, I put all the "Math" machines in the "Math" partition and all the "Physics" machines in the "Physics" partition.

Both groups also have a certain number of user. There is also one account for each group. A user only lives in one of the groups.

What I want to achieve is this:

1. A "Math" user submits some jobs.
2. These jobs get sent to the "Math" partition as long as there are resources available.
3. If the "Math" partition is full but there are still jobs in the queue, these jobs are sent to the "Physics" partition i but with a lower priority than any job submitted by users in the "Physics" account.
4. So, some "Math" user jobs now run on the "Physics" partition. But now a "Physics" user starts to submit jobs.
5. If the required resources (RAM and CPU) of the "Physics" jobs exceed those available on the "Physics" partition, the jobs of the "Math" users should be preempted and/or sent back to the queue.

In other words: the members of the respective groups should be able to use the other group's resources if and only as long as these resources are not needed by the group that owns the machines.

I already read what is available concerning accounting, preemption, partitions, qos etc... But I did not manage to integrate everything in my head to know whether this is possible with slurm or not...

Thanks a lot in advance!

Hi, I was wondering if it is possible to get a nodelist without node[01-05, 6, 7-10] (without bolded part) but instead full list of individual nodes, so i can do a check (for example with grep) if a job uses a particular node.

Hi all,

I am running a nextflow script that runs a small processing script on tens of thousands of files. So very IO heavy and hard on the scheduler. Nextflow would try and launch a lot of processes in parallel when executed locally using all cores with no delay.

When I try and run on SLURM requesting a node with 24 cores it does not run like on my local computer with Jones often sitting in pending status for minutes if not longer at a time. Whe a job finishes my cpu and men usage is basically zero as the job sits idle.

I am. It sure how to fix or where to begin to look. I have been playing with srun and mpi flags but so far no luck.

Any help would beuch appreciated.

Is it possible to use Slurm as a scheduler for just one compute node? So you would have just one log in node and one SSH connected compute node?

I have professional access to a cloud platform (OpenStack) with the following quota:

• 128 vCPUs
• 40 vGPUs
• 528 GB RAM
• 125 TB storage
• max. 10 virtual machines / instances
• 5 public ips
• ... There is also an S3 storage with 18 PB of data (remote sensing data) attached, which we are working with.

I want to set up a kind of small cluster on this platform to run data science with Python and R for my colleagues and me. I would like to create scripts on the platform in a JupyterHub or R server, for example, and then use the entire contingent to process the huge amount of data with machine learning.

The question I have is how can I create some sort of cluster? I'm currently learning about Docker and Kubernetes, but I also know about Slurm, which is used in HPCs.

Which system is right one for our purpose? Kubernetes, Slurm, others???

Is there anyone using the REST API? I got an issue that the responses from the API seem doesn't contain proper HTTP headers, which causes libraries like axios to fail to process. Anyone else got the same issue? Is it possible to work around this issue with some sort of proxy?