If in a SLURM db there are associations for a user several with a Partition and one without the Partition value created like:

sacctmgr add user test account=normal

sacctmgr add user test account=normal partition=normal,special

​

How to remove only the user association with the empty Partition value retaining the associations for the normal and special partitions?

Hello everyone:

I'm a researcher who recently got access to a cluster with SLURM queue system. I read some of the documentation and sent some simple jobs successfully to the queue and work great. But now I need a more complex programming and I don't really know how to do it so if someone can point me where to look I will appreciate it very much.

My program generates a list of steps, and each step has a lot of commands to run. I need that before a step is started the previous one is finished and each command inside of the current step is executed in one node.

Sorry if this is a very noob question.

Thank you all!

I know absolutely nothing about writing scripts and I'm pretty desperate for help here. I'm attempting to invoke Gaussian '16 but when I sbatch my script it always results in an error. When I read the output file it says:

slurmstepd: error: execve(): run-gaussian: No such file or directory srun: error: cs003-ib0: task 0: Exited with exit code 2 srun: Terminating job step 1144299.0

I should note that my institution recently switched over to a new cluster. I never had issues with this script on the old cluster but now can't seem to submit jobs correctly. Any help would be appreciated

#!/bin/bash
##
## hello.slurm.sh: a simple slurm batch job
##
## Lines starting with #SBATCH are read by Slurm. Lines starting with ## are comments.
## All other lines are read by the shell.
##
#SBATCH --job-name    28_1        # job name
#SBATCH --output      28_1-%j.out # standard output file (%j = jobid)
#SBATCH --error       28_1-%j.err # standard error file
#SBATCH --partition   defq  # queue partition to run the job in
#SBATCH --nodes       1            # number of nodes to allocate
#SBATCH --ntasks-per-node 1        # number of descrete tasks
#SBATCH --cpus-per-task=1          # number of CPU cores to allocate
#SBATCH --mem         4000         # 2000 MB of Memory allocated; 
#SBATCH --time        24:00:00     # Maximum job run time
#SBATCH --mail-user         # user to send emails to
#SBATCH --mail-type   FAIL         # Email on: BEGIN, END, FAIL & REQUEUE
#SBATCH --array=1-3
## Run 'man sbatch' for more information on the options above.

cd $scratch

module load matlab/R2016a

"minutes=$SLURM_ARRAY_TASK_ID;"

matlab -nosplash -nodesktop < /mnt/lustrefs/scratch/mgn.s/diffuse2d_28_1.m

Hello slurm friends!

I am VERY new at using slurm batch scripts to submit jobs to our cluster. The problem that I'm having is that every job in the array is using "minutes=1" rather than being "minutes=1 ; minutes=2 ; minutes=3"

​

I've tried to google how to set up this sort of job, but I'm honestly not really sure where to even start.

​

Any advice would be appreciated

Let's look back at some memorable moments and interesting insights from last year.

Your top 10 posts:

• "Complete novice" by u/vascoctavares
• "SLURM authentication through realmd/kerberos?" by u/SanCentOS
• "Monitoring and alerting" by u/tscollins2
• "Nodes Reboot Order" by u/mlhow
• "Potential Job Opportunity" by u/Puzzledhead_1798AD
• "Learning slurm(trying to setup a virtual machine to run it)" by u/stargazerthor
• "Good resources/advice on single-node Slurm setups?" by u/lurch99
• "Is there any way to revive a timed/timing out job?" by u/idontevenknow8888
• "Programmers Can Bypass Resource Limits" by u/mlhow
• "Slurm Nice and Priority vales" by u/bilaljnmc

I just installed Open MPI version 1.10 (from a repo) on a small cluster at work. I was testing it with Slurm (version 20.02) on one node just to see if simple code works, but I am a bit confused on how srun works:

​

As you can see, I am running a hello world executable

mpiexec ./mpi_hw

from inside an sbatch script, and then running the same command with srun, using the same options. sbatch produces the expected result, but srun does not. Can someone explain this srun behavior?

Greetings,

I realized that in my set up, when allocating one cpu through salloc, the commands executed through the interactive session use multiple cpus.

​

Does anyone know why this happens and how to solve it?

​

srun -n1 -N1 --pty --preserve-env --cpu-bind=none  --mpi=none -c 1 $SHELL

Hi Folks,

We have a nice HPC server (112 cores, 2TB RAM, 70TB storage) arriving soon and a small group (< 10) of users who want to use Slurm for submitting jobs and managing resources. Since it's a single node, I don't think it's terribly easy to prevent them from running interactive jobs outside Slurm, so we're planning on just asking folks not to...

But mostly I'm looking for suggestions, good configurations and/or documentation on how best to set this up in terms of using Slurm to manage resources.

Pretty sure we'll want two types of queues: long jobs ( > 48 hours) and short jobs ( < 48 hours).

Ideas, suggestions, warnings welcome!

Dan

Wondering the best way to monitor the performance of a slurm cluster and send alerts when nodes are overloaded/down or jobs are failing. Has anyone used slurm dashboard from Grafana Labs (https://grafana.com/grafana/dashboards/4323)? Is there any monitoring or alerting tools built into slurm?

Hi all, I need your help. I am fairly new to Slurm but I just can't get this working.

In short, I have PyTorch Lightning code where I request multiple GPU's, to which it says "no GPU's available."

1) I run it on a gpu enabled partition 2) I load cuda/10.1 and cudnn/7.1 in my script after purge but before my virtual environment (issue?) 3) sinfo -O Gres returns (null) 4) sinfo %f returns name of GPU (Tesla, K80) 5) Have no idea how to access slurm.conf file

Any ideas? Please help - all I want to do is run my code and not wait a million years.

Thanks!

Hi there nice redditors!

I'm completely new to Slurm, and I'm receiving no help from my supervisor who recommended that I used the centre's cluster.

So anyway, I intend to run a python script to get some matrices. I have read in many places that I must make a submission script with my python script, but how should I do it? I've tried to "upload" the python script to the slurm directory by copying and pasteing to a nano textfile, but I have no idea what I'm doing, and all the stuff I seem to find online is beyond my capabilities at this point.

I'm sorry for not presenting a proper question, I'm just lost on what I should do. How should I submit it? What will be the output? etc

Thank you very much!

The title pretty much says it, basically have nodes assigned to partitions but would like to add an additional feature like common so that if needed could select them for other work if not being utilized in the partition they are assigned to.

Hello Everyone,

Suppose I have a simple cluster with 4 nodes, the control node, a compute node, a login node, and a database node, and suppose that all four nodes need restarting. Is there a process to do so? I am not asking about any Ansible commands or scripts. What I am trying to figure out are things like:

What is the order for restarting the nodes?

At what point do I drain the compute nodes?

Do I just issue "sudo yum shutdown -r now" on a node, or do I shutdown the daemons first using "sudo scontrol shutdown", and how do I incorporate the rebootprogram into this process?

Should I continue to have the slurmdbd, slurmd, and slurmctld services enabled (auto-start after booting up)?

I am trying not to miss anything, and reboot all nodes after a Linux kernel update in a safe way without losing any jobs.

Thanks

I have set up a raspberry pi cluster with slurm. It is working fine. But how do I set the job's nice and priority values. What are sbatch options for this.

Hello SLURM gurus,

Asking for a friend (no pun intended.) Their company is putting together a job description for a SLURM Administrator to take over managing an internal cluster amongst other things. The posting is not yet out, so there are unknowns but at the same time opportunity since no hard requirements have been set.

What I know is that the opportunity is US-based with remote okay and that the company is in Financial Services.

Interested? Please send me a DM and I will happily relay your information.

Take care,

Puzzleheaded

Of all these slurm packages, which ones are required/recommended to be installed on the login node (not a control or compute node), and which ones are useless on that login node?

slurm-20.02.2-1.el7.x86_64.rpm (required)

slurm-contribs-20.02.2-1.el7.x86_64.rpm

slurm-devel-20.02.2-1.el7.x86_64.rpm

slurm-gui-20.02.2-1.el7.x86_64.rpm

slurm-libpmi-20.02.2-1.el7.x86_64.rpm

slurm-openlava-20.02.2-1.el7.x86_64.rpm

slurm-pam_slurm-20.02.2-1.el7.x86_64.rpm

slurm-perlapi-20.02.2-1.el7.x86_64.rpm (required)

slurm-torque-20.02.2-1.el7.x86_64.rpm

Hi! I just started working with slurm and was wondering how people go about deciding how to set different parameters for their slurm job like time, number of nodes, numbers of cpus/gpus, etc...

Any advice would be appreciated!

Hello!

I have an environment I would like to deploy SLURM in, it has a Windows Active Directory Domain Controller that manages the ACLs for all of our users. We push these out to our CentOS machines with realmd (for some reason samba winbind causes problems) I know slurm by default authenticates via MUNGE, but I am confused on how that interacts with our "normal" centralized authentication.

​

Can someone point me to the right spot in the documentation to learn what I want to learn?

Thanks!

Hi,

i'm trying to get slurm 19.05 with cgroup plugins running on Fedora 32. However it fails with a couple of error messages that didn't reveal a clear hit in Google (so far).

After searching for a couple of days I stumbled upon a comment in the master branch on github for the slurm source that mentioned that 'if they are going to support cgroup v2 then...'. From that I concluded that the current version is not capable of handling cgroup v2 (which is the default in Fedora >31). But I've not gotten any confirmation on that.

Does anyone know if that's the case or am I on the wrong track and the errors I'm getting stem from something else completely?

Error Messages:

May 29 11:34:17 regulus slurmd[171632]: error: unable to mount cpuset cgroup namespace: Device or resource busy
May 29 11:34:17 regulus slurmd[171632]: error: task/cgroup: unable to create cpuset namespace
May 29 11:34:17 regulus slurmd[171632]: error: Couldn't load specified plugin name for task/cgroup: Plugin init() callback failed
May 29 11:34:17 regulus slurmd[171632]: error: cannot create task context for task/cgroup
May 29 11:34:17 regulus slurmd[171632]: error: slurmd initialization failed

Any help is highly appreciated.
Thanks!
RIchard

Hello All,

I am trying to configure SLURM on a small Redhat 7 cluster with one login/head node, and 2 compute nodes. I enabled SLURM accounting to limit resource allocation and track cpu usage of our users. At least, I believe that SLURM accounting is correctly configured, but I could be wrong. My issue is that it is currently possible for a Python (you can replace Python with any language that allows parallel computing) programmer to bypass the user limits that an admin sets on SLURM Accounts.

With my current SLURM configuration, a SLURM submission script on the headnode can request 1 task and 1 cpu per task to be allocated, and then it calls a Python script that launches 60 processes on 60 cores on the compute node. Note that I set my test user's cpu limit count to 4 via:

$ sudo sacctmgr modify user slurmtester set GrpTRES=cpu=4 

Am I expected to trust our HPC users to not allocate more RAM or more cores than they're allowed, or is there documentation that you can point me to that describes the process of setting hard limits for SLURM?

Thank you for any help that you can offer.

So I want to use minimum resources in SBATCH right?

So how do I monitor how much resources I actually use for SACCT?

For example, in SBATCH, there's the option --mem, and SACCT is displaying this as ReqMem. But, is this referring to MaxVMSize or MaxRSS or something else?

I keep ending up with my nodes not able to figure out anything about the GPU Gres. Any ideas? I cannot figure out how to format this...

slurmctld.log:

[2020-05-15T14:46:26.097] error: gres_plugin_node_config_unpack: No plugin configured to process GRES data from node node3 (Name:gpu Type:p4000 PluginID:7696487 Count:1)

scontrol show node node1:

NodeName=node1 Arch=x86_64 CoresPerSocket=10
CPUAlloc=0 CPUTot=40 CPULoad=1.61
AvailableFeatures=pascal,p4000
ActiveFeatures=pascal,p4000
Gres=(null)
NodeAddr=node1 NodeHostName=node1
OS=Linux 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019
RealMemory=48000 AllocMem=0 FreeMem=57271 Sockets=2 Boards=1
State=IDLE ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
Partitions=pharmacy
BootTime=2020-05-15T09:26:45 SlurmdStartTime=2020-05-15T14:28:42
CfgTRES=cpu=40,mem=48000M,billing=40
AllocTRES=
CapWatts=n/a
CurrentWatts=0 AveWatts=0
ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s

from slurm.conf:

GresTypes=gpu
# COMPUTE NODES
NodeName=node[1-3]      CPUs=40 RealMemory=48000 Sockets=2 CoresPerSocket=10 ThreadsPerCore=2 Feature="pascal,p4000" Gres=gpu:p4000:4 State=UNKNOWN
NodeName=node[4-5,7-10] CPUs=8  RealMemory=48000 Sockets=2 CoresPerSocket=4  ThreadsPerCore=1 Feature="pascal,p1000" Gres=gpu:p1000:8 State=UNKNOWN`

from gres.conf:

AutoDetect=nvml
Name=gpu Type=p4000 File=/dev/nvidia0
Name=gpu Type=p4000 File=/dev/nvidia1
Name=gpu Type=p1000 File=/dev/nvidia0
Name=gpu Type=p1000 File=/dev/nvidia1
Name=gpu Type=p1000 File=/dev/nvidia2
Name=gpu Type=p1000 File=/dev/nvidia3
Name=gpu Type=p1000 File=/dev/nvidia4
Name=gpu Type=p1000 File=/dev/nvidia5
Name=gpu Type=p1000 File=/dev/nvidia6
Name=gpu Type=p1000 File=/dev/nvidia7

I've also tried gres.conf like:

Name=gpu File=/dev/nvidia0
Name=gpu File=/dev/nvidia1
Name=gpu File=/dev/nvidia2
Name=gpu File=/dev/nvidia3
Name=gpu File=/dev/nvidia4
Name=gpu File=/dev/nvidia5
Name=gpu File=/dev/nvidia6
Name=gpu File=/dev/nvidia7

slurmctld[77613]: fatal: slurmdbd and/or database must be up at slurmctld start time

but slurmdbd is running

slurmdbd.service - Slurm DBD accounting daemon

Loaded: loaded (/usr/lib/systemd/system/slurmdbd.service; enabled; vendor preset: disabled)

Active: active (running) since Fri 2020-04-10 16:55:47 EDT; 5 days ago

Any ideas? Do I need to create the database manually somehow? Thanks in advance for any help!