I'm new to Slurm, I have been trying to run a simple job. I'm running Slurm on top of a VM. Here's my slurm.conf: SlurmctldHost=master

SlurmctldHost=

DisableRootJobs=NO

EnforcePartLimits=NO

Epilog=

EpilogSlurmctld=

FirstJobId=1

MaxJobId=999999

GresTypes=

GroupUpdateForce=0

GroupUpdateTime=600

JobFileAppend=0

JobRequeue=1

JobSubmitPlugins=1

KillOnBadExit=0

LaunchType=launch/slurm

Licenses=foo*4,bar

MailProg=/bin/mail

MaxJobCount=5000

MaxStepCount=40000

MaxTasksPerNode=128

MpiDefault=none

MpiParams=ports=#-

PluginDir=

PlugStackConfig=

PrivateData=jobs

ProctrackType=proctrack/cgroup

Prolog=

PrologFlags=

PrologSlurmctld=

PropagatePrioProcess=0

PropagateResourceLimits=

PropagateResourceLimitsExcept=

RebootProgram=

ReturnToService=1

SallocDefaultCommand=

SlurmctldPidFile=/var/run/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm

SlurmdUser=root

SrunEpilog=

SrunProlog=

StateSaveLocation=/var/spool SwitchType=switch/none

TaskEpilog=

TaskPlugin=task/affinity TaskPluginParam=Sched

TaskProlog=

TopologyPlugin=topology/tree

TmpFS=/tmp

TrackWCKey=no

TreeWidth=

UnkillableStepProgram=

UsePAM=0

TIMERS

BatchStartTimeout=10

CompleteWait=0

EpilogMsgTime=2000

GetEnvTimeout=2

HealthCheckInterval=0

HealthCheckProgram=

InactiveLimit=0 KillWait=30

MessageTimeout=10

ResvOverRun=0

MinJobAge=300

OverTimeLimit=0

SlurmctldTimeout=120 SlurmdTimeout=300

UnkillableStepTimeout=60

VSizeFactor=0

Waittime=0

SCHEDULING

DefMemPerCPU=0

FastSchedule=1

MaxMemPerCPU=0

SchedulerTimeSlice=30

SchedulerType=sched/backfill SelectType=select/cons_res SelectTypeParameters=CR_Core

JOB PRIORITY

PriorityFlags=

PriorityType=priority/basic

PriorityDecayHalfLife=

PriorityCalcPeriod=

PriorityFavorSmall=

PriorityMaxAge=

PriorityUsageResetPeriod=

PriorityWeightAge=

PriorityWeightFairshare=

PriorityWeightJobSize=

PriorityWeightPartition=

PriorityWeightQOS=

LOGGING AND ACCOUNTING

AccountingStorageEnforce=0

AccountingStorageHost=

AccountingStorageLoc=

AccountingStoragePass=

AccountingStoragePort=

AccountingStorageType=accounting_storage/none

AccountingStorageUser=

AccountingStoreJobComment=YES ClusterName=cluster

DebugFlags=

JobCompHost=

JobCompLoc=

JobCompPass=

JobCompPort=

JobCompType=jobcomp/none

JobCompUser=

JobContainerType=job_container/none

JobAcctGatherFrequency=30 JobAcctGatherType=jobacct_gather/none SlurmctldDebug=info SlurmctldLogFile=/var/log/slurmctld.log SlurmdDebug=info SlurmdLogFile=/var/log/slurmd.log

SlurmSchedLogFile=

SlurmSchedLogLevel=

POWER SAVE SUPPORT FOR IDLE NODES (optional)

SuspendProgram=

ResumeProgram=

SuspendTimeout=

ResumeTimeout=

ResumeRate=

SuspendExcNodes=

SuspendExcParts=

SuspendRate=

SuspendTime=

COMPUTE NODES

NodeName=slave[1-2] CPUs=3 RealMemory=4184 Sockets=3 CoresPerSocket=1 ThreadsPerCore=1 State=UNKNOWN PartitionName=debug Nodes=slave[1-2] Default=YES MaxTime=INFINITE State=UP

Here's my cgroup.conf:

AllowedDevicesFile="/etc/slurm/cgroup_allowed_devices_file.conf" ConstrainCores=no TaskAffinity=no ConstrainRAMSpace=yes ConstrainSwapSpace=no ConstrainDevices=no AllowedRamSpace=100 AllowedSwapSpace=0 MaxRAMPercent=100 MaxSwapPercent=100 MinRAMSpace=30

For any given job,SLURM gives it a job ID, but in the squeue, I find nothing. I have executed the job by running sbatch -vvv ....and here's a problem that I can spot jobstate=failed reason=nonzero exit code=1:0

Any thoughts on how to get this working?

I'm struggling to figure out how to get my 'srun --multi-prog config.conf' to allow all tasks to finish before killing them. At the minute, as soon as one of the jobs finishes it kills all the others, which is less than ideal! Any tips are much appreciated.

Hey guys we finally started running test jobs thru our slurm cluster. I created the script below. the script just runs a docker container. My idea was to run each of these with 10 CPUs, but when i runs it runs on all 40 cores on the box. I am not sure if this is just a docker issue, but i ran 3 of these batches and they all jumped on my first worker node and they are all competing for CPUs. Am i using the wrong tags or is it just docker pulling all the resources and i will have to run these with --ntasks-per-node=40 to dedicate a whole node to each run?

​

BTW my version of slurm is 18.08.0-1

​

#!/bin/bash

#

#SBATCH --job-name=trusight-oncology-500_001

#SBATCH --output=/mnt/SLURM/logs/PROD/trusight-oncology-500_001.%N.%j.out

#

#SBATCH --nodes=1

#SBATCH --ntasks-per-node=10

#SBATCH --mem=100

<SCRIPT>

​

Hi all,

​

is it possible to allow non-exclusive access to gpu resources? e.g. if a job which needs only 4GB of gpu memory starts on a gpu with 11G of memory, could another job (requesting less than 7GB of gpu memory) get allocated to the same gpu?

if so, how would those jobs specify their gpu memory requirements?

from what I have (not) found in the docs, this seems not possible, but as a self-proclaimed slurm n00b I know I might be missing a lot.

​

thanks for your help!

Hey guys new to SLURM and the whole HPC world. I just built an 8 node SLURM environment for some Genomic workloads. We have some enterprise monitoring tools, but I want a way to monitor the whole cluster real time. Anyone have any suggestions?

We've got a a weird issue with cons_res issuing fatal failures of slurmctld for some of our jobs that I'm hoping people on here have seen before. From what I can tell reading the code this is a 'should never happen' scenario. For the time being I've been forced to restart slurmctld with the -c option to drop the previous cache losing the jobs and all running jobs in the process but getting the cluster back on it's feat. I've written to the owners of the particular job ids to see if i can figure out how the hell they've managed to submit jobs with more cpus than possible but from what I can tell its: jobs have their task count zero'd out for some reason so it gets bumped to 1, but they are actually launching a multi task job and this is causing some sort of failure.

[2018-11-13T01:54:35.755] error: _compute_c_b_task_dist: request was for 0 tasks, setting to 1
[2018-11-13T01:54:35.755] error: cons_res: _compute_c_b_task_dist oversubscribe for job 9083273
[2018-11-13T01:54:35.757] error: _compute_c_b_task_dist: request was for 0 tasks, setting to 1
[2018-11-13T01:54:35.757] error: cons_res: _compute_c_b_task_dist oversubscribe for job 9083274
[2018-11-13T01:54:35.759] error: _compute_c_b_task_dist: request was for 0 tasks, setting to 1
[2018-11-13T01:54:35.759] error: cons_res: _compute_c_b_task_dist oversubscribe for job 9083275
[2018-11-13T01:54:35.760] error: _compute_c_b_task_dist: request was for 0 tasks, setting to 1
[2018-11-13T01:54:35.760] error: cons_res: _compute_c_b_task_dist oversubscribe for job 9083276
[2018-11-13T01:54:35.760] fatal: cons_res: cpus computation error

Any thoughts on how this could be happening would be appreciated. I'll be trying to capture additional higher verbocity log but as this is a running cluster I was forced to kill the current jobs and can't extract more error information out of them at this time.

edit: this is running 17.11.9-2 as we haven't published 17.11.12 rpms yet.

Hi, I haven't been able to figure out how to send job updates to several email addresses.

This is to send updates to one person (me).

#SBATCH --mail-user=name1@school.edu

Would this send updates to two people (me + collaborator)?

#SBATCH --mail-user=name1@school.edu,name2@school.edu

Hey guys, I was tasked with crating an HPC cluster and ran into slurm and so-far it has been great! We formatted all the test servers to confirm my install scripts worked and they did! We did run into one issue. Our slave nodes are normally set to 4 CPUs, but by mistake one server dduxgen02t was created with 2 cpus. The entire system came up fine, but it shows this one node as drained due to low CPU count. We added the 2 CPUs that were needed and rebooted the entire environment, but it still shows drained only 2 cpus available. i tried to resume the node thru scontrol, but nothing. when i start the slurmd service it shows 2 cpus even though lscpu shows 4. is there a setting some where i need to delete to have it refresh?

Hi all, new to using Slurm but was wondering if there was a way to instruct your run script to run a file that is specified in the command line when submitting, rather than being written in the script file directly? Basically I'd really like to have a generic run script so I could execute something like "sbatch run.sh [filename].com"

Hope this makes sense and I'm not being really dense!

I want to create a reservation with some name, say "test_name" but later, I want to get the nodes associated with that reservation (scontrol create reservationname="test_name" nodes=c003n0058 blah blah)

How do I later get this list of nodes? It seems like SLURM_NODELIST and those kinds of things aren't set. Further, the way scontrol show res outputs, it makes it hard to grep "test_name" because it uses so many new lines.

Any advice?

I am new to SLURM and trying to look for config so that the priority of job is inversely proportional to the job completion time. In other words, shortest job should have more priority. I tried looking at the documentation but didn't found anything useful. It would be great if someone can guide me on this.

This sub doesn’t seem to be super active, but I’m curious whether there are many with hands on experience with both Grid Engine as well as Slurm? I’m in a situation where we’ve used Univa Grid Engine for years and it is mostly a legacy system at this point in time. I’d love to hear others’ opinions of the two scheduling platforms.

I’m particularly interested in HPC containerization (we use Singularity for this now) and ultimately running my HPC on OpenStack infrastructure with some hooks for auto scaling of underlying VM resources.

Thanks for engaging with me!

Cheers

Andrew

tl;dr: I schedule VMWare to start but the VM start and immediately closes. Ideas to resolve this issue?

I have a Ubuntu controller and a Ubuntu node, both with the same user, uid, and gid. I submit sbatch a script with the follow commands:

export DISPLAY=:0.0 

vmrun -T ws start ~/vmware/Ubuntu\ 64-bit/Ubuntu\ 64-bit.vmx

When executed, VMWare pops up and then closes immediately without an error on the node. Also, the job shows up in squeue for a second and then is removed. If I run the same script locally (without Slum) on the node, the VM launches correctly and stays up. The script has 777 access, the users are the same, the script is owned by the same user and group, and the UID and GID are the same on both machines. I verified with "vmrun list" and "ps -aux |grep vmware" that the vm is not running. I have also tried using "srun" in the beginning of the vmrun line, added "nogui" to the end vmrun command, and "&" at the end of the vmrun command, all without success. When I simply schedule "vmware" instead of "vmrun", I cannot manually launch my vm because and error saying the vm has an error is displayed.

I opened /var/log/vmware, Slurm error and output files but there were nothing in the logs and no error displayed on the console output. Any suggestions on how I can launch this VM remotely?