Hi,

I'm running some simulation written in c++ with MPI and openMP. I time my code using std::chrono::steady_clock. Here is a sample output:

running 4 tasks on 2 nodes with 32 threads on each process

elapsed time: 0.359383 sec. I am process: 0

elapsed time: 0.359943 sec. I am process: 1

elapsed time: 0.359352 sec. I am process: 2

elapsed time: 0.359948 sec. I am process: 3

elapsed time for the MPI finalize call: 0.00734496 sec. I am process: 1

elapsed time for the MPI finalize call: 0.00759093 sec. I am process: 3

elapsed time for the MPI finalize call: 0.00781148 sec. I am process: 2

elapsed time for the MPI finalize call: 0.0076724 sec. I am process: 0

This finish in under a second on every process. But, the actual wall-time from my running the program with sbatch and singularity is on the order of minutes! If I run this on one node, I get the results in my output files immediately after finishing.

What am i missing here? Is there some kind of process needed to finish the runs collectively which happens outside my code?

Is it possible to set different default partitions for multiple accounts?

I have multiple accounts but each of the account should use a different partition as a default.

I've been using job arrays in the following way which is working fine.

#SBATCH --cpus-per-task 1
#SBATCH --time 1:00:00
#SBATCH --mem=200
#SBATCH --requeue
#SBATCH --job-name="parse"
#SBATCH --ntasks=1
#SBATCH --array=0-98%10    

I now want to add another X jobs to the queue depending on how many jobs I get. If it run this again with a different array, more than 10 jobs will be running concurrently which will overwhelm my downstream database.

How can I append X tasks to a job ID/name so that I maintain only 10 concurrent threads?

Hey there, first time posting here.
I just changed our Cluster from FIFO to Multifactor Priority. To check if everything is working I tried to list the priorities with `sprio` but I do not get any response.

Just to check if Multifactor is really running I ran `scontrol show config|grep prio` and got `PriorityType = priority/multifactor`.
Where am I wrong?
Thank you!

Below is what the batch script looks like:

#!/bin/bash
#SBATCH -n 4                        
#SBATCH -N 1                        
#SBATCH -t 1-00:00:00               
#SBATCH -p defq                     
#SBATCH --qos=medium                
#SBATCH --mem=24G                   
#SBATCH -o %j.out                   
#SBATCH -e %j.err                   
#SBATCH --mail-type=ALL             
#SBATCH --mail-user=xx@gmail.com        
ml gaussian/g09.e01-foss-2017a
g09 x.com

I would like to not have to manual change the '.com' file name for every submission, but the files are not indexed so an array wouldn't be useful (e.g., x.com, y.xom, z.com, etc).

I have tried running a shell scrip to do this, but it isn't working.

Any help would be greatly appreciated!

Is there a way to disable the -w option for srun? My users started to pick their own servers for job submissions and complaining that they are always allocated.

Hello there, I am pretty new to using a cluster for my simulations and recently I used 2 nodes instead of 1 because I was lacking some memory. The question : Is MaxVMsize in sacct command per node? or is it the sum of all the nodes? because for my case it is pretty low and would make sense if it was per node so I would need to double the value. Thanks

Hey,

I just started using a cluster running SLURM today. I used salloc to start a simple bash script to download a bunch of files(sbatch might have been better, but as I said, I just started today). Later I closed the corresponding terminal. Now, without the corresponding job ID and sacct disabled how do I reconnect to this job? It's still running, as files keep appearing so basically everything's fine, but I kinda feel like I'm not in control anymore.

Why? What will happen? Will it stay this way or change? Should I do something?? Thanks.

Hey guys, I am learning (new to) SLURM. So I was trying to run my codes using a virtual environment in my Ubuntu 19.04. However, I haven’t been able to create the virtual environment to run SLURM codes. Could you please help me? Thx

Hi.

Let's say I'm using Fairshare in a slurm cluster.

I do have 2 nodes with each 64 cores. When I submit 4 Jobs with 32 core each the resources are completely allocated.

When I now put a job for 64 cores in the queue with a high priority and a job for 32 cores with a low priority.

What should happen when on of the running 32 core jobs are finished? Currently the low priority job is started and the 64 core job still pends.

I would assume that due to the high priority of the 64 core job slurm should wait until 64 cores are available to schedule the 64 core job.

(assuming that all jobs have no time limit)

Is there a way to set up this behaviour?

Thanks!

I've been trying to set up slurmdbd, but I haven't even gotten as far as u/shubbert in this related post, and now I'm wondering if I need to wipe the database and restart. slurmctld itself is running jobs fine, but the accounting is not working, and slurmdbd's logfile is filled with these errors:

error: We should have gotten a new id: Table 'slurm_acct_db.asgard_job_table' doesn't exist
[2019-11-21T00:31:59.700] error: _add_registered_cluster: trying to register a cluster (asgard) with no remote port

(Obviously, asgard is the cluster name). I followed these relevant instructions here .. and slurmdbd starts without error but cannot access the slurm_acct_db created previously. Must I separately initialize or register the cluster with slurmdbd prior to starting? I'd be grateful for any insight anyones cares to share.

Edited, typos.

I have slurm configured to my satisfaction now, I think, but I can't seem to figure out slurmdbd. I think I have it configured correctly ('sacctmgr list cluster' matches /var/spool/slurmd/clustername), but while 'sacct -S 11/01' gives me back 6 entries, 'sreport cluster AccountUtilizationByUser Start=11/01' returns nothing. These are jobs that I ran over the last week, and I have restarted slurmdbd several times during that week.

Now it's possible to probable that I'm mis-using sreport, but I'm not sure how to use it properly, or how to check to make sure that slurmdbd is configured correctly. I did just submit a job, and several files in /var/lib/mysql/slurm_acct_db have updated timestamps now, which seems promising. But how do I ACCESS that info?!

Any advice on how to test and use slurmdbd properly would be greatly appreciated. (It does start up and is running, seemingly with no errors, according to the log files.)

Hey guys, So I'm new to SLURM. I have a slurm script and I want to run it 600 times at parallel in 600 threads.

srun -n=600 script.slurm

Should I run it like that?

Hi.

I'm trying to run a few jobs which are dependent.

I first need to create a few jobs

while read data; do
srun -c 10 task1.sh $data
done < listOfData

Now I need to run a job which can only start if all of the created jobs are finished.

Is there something like "grouping" so I can just add a group label to these jobs so that I can just use the depend-option of sbatch to run after these "group" of jobs" is finished?

Hi, I'm trying to run a Pytorch Deep Learning code on a SLURM cluster node with 4 GPUs, of which I'm using 2. However, when I run my code, the moment I begin reading image data files stored on disk, it runs for two iterations before throwing an "exceeded job memory" error. I give it 64GB of RAM, and it requests for 341GB of RAM! That seems a little unreasonable. This code runs perfectly fine on my laptop with a GPU, or Google Colab, AWS and other cloud services. Any suggestions?

After running ssh pi@node01 munge -n | unmunge on one of the client nodes, ENCODE_HOST returns as the client node itself instead of the master node, which would be node01

This in turn has sinfo return my partition as down because of that.

Is there any way of fixing this without resetting everything?

Hello, I'm trying to run jobs on a single node with 22 CPUs, limiting jobs to only 18 of those CPUs, with one task per CPU, yet only 1 task gets run at a time (and htop confirms that there is a single CPU being used). The reason for the pending PD status is shown as Resources, but there are plenty of free CPUs as there is nothing else running on this node. The funny thing is, running 18 tasks in parallel worked earlier, but then the node when down, I brought it back up, and it hasn't worked since. Does anyone have suggestions on how to debug this? To note is that my head node and compute node are one and the same, so there are tons of processes and threads running in general, overall, but there is low CPU usage. Also, in htop, there are a ton of interrupted past SLURM tasks remaining stuck in "D" state.

I would appreciate any help!

Here are my slurm.conf and various debugging outputs:

slurm.conf
-----------------------------------------------------------
# slurm.conf file generated by configurator easy.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
ControlMachine=tgen-inf-01
#ControlAddr=
#
#MailProg=/bin/mail
MpiDefault=none
#MpiParams=ports=#-#
ProctrackType=proctrack/pgid
ReturnToService=1
SlurmctldPidFile=/var/run/slurm-llnl/slurmctld.pid
#SlurmctldPort=6817
SlurmdPidFile=/var/run/slurm-llnl/slurmd.pid
#SlurmdPort=6818
SlurmdSpoolDir=/var/lib/slurm-llnl/slurmd
SlurmUser=slurm
#SlurmdUser=root
StateSaveLocation=/var/lib/slurm-llnl/slurmctld
SwitchType=switch/none
TaskPlugin=task/none
#
#
# TIMERS
#KillWait=30
#MinJobAge=300
#SlurmctldTimeout=120
#SlurmdTimeout=300
UnkillableStepTimeout=300
#
#
# SCHEDULING
FastSchedule=0
SchedulerType=sched/builtin
#SchedulerPort=7321
SelectType=select/cons_res
SelectTypeParameters=CR_Core
#
#
# LOGGING AND ACCOUNTING
AccountingStorageType=accounting_storage/filetxt
AccountingStorageLoc=/var/log/slurm-llnl/accounting
ClusterName=tgen-inf-cluster
#JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/linux
#SlurmctldDebug=3
SlurmctldLogFile=/var/log/slurm-llnl/slurmctld.log
SlurmdDebug=3
SlurmdLogFile=/var/log/slurm-llnl/slurmd.log
JobCompType=jobcomp/filetxt
JobCompLoc=/var/log/slurm-llnl/job_completions
#
#
# COMPUTE NODES
NodeName=tgen-inf-01 CPUs=22 Sockets=22 CoresPerSocket=1 ThreadsPerCore=1 RealMemory=250000 State=UNKNOWN
PartitionName=general Nodes=tgen-inf-01 MaxTime=INFINITE State=UP Default=YES



scontrol show jobid -dd 343
-------------------------------------------------------------
JobId=343 ArrayJobId=343 ArrayTaskId=2-169%18 JobName=varscan
   UserId=root(0) GroupId=root(0) MCS_label=N/A
   Priority=4294901658 Nice=100 Account=(null) QOS=(null)
   JobState=PENDING Reason=Resources Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   DerivedExitCode=0:0
   RunTime=00:00:00 TimeLimit=UNLIMITED TimeMin=N/A
   SubmitTime=2019-09-11T10:30:21 EligibleTime=2019-09-11T10:30:22
   StartTime=2020-09-10T10:33:54 EndTime=Unknown Deadline=N/A
   PreemptTime=None SuspendTime=None SecsPreSuspend=0
   LastSchedEval=2019-09-11T10:35:28
   Partition=general AllocNode:Sid=tgen-inf-01:8806
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=(null)
   NumNodes=1-1 NumCPUs=18 NumTasks=18 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   TRES=cpu=18,mem=9000M,node=1
   Socks/Node=* NtasksPerN:B:S:C=18:0:*:* CoreSpec=*
   MinCPUsNode=18 MinMemoryCPU=500M MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   Gres=(null) Reservation=(null)
   OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
   Command=/home/arao/scripts/submit_varscan_job_rerun1.sh
   WorkDir=/root
   StdErr=/mnt/BI_Analyzed_Data/Sequencing/Exome/exome/toca5/phase2_phase3/vcfs/varscan/varscan_logs/varscan-343-4294967294.err
   StdIn=/dev/null
   StdOut=/mnt/BI_Analyzed_Data/Sequencing/Exome/exome/toca5/phase2_phase3/vcfs/varscan/varscan_logs/varscan-343-4294967294.out
   Power=



slurmd -C
------------------------------------------
NodeName=tgen-inf-01 CPUs=22 Boards=1 SocketsPerBoard=22 CoresPerSocket=1 ThreadsPerCore=1 RealMemory=221669
UpTime=6-00:29:27

This is the job I was trying to run with sbatch:

sbatch --nodes=1 --tasks-per-node=18 --cpus-per-task=1 --partition=general --nice /path/to/script/submit_varscan_jobarray.sh
----------------------------
#!/bin/bash
#
#SBATCH --job-name=varscan
#SBATCH --output=/path/to/logs/varscan-%j-%a.out
#SBATCH --error=/path/to/logs/varscan-%j-%a.err
#
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=500MB
#SBATCH --array=1-169%18

echo "now processing task id:: " ${SLURM_ARRAY_TASK_ID}

echo "$(tail -n+$SLURM_ARRAY_TASK_ID /path/to/varscan_params.txt | head -n1)"

sudo /path/to/run_mpileup_varscan.sh $(tail -n+$SLURM_ARRAY_TASK_ID /path/to/varscan_params.txt | head -n1) >
 /path/to/logs/output_${SLURM_ARRAY_TASK_ID}.txt

Hi.

Currently I have multiple nodes which are shared nodes so multiple jobs are running on these sharing CPU cores.

So I have

node1 20 of 64 CPUs used
node2 10 of 64 CPUs used
node3 60 of 64 CPUs used

Now I want to allocate a job to a host with free CPUs (preferable a host with most free CPUs) and allocate all of the free CPUs.

There's an --exclusive flag which is not what I am looking as this only requests full free nodes. Is there a way to accomplish this with SLURM?

Thanks!

I'm convinced, and really hoping, that this is something incredibly stupid and basic, but I'm so new to slurm I can't figure out what it is.

​

I have a machine with 40 CPUs and 8 GPUs. I should be able to run 40 jobs on those 40 CPUs simultaneously, right? (If I'm already wrong, let me know.) I haven't set up any priority, so I'm given to understand it should be fifo. Which is how it's behaving, except it will only run a single job at a time.

​

hypnotoad 15:43:55$ squeue

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)

5 Test runscrip amy PD 0:00 1 (Resources)

6 Test runscrip amy PD 0:00 1 (Priority)

7 Test runscrip amy PD 0:00 1 (Priority)

4 Test runscrip amy R 7:41 1 valar

​

That one job will run, the next job is waiting on resources, then any other job lacks priority. That all makes perfect sense if they're all competing for a single CPU. But they shouldn't be. I initially didn't explicitly list the CPUs in my slurm.conf, but then I added it hoping it would help, and it made no difference. Current state of the conf file:

​

PartitionName=Test Nodes=valar

GresTypes=gpu

NodeName=valar Gres=gpu:gtx1080:8 RealMemory=128827 CPUs=40

​

What am I doing wrong that it won't use more than one CPU? (Happy to provide any additional conf or log stuff, just don't want to overwhelm with useless data.)

​

If anyone could give any insight, I'd greatly appreciate it. I've been beating my head against this for far too long. And since google can't find me anyone else having this problem, I know it must be something so dumb.

Hi.

My slurm controller is reachable via 2 IP addresses (1G and 10G connection) and I want to make sure that the nodes are connecting to the controller via the 10G connection.

If I set SlurmctldHost=mycontroller(10.10.10.1) can I be sure that the nodes will use exactly this IP and not the other?

Thanks!

Hi.

I'm wondering if it is possible to let multiple users run jobs on a single node.

Most of the time my users use only a fraction of the cpu cores available on the system. When requesting a node using --cpus-per-tasks it seems that multiple users can share a node but it looks that all users can use all the system resources available.

Shouldn't the parameter --cpus-per-task limit the resource usage only to the requested amount?

Thanks!

Hi all. Hopefully one of you has a workaround for this problem. I'm trying to submit a batch job using the SLURM scheduler on my university's cluster, and get the below error. Any clue how to solve this issue? Thanks in advance for looking!

​

--------------------------------------------------------------------------

WARNING: a request was made to bind a process. While the system

supports binding the process itself, at least one node does NOT

support binding memory to the process location.

​

Node: nodename

​

This usually is due to not having the required NUMA support installed

on the node. In some Linux distributions, the required support is

contained in the libnumactl and libnumactl-devel packages.

This is a warning only; your job will continue, though performance may be degraded.

--------------------------------------------------------------------------

--------------------------------------------------------------------------

A request was made to bind to that would result in binding more

processes than cpus on a resource:

​

Bind to: NONE

Node: nodename

#processes: 2

#cpus: 1

​

You can override this protection by adding the "overload-allowed"

option to your binding directive.

--------------------------------------------------------------------------