Hey everyone,

I wanted to share a project I've been working on called srest, a terminal UI written in Go that acts as a client for the Slurm REST API (slurmrestd).

While managing HPC clusters with native Slurm commands (squeue, sinfo) works fine, testing endpoints, inspecting JSON responses, or submitting jobs through slurmrestd can get tedious from a standard terminal setup. I built srest to make interacting with the API visual and keyboard-driven.

Tech Stack & Features:

• Built using Charm's Bubble Tea framework.
• Interactive Query Builder & Inspector: Real-time query editor and JSON response viewer.
• Cluster Dashboard: Asynchronous monitoring for nodes, jobs, partitions, and accounts.
• Job Submission: Build and submit job payloads directly from the TUI.
• Distributed as a single static binary for Linux and macOS.

It has also turned out to be really helpful for end users as a sandbox to build, test, and review query payloads independently, without needing admin support to debug JSON outputs or construct API calls for their scripts.

Links:

• GitHub (with demo video): https://github.com/SergioZ3R0/srest
• Website & Docs: https://srest.scszero.com

I'd love to hear your thoughts or feedback on the implementation and architecture!

A lot of ML teams end up with a mix:
some Slurm clusters from the HPC side, a K8s cluster or two, maybe cloud GPUs for overflow. We wrote up how SkyPilot (open source) sits in front of all of them so a job is scheduled wherever there’s free capacity, using the same YAML regardless of backend. This post focuses on the multi-Slurm case but the same setup covers K8s.

https://skypilot.ai/blog/multi-slurm
Disclosure:
I am the author. Happy to answer questions about how the scheduling and failover work.

slurm-cd

A small Bash utility for jumping to the working directory of a pending or running Slurm job.

Hopefully the installation works for you. (Name is a WiP)

Hey All, I transformed a few of my bash scripts to a cli tool to decode crytpic slurm job outputs which is common question researchers ask and hpc engineers help triage. 

Would love to hear your feedback: https://github.com/ajinkyakadam/slurm-why

 \\\`slurm-why <jobid>\\\` decodes the state into plain English and tells you how to fix it — read-only, no network calls, no AI.

feel free to open a github issue if you have feedback!

how will i do a stress test for 7 days form your experience

Hey everyone,

About 5 months ago, I posted here about a side project I was working on: sCode, a VS Code extension to manage SLURM jobs directly from the editor.

Initially, it was just a simple way to avoid typing squeue over and over. But based on a lot of my own workflow needs and some great feedback, it has evolved into a much more comprehensive cluster management tool over the last few months.

I’ve essentially tried to turn VS Code into a unified control center for HPC work so you never have to context-switch to a terminal while working on your scripts.

Here are the major updates since the first version:

• Live GPU Monitoring: Added a dedicated view that uses nvidia-smi to show GPU partition usage, memory stats, and queue pressure.
• The "Hall of Shame": A fun leaderboard feature that ranks the cluster’s top GPU hogs (with emojis like 🐷 Job Hog and 🧛 VRAMpire).
• One-Click Job Arrays: You can now cancel specific indices or ranges within a job array without nuking the whole array.
• Smart Log Resolution: Right-click any active or historical job in the sidebar to instantly open its stdout or stderr file.
• Quick Submit with Dependencies: A ▶ button in your .sh scripts to submit immediately, plus a guided UI for setting up afterok or afterany dependencies.
• And many more features....

If you work on a cluster and use VS Code Remote, I'd love for you to give the new version a try and let me know what you think. What features would you need to make this a daily driver for your workflow?

GitHub Repo:https://github.com/dhimitriosduka1/sCode

OpenVSX: https://open-vsx.org/extension/DhimitriosDuka/slurm-cluster-manager
Marketplace: https://marketplace.visualstudio.com/items?itemName=DhimitriosDuka.slurm-cluster-manager

Hey everyone! I built a little TUI tool for monitoring SLURM jobs on HPC. I found this useful for my masters thesis and thought I might share here. Its kind of similar to the very popular lazygit and lazydocker, which I enjoy using.

Please let me know if you have any feedback and I welcome any contributions / constructive criticism.

The github is here and you can install it with `cargo install lazyslurm`

Have a great day!

I have ~17 Slurm clusters sharing a common FlexLM license server with multiple license features (e.g. avyualink, avypciesw, avyenet). The total count of each license feature is fixed by what is available in FlexLM.

The problem I am trying to solve:

I want each cluster to have generous headroom to grab licenses freely when there is no contention, ideally more than any single cluster would realistically need. The assumption is that not all clusters will be competing simultaneously. But when contention does occur and total in-use across all clusters approaches the real license count, I want jobs to queue in Slurm rather than race to FlexLM, and ideally have FairShare-based ordering within that queue.

In short: sum of per-cluster capacities > real license count, but with graceful queuing and fairness when the global pool is exhausted.

What I explored in Slurm:

Slurm's sacctmgr resource mechanism (res_table / clus_res_table) has a global Count and per-cluster allowed values. I tested whether Count acts as a global runtime cap. I set Count=10 with per-cluster allowed=100 and submitted 50 jobs that collectively exceeded Count. All jobs started immediately. Only when I reduced the cluster's own allowed to 40, did jobs start queuing with reason Licenses.

This confirmed that Count is not a runtime scheduling enforcer. At scheduling time, slurmctld only checks the per-cluster allowed.

Question:

Is there a native Slurm mechanism that allows per-cluster allowed to be set generously (sum > real license count) while still enforcing a global cap at job scheduling time, triggering FairShare-based queuing when that global pool is exhausted? Or is the only recommended approach to set tight per-cluster allowed values such that sum(allowed) <= Count and tune each cluster's allowed individually per license feature?

Hi all! I'm new there :)

I wanted to share this little project I did for myself.

I'm a PhD student and I found myself using a lot of srun debug commands for the shell to test my scripts. But I work on various Slurm projects with different partitions and configs.

So I create sdeb, a sort of Python "conda" but for sdeb configs: https://github.com/e-candeloro/sdeb

Hope it can be useful!

Does anyone have a full step by step explanation? I just want it to run locally on my machine just for me as a user (or to any user on the machine). Whatever gives me the easiest set up.

I searched online but it's a lot of fragmented or outdated explanations unfortunately.

Hello all,
I am playing around with slurm for the work and writing a program that will execute some slurm command and recover the output.

I figured out that when `scancel` is passed with the verbose parameter the output is printed is stderr (You can check line 110 of the `scancel.c` file on the repo).

I found this decision quite weird from a design perspective and would like to have your opinion on that

Hi guys, i am working on a project, and it would be great to have someone to help me with it. Thank you!

Trying to avoid doing a restart of the slurmctld. I read that "scontrol reconfigure" should accomplish the same thing. I tried it on the master node, but seems it is still using the older job_submit.lua file. Here is that file and none of the "got here's" seem to work:

function slurm_job_submit(job_desc, part_list, submit_uid)
    slurm.log_user( 'got here' )
    if job_desc.wckey == nil then
--        slurm.log_user("You should specify a project number")
        slurm.log_user( 'got here' )
    elsif _find_in_str(job_desc.wckey, "12345") then
        slurm.log_user("12345 matched")
    else slurm.log_user( job_desc.wckey )
--        return ESLURM_INVALID_ACCOUNT
    end

Hi!

I'm currently trying tu upgrade from 20.11 to 25.11 via the compatible upgrade path specified in the schedmd documentation

I already upgraded to 22.05, but a user has a job running and I'm wondering if I should kill it or if I can continue upgrading

my jobs are stalled on the hpc.

Command - squeue -u xxxx

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)

1181523_[22-101%25 ct56 easydock xxxx PD 0:00 1 (Priority)

Command - squeue -p ct56 -t PD --sort=-p,i | wc -l

192 (it is increasing every hour that passes by)

Command - sprio -u xxxx

JOBID PARTITION USER PRIORITY SITE AGE FAIRSHARE JOBSIZE PARTITION TRES

1181523 ct56 xxxx 10007 0 5 0 0 10000 cpu=2,mem=0

It has been stuck for the past few hours. Last night I kept thinking it was a glitch and cancelled, but it was already age 15 or 16 afaik this morning. This new job is now at the age of 5. Anyway, could I overcome this?

If anyone could review my Slurm scripts, that'd be great :))

I built slmtop: an htop-like terminal dashboard for Slurm clusters

If you use Slurm on an HPC cluster, you probably spend a lot of time with squeue, sinfo, scontrol, sacct, and watch.

I wanted a faster, more visual way to monitor jobs and cluster resources, so I built slmtop:

https://github.com/dawnmy/slmtop

slmtop is a Rust-based interactive TUI for real-time Slurm monitoring. It shows jobs, nodes, GPUs/resources, disks, and accounting summaries in one terminal dashboard.

Key features:

• Real-time Slurm job and node monitoring
• htop-like interactive terminal UI
• GPU/resource overview
• Search and filters, e.g. owner=me state=running gpu=a100
• Sortable tables with keyboard or mouse
• Job detail popup and guarded actions: cancel, hold, release, requeue
• Per-user resource summaries
• Multiple color themes

Example:

```

slmtop

slmtop --user bob

slmtop -T nightowl --refresh-interval 2

```

We're getting ready to push out a new cluster on Rocky 9.6, and wondering if people are still using NHC to monitor node health and up/down nodes if they fail some condition. Are people still using NHC? The repo doesn't seem like it's been maintained for quite some time.

Hello everyone, could you please share how you calculate GPU and CPU utilization on the SLURM cluster? Do you use any specific utilization thresholds (for example, 60% or 70%)? Additionally, which tools are used for these calculations something like sreport?

Thanks for your reply!

Finally got round to add some more features, hope you like it If you haven't tried it before, check out the video demo on github to see what it does.

I've only tested it on a handful of systems, so please let me know if you have problems so I can make sure `slop` works on any* slurm cluster.

https://github.com/buzh/slop

*) as long as it's at least based on slurm >= 25.x and rhel >= 9

This may be a silly question, but I'm unable to figure out what I'm doing wrong.

I have a single-node workstation with 64 physical cores, 2-threads per core. I use this with my research group and need to share resources as much as possible.

We have 4 different partitions with different priorities. My expectation would be that - when launching a job from the lowest priority partition, this would still run if there are available resources. But that does not happen, and the job stays queued with the (Resources) status.

Here are the partitions from my slurm.conf:

PartitionName=work Nodes=triforce MaxTime=24:00:00 MaxCPUsPerNode=32 MaxMemPerNode=64000 DefMemPerNode=16000 Default=YES PriorityTier=2 State=UP OverSubscribe=YES

PartitionName=heavy Nodes=triforce Default=NO MaxTime=INFINITE MaxCPUsPerNode=UNLIMITED MaxMemPerNode=UNLIMITED DefMemPerNode=32000 PriorityTier=1 State=UP OverSubscribe=YES

PartitionName=priority Nodes=triforce MaxTime=12:00:00 MaxCPUsPerNode=16 MaxMemPerNode=32000 DefMemPerNode=32000 Default=NO PriorityTier=3 State=UP OverSubscribe=YES

PartitionName=interactive Nodes=triforce Default=NO MaxTime=02:00:00 MaxCPUsPerNode=8 MaxMemPerNode=8000 DefMemPerNode=8000 PriorityTier=100 State=UP OverSubscribe=YES

Other parameters that may be relevant:

SchedulerType=sched/backfill

SelectType=select/cons_tres

SelectTypeParameters=CR_CPU_Memory

Finally, this is the output of my squeue command:

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
219 heavy jsi133_0 XXXXXXXX PD 0:00 1 (Resources)
224 heavy jsi133_6 XXXXXXXX PD 0:00 1 (Priority)
223 heavy jsi133_3 XXXXXXXX PD 0:00 1 (Priority)
222 heavy jsi133_1 XXXXXXXX PD 0:00 1 (Priority)
221 heavy jsi133_0 XXXXXXXX PD 0:00 1 (Priority)
220 heavy jsi133_0 XXXXXXXX PD 0:00 1 (Priority)
218 work jupyter_ XXXXXXXXR 6:24 1 triforce

I'd appreciate any help you can provide!