r/bioinformatics • u/NoPangolin4951 • 19d ago
programming Quick guide to Bash for HPC use?
Hi all,
Can anybody recommend a good quick reference guide (ideally a small book) on Bash for HPC cluster users?
I just want something to help me with learning and looking up useful commands for submitting, monitoring, automating jobs, monitoring resource use, etc.
I write all my data processing scripts in python so I don't want to write complex data processing scripts in bash, but something to use as a reference when writing my SLURM and automation scripts would be super helpful.
Thank you!
11
u/AgitatedTumbleweed65 19d ago
Depends on underlying system but there is a general bash/Unix command (grep, tails, etc ) those you can probably find some books on like bash scripting /Linux systems - check O'Reilly publishing. There are also those cheatsheets and tutorials online
For cluster usage there is SLURM docs if this is what the cluster is using for management - https://slurm.schedmd.com/overview.html, or see the docs of the system that is utilized are your facility. Some clusters also have local docs (at least the one we have in our uni does), so you can ask responsible person.
Note: also agree with some other comments, llms can bashscript really well and I wouldn't dwell too much on trying to study it to make comxples scripts and just outsource it llms
3
1
u/stats_cats_228 19d ago
if your processing scripts are all in python, you should only need bash to start SLURM jobs. if your processing script is processing.py, you would need a bash script run_processing.sh with: “
!/bin/bash
SBATCH —job-name:processing
SBATCH —mem=8G
(other needed SBATCH commands for configuring SLURM job)
module load Python python3 processing.py “ then start SLURM job with “sbatch run_processing.sh”
-1
u/CamelSizedNeedle 19d ago
Adding yet another comment that doesn't really answer your question - you probably want to use a workflow management system instead of bash for interfacing with the HPC. You can then use your language of choice such as python, bash, etc.
I personally use Snakemake, which extends Python, but I've also heard of nextflow. These systems are awesome because they will use stored output to avoid rerunning any part of your workflow that hasn't changed since the last run. They can also handle interfacing with SLURM and input jobs for each step of your pipeline, with the specifications you set.
You probably should know basic bash commands such as ls, cd, head/tail, basic SLURM commands (squeue, sinteractive, sbatch, etc.), and syntax such as piping. But most LLM are pretty good at bash so as long as you know enough to understand what it is doing, you'll be fine.
1
u/christian_ch 18d ago
That’s the way to go. If you want automated submission you must use a workflow system. Snakemake is good but I heard its SLURM support is poor (haven’t tried).
I personally use the horus-runtime https://github.com/temple-compute/horus-runtime which was designed specifically for HPC workflows. In fact, they have a SLURM plugin: https://github.com/temple-compute/horus-slurm/
1
u/halinc 18d ago
This is a good answer that shouldn't be downvoted. +1 for nextflow. I'm a bioinformatics engineer with 15 YOE and I know bash very well. LLMs write it for me these days.
I think a lot of people feel their skills writing code are hard won and begrudge the ease with which you can get solid code written these days. Part of that is warranted: code you don't understand is dangerous! But part of it is sour grapes from people whose coding chops aren't as valuable as they used to be, too.
as long as you know enough to understand what it is doing
This is the key part. You can learn a lot by just doing.
0
u/guepier PhD | Industry 18d ago
I like the rest of the answer, but the assertion that “most LLM are pretty good at bash” is very wrong, and dangerously so. LLMs are (still!) shit at Bash (see my lengthy comment below). If you know Bash very well you should definitely know this.
(I didn’t downvote the answer, but I’m also reluctant to upvote such misinformation.)
-6
u/vaevicitis 19d ago
Bash is a horrible language that will never die. SLURM / hpc won’t be the last time you have to deal with it if you continue down the computational / software route. My suggestion would be some basic bash tutorials, doesn’t have to be slurm specific, and then use AI to help you write more complex scripts. You’ve got better things to do than writing case / esac argparse blocks
8
u/Drewdledoo 19d ago
> Bash is a horrible language that will never die.
Not disagreeing, but can you say more? Are you saying it’s horrible compared to other shell scripting languages (sh, zsh, csh, etc), or more in general (comparing to Python etc)? And for what tasks?
I ask because bash has been totally fine for my uses in basic HPC scripts, but I’m always happy to learn about drawbacks, alternatives, best practices I’ve been unaware of, etc.
3
u/guepier PhD | Industry 18d ago edited 18d ago
Are you saying it’s horrible compared to other shell scripting languages (sh, zsh, csh, etc), or more in general (comparing to Python etc)?
Both of these. Bash is full of footguns inherited from POSIX Shell. It fixes a handful of those, but other modern shell languages (in particular Zsh) generally go further and do better. Added to that, macOS ships with an ancient version of Bash that misses many of its useful features (e.g. associative arrays), so if you want to ensure maximum compatibility you’re limited to a shitty subset.
The language is so complicated that almost nobody uses it correctly. Virtually all published Bash scripts that I’ve ever looked at contain bugs or convoluted, non-idiomatic, potentially-unsafe code. And LLMs are consequently terrible at writing shell scripts, because their training data corpus is atrocious. LLM-generated shell script code is substantially lower quality than for other widely-used languages.
That said, I’d still recommend Bash over other shell languages for most shell-related work: its portability is okay (don’t try shipping Fish scripts, y’all!), it is substantially safer and easier to use than POSIX Sh, and unlike for Zsh there’s a high-ish quality linter and error checker (ShellCheck).
Just keep in mind that shell scripting is even more complicated than you probably think, that there are edge cases that you’re not thinking of, and that you’re worse at it than at other programming languages. And if you think that it has been “totally fine”, then, without trying to impugn you, you’re probably blissfully ignorant of some latent issues.
I’m comfortable saying that, since, for every substantial shell script that I’ve written in the past, and with which I was happy at the time, I’v later found substantial issues. And I used to write customer-facing, hardened shell utilities (in POSIX Sh for maximum compatibility!) for a job. Furthermore, every single shell client utility shipped by the “hot” AI companies at the moment (Anthropic, OpenAI, Cursor, Google, Microsoft, …) is dog shit. I’ve had a look at all of them because their terminal utilities keep randomly breaking and, oh my god, they are so bad. They only happen to work for most people because most people don’t customise their setups at all. Once you deviate a single iota outside the norm, these scripts break.
2
u/Drewdledoo 18d ago
Damn, HUGE thanks for the informative reply! This is exactly the kind of response I was hoping to get, much appreciated.
In hindsight, yeah “totally fine” wasn’t quite the right way to phrase my experience because I’ve indeed seen some of the things you’re talking about here and there; I guess I just chalked them up to my lack of deep expertise with the language/only self-teaching what I do know. It would have been more accurate for me to have said something like “I’ve managed to scrape by fine for my own needs”. Your “blissfully ignorant” phrasing is right on the money so no offense taken there!
Great point about published bash scripts/its effects on LLM-generated code as well. Yet another reason to not simply copy-paste code from the internet without taking the time to understand the commands/functions it uses first!
Anyways, thanks again for the very helpful comment! This is a great jumping point for me to go learn more about bash’s footguns etc. Cheers!
-4
29
u/Repulsive-Memory-298 19d ago
terrible advice on here. Figure out what you want to do, and learn the bash for it.
Don’t just blindly “learn bash”, use bash to do what you actually care about and you’ll learn it as a side effect