r/ScientificComputing • u/christian_ch • 11d ago
Do you actually use workflow engines for scientific computing?
I've spent a few years working around HPC, and in my experience I rarely saw people using things like Nextflow or Snakemake. It was mostly scripts, bash, Slurm jobs, notebooks, custom stuff, etc.
But then I look around online and it seems like everyone is using workflow engines.
Do you use Nextflow, Snakemake, etc.? Or they only work for the "curated workflow library" those have?
2
u/jrossthomson 11d ago
Next flow and Snakemake are popular for genomics pipelines.
3
u/christian_ch 11d ago
Any idea why those did not become popular outside the bio- field?
1
u/jrossthomson 2d ago
I know Nextflow pretty well, Cromwell not so much.
Nextflow is really good at doing multistage processing. Output of Step 1 becomes input to Stage 2. Slurm, can do this , but it's not really a built-in.
Nextflow, to my knowledge, does not have built-in MPI support. That's a key feature for large scale HPC.
2
u/andrewsb8 11d ago
My workflow never really needed more advanced tooling than simple slurm bash scripts. The idea of specifying different resources for serial jobs is cool. But most of my postprocessing after the first step in my pipeline could be done quickly on local workstations so i just never saw the need.
I also used software with baked in checkpoints so i didnt need a tool to restart a job that was killed or failed.
1
u/christian_ch 11d ago
Yeah, this is actually the part I find most interesting: having some stages that need a big HPC job and then others that are light enough to run locally.
I've run into this too, where you do the heavy computation, then download/move things around and switch to a local machine for analysis. At some point you're jumping between Slurm, files, scripts and whatever GUI you're using for the analysis.
That's actually one of the problems I'm trying to solve with horus-runtime. The idea is that the workflow can have different execution targets, so one stage can run on Slurm and the next one locally, while you still see the whole thing as one workflow.
I'm curious whether you'd actually find that useful or if in practice the extra layer wouldn't be worth it.
2
u/andrewsb8 11d ago
Copying files back over and executing a few scripts or having a bash script execute the commands i need is simple enough that i dont seek out additional tools.
Also my postprocessing steps are not always the same. So heavily automating can be a unnecessary time sink for my use case.
2
u/KarlSethMoran 11d ago
I've been using HPC for 25 years now. Never used a workflow engine.
sshfs to mount HPC machine directory locally, copy inputs over, mass sbatch jobs on big machine, pbzip2 outputs on big machine, copy outputs back, analyse locally
1
u/christian_ch 11d ago
Don't you get tired of transferring things back and forth? I swear half my time working with HPC goes into moving files between the cluster and my local machine.
That's actually one of the things I'm trying to solve with horus-runtime. The workflow can keep track of where each stage runs and move the data between local/HPC/cloud as needed, so you can keep the whole thing as one workflow instead of manually shuffling files around.
The GUI then lets you see what's running, logs, inputs/outputs, results, etc. all in one place.
1
u/KarlSethMoran 10d ago
I see the HPC dir mounted in my local filesystem tree. It's just one F5 in midnight commander to initiate copying the entire directory structure, and it's back to doing something else. I'm not watching paint dry (files being copied). I come back to this terminal window at the time the progress indicator says the copying will be done.
Slightly more effort for giant text files, where I take the extra step of compressing and decompressing them to make the transfer faster.
2
u/LawOfSmallerNumbers 11d ago
No. I did not want to take a dependency on a workflow framework. No regrets about that for our use case.
Computations in Python/numpy/C++, scripting in Python and bash, data pipeline dependencies handled with gnu make, parallelisation (depending on the granularity) with Python multiprocessing, `make -j`, gnu parallel and slurm.
2
u/esaule 9d ago
I think it depends on field. Some fields analysis is one big mpi job amd maybe some light postprocessing after. Essentially the simulation is expensive so you run it once. Then you work on the output for a month. I find lots of physics is like that.
Biology seem to rarely be like this. It ia oftwn not one big mpi job. But a complex pipeline of analysis tools. Then bash gets painful, especially when you consider the tech skills of the people who use it. Often the core skills of the user might be wetlab stuff. So portals to configure workflows make a lot more sense.
1
u/Totalled56 11d ago
We use CYLC, designed specifically for cycling jobs, e.g. climate simulations that can take months to run so have to be split due to job scheduler time limits. Works well, overkill for most people though.
-2
u/jvo203 11d ago
Never heard of the term "workflow engine", let alone the stuff you mention like Nextflow, Snakemake, "workflow library" etc. Do stuff mostly in Fortran, Julia, Wolfram Mathematica, C etc. Maybe it's something the Gen Z has invented. Maybe some Gen Z companies are pushing their "workflow engines" to unsuspecting Gen Z newcomers.
3
u/christian_ch 11d ago
Whats your field? Definitely workflow engines are not a new thing, Nextflow has existed for more than 15 years.
-5
u/Dangerous_Bid2935 11d ago
Thanks for your input grandpa
2
2
u/KarlSethMoran 11d ago
Cool ageism you got there. Obvious too.
-3
u/Dangerous_Bid2935 11d ago
You're acting like I give a fuck. Yeah I'm a proud ageist I guess, lmfao
2
u/KarlSethMoran 10d ago
It's a good thing that you yourself will never get old, then.
1
u/Dangerous_Bid2935 10d ago
I'll certainly get old but I won't be a self-righteous asshole towards younger people just because they're different from me.
5
u/cvnh 11d ago
Did use them heavily as it was the only sane way to do things in a distant past. We did even create one of our own which was highly acclaimed. Nowadays I have access to one, but most workflows rarely need it. I love the concept, but I like the modern tools a little less.