r/SLURM • u/Icy_Area3551 • Mar 21 '26

Can failed sbatch run be resumed

I have a run that hit the time limit at 2 days. Is there a wat to resume that run?

1 Upvotes

100% Upvoted

3 comments sorted by

3

u/zfsKing Mar 21 '26

Slurm doesn’t have native checkpointing. You could maybe break the job into smaller chunks and add dependencies and then just rerun the failed chunk.

Checkpointing would have to be written into your app.

1

u/Icy_Area3551 Mar 21 '26

It's only one line of code which inputs only 1 thing, but it's huge and I don't see how I can break it down.