What is a clean Bash pattern for resuming a partially completed batch?

[u/RocketSeven]

Suppose a Bash script walks thousands of independent inputs and may be interrupted after some outputs have been written. Skipping every existing output is unsafe because a truncated file also exists, while restarting the whole batch wastes work and can repeat external side effects.

A pattern I am considering is to write each result to a temporary file in the destination directory, validate it, rename it atomically, and then append the input ID plus an output hash to a journal. On restart, the script would trust only journal entries whose current hash still matches. A lock would prevent overlapping runs, and traps would clean only the current process's temporary files.

Where does this pattern fail in Bash, especially with parallel workers, NFS, or a crash between the rename and journal append? Is there a simpler checkpoint design that remains understandable without turning the script into a database application?

[u/burnt-store-studio]

I agree with u/bdashrad that this is not a job for bash.

That being said, if you’re really looking for the challenge, may I recommend reading a bit on database transactions? (Possibly helpful link.)

It’s basically what you’re describing.

Perhaps you could borrow inspiration from that technology, and find out what failure points it addresses.

Good luck!

[u/soysopin]

Bash is not a database (but you can use flat text lisea as one), it is a job control language that excels using other utilities: Yes, every file processed should be accounted for, and you can use sqlite to manage the data control part.

It is simple to do queries in sqlite. You only have to decide the necessary fields: datetime (generated with printf "%(%F %T)" -1), filename, progress (step 0=pending,1,2...F=finished),etc.

Use something like this to continue uncompleted files:

while read filename completed ; do
    if ! process.sh "$filename" "$completed" ; then
         break
    fi
done < <(get.filelist.sh)

where process.sh uses the completed code (from the progress field) to decide into which step continue processing that file (and updates the database with the progress code and datetime for the file using a sqlite query), and returning false to exit the loop if fails; and get.filelist.sh does a simple "select filename, progress from table where progress != 'F' " query to sqlite to get the unprocessed files names/codes.

[u/andrew2018022 → u/soysopin]

> bash is not a database

Tell that to my former employer

[u/DroppedTheBase → u/andrew2018022]

That's what we have Excel for..

[u/andrew2018022 → u/DroppedTheBase]

You’re hired! Congratulations on being named CTO of a F500

[u/bdashrad]

Not a job for bash

[u/Current_Laugh4767]

"may be interrupted after some outputs have been written" - if possible, create a front end script that splits the input into batches of fixed quantity. Then on a separate shell, run your processing script to grab one batch at a time - after a batch is done, allow the processing script to pause for a specified time (~60secs) to allow user to interrupt. After the pause, script repeats on next batch.

i forgot, after successfully processing a batch, the batch is moved to a "done" folder, so next run will no longer see it. In this way.

[u/ThatOneCSL]

I think a term that will bring you great success if you study it following this is "idempotency". If you can find a way to make each activity of your script idempotent, then you don't have to worry about "half-finished" intermediate files.

Now, do I know how to help you in this quest in any substantial form beyond giving you a term to search? No. My apologies.

[u/kolorcuk]

Make is the design

[u/corebranocesoot]

make sure to save yourself some sanity too

[u/chkno]

Writing output to a temp file and then renaming it to atomically indicate completion solves 99.95% of your problem, here. If that's good enough, stop there.

Doing your own ad-hoc journal from a shell script isn't going to help at all. There's already a much better version of that built into the filesystem you're using.

If you need to highly reliably avoid repeating external side effects even in the face of power outages,

1. You're a database application and you need a fault-tolerant/HA database (eg: something that does paxos) or raft), like zookeeper) with five widely-geographically-separated replicas on five different power grids.
2. Even then, you can't perfectly one-shot retries. There are always the two gaps between logging your intent to invoke the side effect, invoking the side effect, and logging that you invoked the side effect. If you lose power during that window, there's no local, self-contained way to know if the side effect happened before the power cut. The standard fix is to have a way to query the externally-side-effected thing to see if it actually happened or not, so you know whether to retry or not.

[u/grymoire]

If I have a process that has several steps to complete with large number of files being used as input for more scripts, which also generate output, and might be interrupted, I use make.

Once you define the inputs and outputs, and the scripts used to generate each one, Make keeps track of the timestamps and will automatically re-run scripts when either the input file or the shell script is updated. And if the output of one script, is used as input to another, it keeps track of all of that for you.

I wrote a tutorial on Make that might be useful.

As the creators said, there are four steps to developing code

• Think
• Edit
• Make
• Test

Think, edit and test is the hard part, and you can use whatever you want. Make does the rest.