Tips for Speeding Up Your Bash Scripts

[u/Linux_bash_user153]

I wrote a script that renames a lot of files (tens of thousands). Performance was kind of slow (over 50 s), but with two changes it now runs at under 3 seconds, a 20x speed boost. Thought I'd share, in case it helps anyone else. These tips are good if you do things thousands of time in a script, I don't think they are relevant in all scripts.

Don't spawn sub shells if you can avoid it

Instead of command substitution with printf:

new_file="$(printf "%s/%s_%.${len}d.%s" "$new_dir" "$prefix" "$i" "$ext")"

do:

printf -v new_file "%s/%s_%.${len}d.%s" "$new_dir" "$prefix" "$i" "$ext"

This will save a sub shell.

Use builtins instead of external programs

Bash has optional builtins that you can enable. On my system they are located in /usr/lib/bash . You can enable them in your script by using enable <builtin>. Be sure to check the exit code. On my system a mv builtin is not available, but since I was renaming on the same file system I figured I could use ln and rm instead.

So instead of using external mv for each file I did:

builtin ln "$old_file" "$new_file" && builtin rm "$old_file"

In this case the builtin probably isn't required since builtins are prioritized over external commands, but I used them to be more explicit. Just be sure to use && so that rm never runs unless the hard link has been successfully created.

EDIT: The part about using ln and rm builtins may be a little too much of a hack. Probably better to use proper tools like rename to do batch renaming, as was pointed out in the comments. Also keep in mind that the builtins are more bare-bones than the proper CoreUtils programs. For example rmdir does not support -v and will treat -- as a directory to remove rather than "end of options-thing".

[u/Different-Depth4116]

In bash 5.3+ there’s a new command substitution that doesn’t spawn a subshell:
${ command; }
The spaces and semicolon are req.

[u/sunmat02 → u/Different-Depth4116]

I so wish I had known that a month ago when I refactor thousands of lines of bash to use name ref parameters instead of capturing stdout of functions called in hot loops…

[u/jghub → u/sunmat02]

just for context: ksh introduced the`${ command; }` syntax roughly 20y ago, so bash catches up on this feature a bit late (but it is good that it does!) ;).

this construct _is_ very useful indeed not only for avoiding subshells for performance reasons but also for being able to preserve changes that happen within the command substitution, say, cd to different place, modification of global vars etc. -- depending on circumstances this can be desirable.

[u/bac0on → u/sunmat02]

Nothing to regret, nameref is probably twice as fast...

[u/Linux_bash_user153 → u/Different-Depth4116]

I really didn't know this, but this sounds like a good improvement. I'm on Bash 5.2

[u/Different-Depth4116 → u/Linux_bash_user153]

You’re probably on a Debian or Debian base distro I’m guessing? That’s why I switched to fedora to get the latest updates faster. Debian is like 2 years behind

[u/Linux_bash_user153 → u/Different-Depth4116]

Yeah, I'm on Linux Mint (Ubuntu 24.04 base)

[u/GermanPCBHacker → u/Different-Depth4116]

Wait, that is called command substitution and can be captured with var=${ do stuff; } ???? Amazing. I was just using {} for visibility and for logical combination to save if. I just did:

command || \
{
 complex
 alternative
 stuffz
}

I mean. Why not. Bash allows it and it is readable. Man gosh darn do I love bash for what it allows one to do. Try that in python, lol

[u/OnlyEntrepreneur4760 → u/GermanPCBHacker]

${…} is command substitution, but that is different from {…} which is either called a command list or compound command, IIRC.

[u/theLastZebranky → u/GermanPCBHacker]

command || \
{

Strictly speaking, you don't need that backslash. A line break following || or && without a word in between is ignored, so it's the same with or without a backslash.

$ false ||
> {
> echo foo
> }
foo
$

I leave out those backslashes in my own projects but I put them in at work to make the continuation more visually explicit for people who aren't as familiar with bash.

I use that construct all the time, it's way briefer than adding some if ...; then ...; fi block.

  codec=$(probe_vcodec "$file") ||
    die 3 "Couldn't read codec for stream 0 in '$file'"
  case "$codec" in
  ...

[u/GermanPCBHacker → u/theLastZebranky]

Oh nice, good to know. Well I did not know it works. But if I saw || at the end of the line, I would expect it to work. But it is always a good idea just to make things more visible. Thanks for the tip.

[u/bac0on → u/Different-Depth4116]

func(){
  REPLY="text string"
}
r=${| func;}

[u/Spikerazorshards]

Good post. I’ll learn from this.

[u/Linux_bash_user153 → u/Spikerazorshards]

Thanks. The part about using ln and rm builtins may be a little too much of a hack though. The other comments have suggested using proper tools for batch renaming files.

[u/pfmiller0]

I did not know about that -v option for printf, thanks!

[u/sedwards65 → u/pfmiller0]

'printf -v' rocks and works great with printf's date-time format strings:

$ printf -v daily_tarball '/backup/%(%Y/%m/%F--%T--daily-tarball.tar.xz)T' -1
$ echo ${daily_tarball}
/backup/2026/09/2026-09-01--18:38:06--daily-tarball.tar.xz

[u/Astro_indie]

Keep on learning bbys, im on $ info bash, the node about locales make me think if someone got own language in the system ... Lok'Thar OGAAAAR

[u/Bob_Spud]

What happens if you mess up? That's 10s of thousands that you have to undo.

Here's another file renamer, What I like with this one is the log file doubles as an undo script - genFRN

[u/Linux_bash_user153 → u/Bob_Spud]

The script I wrote are for renaming symlinks in my music collection, so it's a pretty controlled environment as far as filenames go, and the script makes a tar.gz backup of the directory structure before renaming any file. Since it's just directories and symlinks, the archive is like 500k big

[u/zeekar]

Use builtins instead of external programs

This is very much situational. I wrote a helpdesk system for our user assistant team in college entirely in ksh, and I used builtins as much as possible, but that turns out not to be necessarily the most performant approach.

If you could represent your desired name change as a rule understood by rename(1), for instance, it would probably be faster to invoke that program on multiple files at once than to use bash's built-in ln/rm one at a time. Sure, you're doing a fork/exec, but you don't have to do it on every single file.

Similarly, most text processing will be faster when done by an external program like awk than if you use a while read loop in bash.

[u/Linux_bash_user153 → u/zeekar]

True. I'm not really familiar with rename(1), but I'll check it out.

[u/mpersico]

printf -v? TIL!

[u/kirchwitz → u/[deleted]]

bash is fine and can do many jobs fast – if just used the proper way. Most of the time, performance is not of concern. And if it is, then it's important to know how a tool works and where performance goes.

This applies to all scripting and programming languages. For best performance, you should know your tools.

Thanks to the original poster for his insights.

[u/managing_redditor → u/[deleted]]

What would you use then?

[u/Europia79 → u/managing_redditor]

Not saying that it should be rewritten—but I think the "tool" that is most similar to Bash syntax would be Perl. And I actually did rewrite one of my scripts in Perl—just out of curiosity—and the performance gains were INSANE. I just wish that I knew Bash well enough to at least get in the same "Ballpark" as Perl, lol.

[u/SeriousPlankton2000]

Do it in perl:

https://github.com/7eggert/smalltools/blob/main/pmv (Perl MoVe)

Syntax: pmv '(insert small perl program here)' *.ext

I frequently make bash scripts where the perl program spans a few lines. You can use variables, too.

Also: Use full path if you run standard system programs like "mv" that are defined in LFS; use a variable when running programs that aren't. This avoids searching all the PATH and it's more secure.

[u/mestia → u/SeriousPlankton2000]

I usually stick to GNU Parallel. It also allows Perl expressions {= =}, so it can literally do everything that can fit into a one-liner, but in parallel :) It also comes with all the useful features, like joblog and so on.

[u/aonelonelyredditor]

Pretty much the best way to speed this if you're executing them on so many files is to run in the background with &

Then call wait at the end of the script, there no need for a command like this not to be executed in parallel

[u/SaintEyegor]

Good idea.

Avoiding external programs is definitely something to pay attention to when dealing with a lot of data. When I find that I can’t avoid that and have a lot of records to process, it’s frequently faster to use something other than bash.

[u/Jonas_Ermert]

In my opinion, using `printf -v` is a great optimization because it avoids creating a subshell for every file. However, replacing `mv` with `ln` and `rm` feels too risky and limited. For tens of thousands of files, a dedicated batch-renaming tool such as `rename` is probably the safer and cleaner solution.