🏗️ Production-Grade Bash Script

A production-grade bash script: a template you can copy, and how to test it
./backup-site.sh --dry-run -s /srv/site -d /backup/site; ./test_backup.sh

A script you run once at a terminal can be sloppy. A script that runs every night from cron, on a server nobody is watching, cannot. It needs to explain itself (--help), to say what it is doing (logging), to be safe to try (--dry-run), to refuse to run twice at once (a lock), to clean up after itself even when it is killed (traps), to publish its result only when it is complete (atomic rename), to fail with a clear message and a meaningful exit code, and to be tested, including on the days things go wrong.
This page builds one real script with all of those properties, a backup tool that takes snapshots of a folder with rsync, and then tests it. It is meant to be copied and adapted. Everything was run for real on Ubuntu 24.04 with GNU bash 5.2.21 and rsync 3.2.7, and the output is the real output, including the bugs that testing found in the first draft (they are described, because the mistakes are the useful part). The other Shell Scripting pages explain the pieces in depth: Error Handling, Arguments and Options, Gotchas and POSIX vs bash.

  Strict mode:       set -Eeuo pipefail, inherit_errexit, quoted variables, no unset variables
  Usable:            --help, clear usage errors, exit code 2 for bad usage, long and short options
  Safe to try:       --dry-run that really changes nothing
  Observable:        timestamped log lines with levels, on stderr and optionally in a file, a --verbose switch
  Single instance:   a lock, so a slow run is not joined by the next scheduled one
  Clean exit:        an EXIT trap that removes partial work, INT and TERM traps that exit with 130 and 143
  Atomic:            build in a temporary name and rename when complete, so readers never see half a result
  Meaningful codes:  one number per kind of failure, documented in --help
  Idempotent:        running it twice gives the same result as running it once
  Testable:          logic in functions, the main program behind a guard, and tests that cover the failure paths

This is the whole script as tested (115 lines including comments). Read it once, then the notes that follow.
$ cat backup-site.sh
#!/usr/bin/env bash
# backup-site.sh - snapshot a folder with rsync, keeping the last N snapshots
set -Eeuo pipefail
shopt -s inherit_errexit 2>/dev/null || true

readonly PROG=${0##*/}
readonly E_USAGE=2 E_LOCK=3 E_SOURCE=4 E_RSYNC=5

# ---- defaults (overridable by options) -------------------------------------
source_dir=""; dest_dir=""; keep=3; dry_run=0; verbose=0; log_file=""
partial=""

# ---- helpers ----------------------------------------------------------------
usage() {
  cat <<USAGE
Usage: $PROG -s SOURCE -d DEST [-k N] [-n] [-v] [-l LOGFILE]
  -s, --source DIR    folder to back up (required)
  -d, --dest DIR      folder that holds the snapshots (required)
  -k, --keep N        snapshots to keep (default $keep)
  -n, --dry-run       show what would happen, change nothing
  -v, --verbose       log debug messages too
  -l, --log FILE      also append the log to FILE
  -h, --help          this help
Exit codes: 0 ok, 2 usage, 3 another run holds the lock, 4 bad source, 5 rsync failed
USAGE
}

log() {  # log LEVEL message...
  local level=$1; shift
  [[ $level == DEBUG && $verbose -eq 0 ]] && return 0
  local line; line=$(printf '%s %-5s %s' "$(date '+%F %T')" "$level" "$*")
  printf '%s\n' "$line" >&2
  if [[ -n $log_file ]] && ! printf '%s\n' "$line" >> "$log_file" 2>/dev/null; then
    printf '%s WARN  cannot write to log file %s, logging to the terminal only\n' "$(date '+%F %T')" "$log_file" >&2
    log_file=""
  fi
}
die() { local rc=$1; shift; log ERROR "$*"; exit "$rc"; }

run() {  # run a command, or only announce it in dry-run mode
  if (( dry_run )); then log INFO "DRY-RUN: $*"; else log DEBUG "run: $*"; "$@"; fi
}

cleanup() {
  # runs on every exit; the script's exit status is kept, so never "return N" here
  if [[ -n $partial && -d $partial ]]; then
    log WARN "removing incomplete snapshot $partial"; rm -rf -- "$partial"
  fi
  return 0
}

parse_args() {
  while (( $# )); do
    case $1 in
      -s|--source)  [[ $# -ge 2 ]] || die $E_USAGE "$1 needs a value"; source_dir=$2; shift ;;
      -d|--dest)    [[ $# -ge 2 ]] || die $E_USAGE "$1 needs a value"; dest_dir=$2; shift ;;
      -k|--keep)    [[ $# -ge 2 ]] || die $E_USAGE "$1 needs a value"; keep=$2; shift ;;
      -l|--log)     [[ $# -ge 2 ]] || die $E_USAGE "$1 needs a value"; log_file=$2; shift ;;
      -n|--dry-run) dry_run=1 ;;
      -v|--verbose) verbose=1 ;;
      -h|--help)    usage; exit 0 ;;
      *)            usage >&2; die $E_USAGE "unknown argument: $1" ;;
    esac
    shift
  done
  [[ -n $source_dir ]] || die $E_USAGE "--source is required"
  [[ -n $dest_dir ]]   || die $E_USAGE "--dest is required"
  [[ $keep =~ ^[1-9][0-9]*$ ]] || die $E_USAGE "--keep must be a positive number, got '$keep'"
}

prune_old() {  # prune_old DIR KEEP: delete all but the newest KEEP snapshots
  local dir=$1 keep=$2 snaps=() s i
  while IFS= read -r s; do snaps+=("$s"); done < <(find "$dir" -mindepth 1 -maxdepth 1 -type d -name '20*' | sort)
  local n=${#snaps[@]}
  (( n > keep )) || { log DEBUG "$n snapshot(s), nothing to prune"; return 0; }
  for (( i = 0; i < n - keep; i++ )); do
    log INFO "pruning old snapshot ${snaps[i]##*/}"
    run rm -rf -- "${snaps[i]}"
  done
}

main() {
  trap cleanup EXIT
  trap 'log ERROR "line $LINENO: $BASH_COMMAND"' ERR
  trap 'log WARN "interrupted (INT)"; exit 130' INT
  trap 'log WARN "terminated (TERM)"; exit 143' TERM
  parse_args "$@"

  [[ -d $source_dir && -r $source_dir ]] || die $E_SOURCE "cannot read source folder: $source_dir"
  run mkdir -p -- "$dest_dir"
  [[ -d $dest_dir ]] || (( dry_run )) || die $E_USAGE "cannot create $dest_dir"

  if (( ! dry_run )); then
    exec 9>"$dest_dir/.lock"
    flock -n 9 || die $E_LOCK "another backup is already running (lock: $dest_dir/.lock)"
  fi

  local stamp snapshot link_dest=()
  stamp=$(date '+%Y-%m-%d_%H%M%S')
  snapshot=$dest_dir/$stamp
  partial=$dest_dir/.partial-$stamp
  [[ -L $dest_dir/latest ]] && link_dest=(--link-dest="$(readlink -f "$dest_dir/latest")")

  log INFO "backing up $source_dir -> $snapshot (keep $keep)"
  if ! run rsync -a --delete "${link_dest[@]}" -- "$source_dir"/ "$partial"/; then
    die $E_RSYNC "rsync failed"
  fi
  run mv -- "$partial" "$snapshot"
  run ln -sfn -- "$stamp" "$dest_dir/latest"
  [[ -d $dest_dir ]] && prune_old "$dest_dir" "$keep"
  log INFO "done"
}

# run main only when executed, not when sourced (lets the tests load the functions)
if [[ ${BASH_SOURCE[0]} == "$0" ]]; then main "$@"; fi

  Header: the shebang names bash (see the POSIX page), set -Eeuo pipefail turns on the safety options, and inherit_errexit extends set -e into command substitutions (the 2>/dev/null || true guards old bash versions that lack it).
  Constants: readonly names for the exit codes (E_USAGE=2, E_LOCK=3, E_SOURCE=4, E_RSYNC=5), so the numbers appear once and the code reads like prose: die $E_LOCK "...". They match the line at the bottom of --help.
  Defaults in variables at the top; options only change them. A reader sees every setting in one place.
  usage(): one function, used for --help (stdout, exit 0) and for mistakes (stderr, exit 2). The here-document is unquoted so $PROG and the default $keep are filled in, which keeps the help text from going out of date.
  log(): one function writes every message: a timestamp, a left-aligned level (the %-5s), the text. It goes to stderr (so stdout stays clean for data), and to a file when -l is given. DEBUG lines appear only with --verbose. If the log file cannot be written, it warns once and switches the file off instead of killing the backup (see "What testing found").
  die(): log an ERROR and exit with the given code; the exit code is the FIRST argument (the Error Handling page shows why).
  run(): every command that changes something goes through it. In dry-run mode it only prints "DRY-RUN: command"; otherwise it logs the command at DEBUG level and runs it, preserving quoting by using "$@". Reads (find, date, readlink) are not wrapped: dry-run must still be able to look.
  cleanup(): the EXIT trap. It removes the half-finished .partial folder if one exists, and always ends with return 0. The script's own exit status passes through untouched, so a failing run still exits non-zero.
  parse_args(): the while/case loop from the Arguments page. Every option that takes a value checks that there is one, and the three required conditions are checked together after the loop, with exit code 2.
  prune_old(): collects the snapshot folders (names starting with 20 and sorted by name, which is also by date because of the timestamp format), counts them and removes all but the newest N, through run() so dry-run covers it. It reads the list with while read < <(find ...) so the array is filled in the main shell (the pipe-subshell trap from the Gotchas page).
  main(): installs the traps, parses the arguments, checks the source, takes the lock, builds the new snapshot in a .partial folder, renames it into place, moves the latest link, and prunes. Every step either succeeds or ends in die with a specific code.
  The last two lines: main runs only when the file is executed. When another script (the tests) sources the file, ${BASH_SOURCE[0]} differs from $0 and nothing runs, so the functions can be loaded and tested one by one.

$ ./backup-site.sh --help
Usage: backup-site.sh -s SOURCE -d DEST [-k N] [-n] [-v] [-l LOGFILE]
  -s, --source DIR    folder to back up (required)
  -d, --dest DIR      folder that holds the snapshots (required)
  -k, --keep N        snapshots to keep (default 3)
  -n, --dry-run       show what would happen, change nothing
  -v, --verbose       log debug messages too
  -l, --log FILE      also append the log to FILE
  -h, --help          this help
Exit codes: 0 ok, 2 usage, 3 another run holds the lock, 4 bad source, 5 rsync failed
[exit 0]
Mistakes get a one-line message and the right exit code (this is the output after the cleanup fix described below):
$ ./backup-site.sh
2026-10-02 07:31:07 ERROR --source is required
[exit 2]
$ ./backup-site.sh -s src -d out -k zero
2026-10-02 07:31:08 ERROR --keep must be a positive number, got 'zero'
[exit 2]
$ ./backup-site.sh -s /nonexistent -d out
2026-10-02 07:31:08 ERROR cannot read source folder: /nonexistent
[exit 4]
A dry run shows every step and creates nothing:
$ ./backup-site.sh -s src -d out -n
2026-10-02 07:32:57 INFO  DRY-RUN: mkdir -p -- out
2026-10-02 07:32:57 INFO  backing up src -> out/2026-10-02_073257 (keep 3)
2026-10-02 07:32:57 INFO  DRY-RUN: rsync -a --delete -- src/ out/.partial-2026-10-02_073257/
2026-10-02 07:32:57 INFO  DRY-RUN: mv -- out/.partial-2026-10-02_073257 out/2026-10-02_073257
2026-10-02 07:32:57 INFO  DRY-RUN: ln -sfn -- 2026-10-02_073257 out/latest
2026-10-02 07:32:57 INFO  done
[exit 0]
$ [ -e out ] && echo yes || echo no
no
The folder out does not exist afterwards. The printed commands are exactly what the real run will execute, so the dry run doubles as documentation.

$ ./backup-site.sh -s src -d out -l run.log -v
2026-10-02 07:32:57 DEBUG run: mkdir -p -- out
2026-10-02 07:32:57 INFO  backing up src -> out/2026-10-02_073257 (keep 3)
2026-10-02 07:32:57 DEBUG run: rsync -a --delete -- src/ out/.partial-2026-10-02_073257/
2026-10-02 07:32:57 DEBUG run: mv -- out/.partial-2026-10-02_073257 out/2026-10-02_073257
2026-10-02 07:32:57 DEBUG run: ln -sfn -- 2026-10-02_073257 out/latest
2026-10-02 07:32:57 DEBUG 1 snapshot(s), nothing to prune
2026-10-02 07:32:57 INFO  done
[exit 0]
$ ls -l out | awk 'NR>1{print $9,$10,$11}'
2026-10-02_073257
latest -> 2026-10-02_073257
The second run found the "latest" link and passed it to rsync as --link-dest, so unchanged files are hard links to the previous snapshot and cost no space. Both snapshots show the same inode for the same file:
$ ./backup-site.sh -s src -d out -l run.log
2026-10-02 07:32:58 INFO  backing up src -> out/2026-10-02_073258 (keep 3)
2026-10-02 07:32:58 INFO  done
$ for d in out/20*; do printf '%s  inode of index.html: ' "${d#out/}"; stat -c %i "$d/index.html"; done
2026-10-02_073257  inode of index.html: 19012412
2026-10-02_073258  inode of index.html: 19012412
After the file was changed and two more runs were made, the oldest snapshot was pruned, and only the changed file got a new inode:
$ ./backup-site.sh -s src -d out -l run.log
2026-10-02 07:33:00 INFO  backing up src -> out/2026-10-02_073300 (keep 3)
2026-10-02 07:33:00 INFO  pruning old snapshot 2026-10-02_073257
2026-10-02 07:33:00 INFO  done
$ ls out | grep '^20' | tr '\n' ' '; ls -l out/latest | awk '{print $9,$10,$11}'
2026-10-02_073258 2026-10-02_073259 2026-10-02_073300
out/latest -> 2026-10-02_073300
$ for d in out/20*; do printf '%s  ' "${d#out/}"; stat -c %i "$d/index.html"; done
2026-10-02_073258  19012412
2026-10-02_073259  19012420
2026-10-02_073300  19012420
Three snapshots remain (keep 3), "latest" points to the newest, and the log file has the same lines as the terminal. (The timestamp has one-second resolution, which is why the lab slept one second between runs; the lock stops two real runs from overlapping.)

The happy path is the easy part. A fake rsync (a small script placed first in PATH, see "Mocking") lets the lab make rsync fail or hang on demand.
1. rsync fails halfway: the script exits 5, the half-written folder is removed, and nothing is published (only the lock file remains):
$ ./backup-site.sh -s src -d out2          # PATH has a fake rsync that creates the target and exits 23
2026-10-02 07:31:28 INFO  backing up src -> out2/2026-10-02_073128 (keep 3)
rsync: fake error: connection unexpectedly closed
2026-10-02 07:31:28 ERROR rsync failed
2026-10-02 07:31:28 WARN  removing incomplete snapshot out2/.partial-2026-10-02_073128
[exit 5]
$ ls -A out2
.lock
2. Another run is active: a slow first run holds the lock; a second run, started a second later, refuses at once with exit 3:
$ ./backup-site.sh -s src -d out2
2026-10-02 07:31:29 ERROR another backup is already running (lock: out2/.lock)
[exit 3]
The first run was not disturbed and finished with exit 0. The lock belongs to the open file descriptor 9, so it is released by the kernel when the script ends for ANY reason, including kill -9: there is no stale lock file to clean up by hand, which is the weakness of "touch a pid file" schemes.
3. TERM during the copy: the script logs it, removes the partial folder and exits with 143 (the first draft exited 130; see below):
$ ./backup-site.sh -s src -d out3 & kill -TERM $!; wait $!          # PATH has the slow fake rsync (FAKE_MODE=slow)
2026-10-02 07:31:55 INFO  backing up src -> out3/2026-10-02_073155 (keep 3)
2026-10-02 07:32:01 WARN  terminated (TERM)
2026-10-02 07:32:01 WARN  removing incomplete snapshot out3/.partial-2026-10-02_073155
[exit 143]
Look at the times: the TERM was sent one second after the start, but the handler ran six seconds later, at 07:32:01. Bash runs a trap only when the command it is currently running ends, and the fake rsync was a six-second foreground command. A real rsync of a large tree would delay the handler for as long as it runs. Two consequences: a service manager that sends TERM and waits only a few seconds will follow up with KILL, which no trap can catch (the leftover .partial folder then stays until someone removes it; the notes near the end show a cheap fix), and a long foreground command is better started in the background with wait, so the shell can react at once (see the Error Handling page).
4. The log file cannot be written (the lab used /dev/full, a device that always answers "no space left"):
$ ./backup-site.sh -s src -d out3 -l /dev/full
2026-10-02 07:31:54 INFO  backing up src -> out3/2026-10-02_073154 (keep 3)
2026-10-02 07:31:54 WARN  cannot write to log file /dev/full, logging to the terminal only
2026-10-02 07:31:55 INFO  done
[exit 0]
The backup went ahead and the warning appeared once. Whether a log failure should abort the job is a decision for you to make on purpose; for a backup, losing the copy because the log disk is full is the worse outcome.

Running the failure paths showed three mistakes in code that had looked right when written. They are the best argument for testing:
  1. Every error exit printed a second, bogus ERROR line. The first cleanup() ended with return "$rc" to "preserve the exit status". But a non-zero return from a function counts as a failing command, and with set -E the ERR trap fires for it, inside the EXIT trap. The trap printed the last command it knew about:
     2026-10-02 07:30:51 ERROR --source is required
     2026-10-02 07:30:51 ERROR finished with status 2
     2026-10-02 07:30:51 ERROR line 1: exit "$rc"
     The last two lines are the bug. An EXIT trap does not need to return the status: the script's exit status is kept unless the trap itself calls exit. The fix is return 0 (and no "finished with status" line, because die has already said why).
  2. A full disk for the log file killed the whole backup, with a confusing message. The write to the log file was an ordinary command, so under set -e its failure ended the script, and the ERR trap's own attempt to log failed too:
     ./backup-site.sh: line 33: printf: write error: No space left on device
     2026-10-02 07:31:40 ERROR line 33: printf '%s\n' "$line" >> "$log_file"
     ./backup-site.sh: line 33: printf: write error: No space left on device
     [exit 1]
     The fix is the if ! printf ... >> "$log_file" 2>/dev/null; then warn once and switch the file off; fi that is in the final script.
  3. The signal handler used one number for two signals. The first draft had trap '...; exit 130' INT TERM, so a TERM exited with 130 (the code for Ctrl+C), which confuses anything that reads the status. The final script has one trap per signal: 130 for INT, 143 for TERM.
A smaller wart was removed too: the first timestamp was cut from nanoseconds with cut -c1-21 to avoid collisions, which produced odd names like 2026-10-02_0731080616. The lock already prevents two runs from overlapping, so the plain format %Y-%m-%d_%H%M%S is enough.

Scripts are hard to test when everything happens at the top level: loading the file runs the whole job. Two habits fix that:
  1. Put the logic in functions that take arguments and return values (or set well-named variables), with one main function that wires them together.
  2. Run main only when the file is executed, not when it is sourced:
     if [[ ${BASH_SOURCE[0]} == "$0" ]]; then main "$@"; fi
  When you run ./backup-site.sh, BASH_SOURCE[0] and $0 are both the script, so main runs. When a test does source ./backup-site.sh, $0 is the test, the condition is false, and the test gets all the functions without any side effects. This is the shell version of Python's if __name__ == "__main__".
Things that make functions testable: they use variables passed as arguments rather than hidden globals where possible (prune_old takes DIR and KEEP), they write results to stdout or to named variables, and they use exit only through die (which the test can run in a subshell and catch).

No framework is needed to start. This harness is 15 lines, counts passes and failures, prints a line for each test, and exits non-zero if any failed. The rest of the file is the tests. It sources the script, switches off errexit so that a failing test does not kill the harness, and runs everything in a temporary folder removed by a trap:
$ cat test_backup.sh
#!/usr/bin/env bash
# test_backup.sh - a tiny test harness for backup-site.sh
set -u
here=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
pass=0; fail=0
ok()  { pass=$((pass+1)); printf 'ok    %s\n' "$1"; }
bad() { fail=$((fail+1)); printf 'FAIL  %s\n' "$1"; [[ -n ${2:-} ]] && printf '        %s\n' "$2"; return 0; }
assert_eq()     { if [[ $2 == "$3" ]]; then ok "$1"; else bad "$1" "expected [$2], got [$3]"; fi; }
assert_status() { if [[ $2 -eq $3 ]]; then ok "$1"; else bad "$1" "expected status $2, got $3"; fi; }

# load the functions without running main (the script only runs main when executed)
source "$here/backup-site.sh"
set +e +E                                   # the harness must survive failing code
tmp=$(mktemp -d); trap 'rm -rf -- "$tmp"' EXIT

# --- parse_args (run in subshells: it calls exit on errors) ---------------------
out=$( parse_args -s /a -d /b -k 5 -n; echo "$source_dir $dest_dir $keep $dry_run" )
assert_eq "parse_args stores the options" "/a /b 5 1" "$out"
( parse_args -d /b ) 2>/dev/null;               assert_status "missing --source exits 2" 2 $?
( parse_args -s /a -d /b -k 0 ) 2>/dev/null;    assert_status "--keep 0 exits 2" 2 $?
( parse_args -s /a -d /b -k abc ) 2>/dev/null;  assert_status "--keep abc exits 2" 2 $?
( parse_args -s /a -d /b --bogus ) >/dev/null 2>&1; assert_status "unknown option exits 2" 2 $?

# --- prune_old ------------------------------------------------------------------
snaps=$tmp/snaps; mkdir -p "$snaps"
for d in 2026-01-01_000001 2026-01-02_000001 2026-01-03_000001 2026-01-04_000001 2026-01-05_000001; do mkdir "$snaps/$d"; done
ln -s 2026-01-05_000001 "$snaps/latest"; : > "$snaps/.lock"
prune_old "$snaps" 2 2>/dev/null
assert_eq "prune_old keeps the newest 2" "2026-01-04_000001 2026-01-05_000001" "$(cd "$snaps" && ls -d 20* | tr '\n' ' ' | sed 's/ $//')"
assert_eq "prune_old leaves latest and .lock alone" "yes yes" "$([[ -L $snaps/latest ]] && echo yes) $([[ -e $snaps/.lock ]] && echo yes)"
prune_old "$snaps" 5 2>/dev/null
assert_eq "prune_old with fewer than KEEP removes nothing" 2 "$(ls -d "$snaps"/20* | wc -l)"

# --- log ------------------------------------------------------------------------
verbose=0; out=$(log DEBUG "hidden" 2>&1);  assert_eq "DEBUG is silent without --verbose" "" "$out"
verbose=1; out=$(log DEBUG "shown" 2>&1);   assert_eq "DEBUG is shown with --verbose" "1" "$(grep -c 'DEBUG shown' <<<"$out")"
verbose=0; log_file=$tmp/test.log; log INFO "to file" 2>/dev/null
assert_eq "log appends to the log file" "1" "$(grep -c 'INFO  to file' "$log_file")"
log_file=/dev/full; log INFO "x" 2>/dev/null
assert_eq "an unwritable log file is switched off, not fatal" "" "$log_file"

# --- the whole script, end to end -------------------------------------------------
src=$tmp/src; mkdir -p "$src/sub"; echo one > "$src/a.txt"; echo two > "$src/sub/b.txt"
dst=$tmp/dst
"$here/backup-site.sh" -s "$src" -d "$dst" >/dev/null 2>&1
assert_status "a normal run exits 0" 0 $?
assert_eq "the snapshot is a copy of the source" "" "$(diff -r "$src" "$dst/latest" 2>&1)"
sleep 1; "$here/backup-site.sh" -s "$src" -d "$dst" >/dev/null 2>&1
assert_eq "a second run adds a second snapshot" 2 "$(ls -d "$dst"/20* | wc -l)"
assert_eq "unchanged files are hard-linked between snapshots" "1" "$(stat -c %i "$dst"/20*/a.txt | sort -u | wc -l)"
dry=$tmp/dry; "$here/backup-site.sh" -s "$src" -d "$dry" -n >/dev/null 2>&1
assert_eq "--dry-run creates nothing" "no" "$([[ -e $dry ]] && echo yes || echo no)"
"$here/backup-site.sh" -s "$tmp/nope" -d "$tmp/d2" >/dev/null 2>&1; assert_status "bad source exits 4" 4 $?

# a fake rsync that fails, put first in PATH
mkdir "$tmp/fakebin"; printf '#!/bin/sh\nmkdir -p "$(eval echo \\${$#})"; exit 23\n' > "$tmp/fakebin/rsync"; chmod +x "$tmp/fakebin/rsync"
PATH="$tmp/fakebin:$PATH" "$here/backup-site.sh" -s "$src" -d "$tmp/d3" >/dev/null 2>&1
assert_status "a failing rsync exits 5" 5 $?
assert_eq "a failed run leaves no .partial folder" 0 "$(ls -A "$tmp/d3" | grep -c partial)"

printf '\n%d passed, %d failed\n' "$pass" "$fail"
(( fail == 0 ))
Points worth copying. Calls that end in exit are run in a subshell ( parse_args ... ) so that the exit ends only the subshell, and the status is read with $?. The tests use a temporary folder made by mktemp -d and removed by a trap, so a failed test run leaves nothing behind and never touches real data. Each assertion has a name that reads like a requirement. And the harness itself returns a status, so cron, a pre-commit hook or CI can use it.

$ ./test_backup.sh
ok    parse_args stores the options
ok    missing --source exits 2
ok    --keep 0 exits 2
ok    --keep abc exits 2
ok    unknown option exits 2
ok    prune_old keeps the newest 2
ok    prune_old leaves latest and .lock alone
ok    prune_old with fewer than KEEP removes nothing
ok    DEBUG is silent without --verbose
ok    DEBUG is shown with --verbose
ok    log appends to the log file
ok    an unwritable log file is switched off, not fatal
ok    a normal run exits 0
ok    the snapshot is a copy of the source
ok    a second run adds a second snapshot
ok    unchanged files are hard-linked between snapshots
ok    --dry-run creates nothing
ok    bad source exits 4
ok    a failing rsync exits 5
ok    a failed run leaves no .partial folder

20 passed, 0 failed
Twenty tests, all passing, and the whole run takes about a second and a half. Passing tests prove little until you have seen one fail, so the lab broke the script on purpose: a copy in which prune_old removes one snapshot too many (the loop bound i < n - keep changed to i < n - keep + 1):
$ ./test_backup.sh | grep -v '^ok'
FAIL  prune_old keeps the newest 2
        expected [2026-01-04_000001 2026-01-05_000001], got [2026-01-05_000001]
FAIL  prune_old with fewer than KEEP removes nothing
        expected [2], got [1]

18 passed, 2 failed
$ ./test_backup.sh >/dev/null; echo "exit status: $?"
exit status: 1
The two tests that cover the off-by-one failed, with the expected and the actual value, and the other 18 still passed. The exit status was 1, so a CI job or a git hook would stop. This is the point of the exercise: the next person to edit prune_old finds out in two seconds, not when a backup is missing.
What to test, in order of value: the failure paths (bad input, missing files, a failing command, a held lock, a signal), the exit codes (they are your interface), the pure functions (option parsing, name and path handling, anything with a loop or an off-by-one), idempotence (run it twice) and dry-run (it really changes nothing). Do not test the tools you call: rsync is not yours to test, but the way you call it is.

To test how the script reacts to a failing rsync, a full disk or a dead network, replace the command with a fake. The shell looks in PATH from left to right, so a fake first in PATH wins, for this one command only:
$ cat fakebin/rsync
#!/bin/bash
# fake rsync: create the destination (last argument), then fail or hang depending on FAKE_MODE
dest=${@: -1}; mkdir -p "$dest"; echo partial > "$dest/half-copied.file"
case ${FAKE_MODE:-fail} in
  fail) echo "rsync: fake error: connection unexpectedly closed" >&2; exit 23 ;;
  slow) sleep 6 ;;
esac
$ FAKE_MODE=fail PATH="$PWD/fakebin:$PATH" ./backup-site.sh -s src -d out2
The fake leaves a half-copied folder behind, as a real interrupted rsync would, so the test checks that the script cleans up. The same trick fakes date (to test time-dependent code), curl (an API that is down), df (a full disk), ssh and mail. The mock must be placed outside the script's folder, and the script must call the command by name and not by absolute path (/usr/bin/rsync would bypass the fake). For a device that always fails to write, /dev/full is built in: it was the cheapest way to test the "log disk is full" path.

  bash -n script.sh        parses without running: catches syntax errors (the first thing to run in a commit hook). It printed nothing for both files here, which means "syntax ok"
  shellcheck script.sh     finds quoting mistakes, unused variables, bad tests and many bashisms before you run anything; run it on the script and on the tests. ShellCheck is not installed on the lab machine, so it was not run on this code; install it with sudo apt install shellcheck, and expect a few notes about sourcing the script and about the printf in the fake rsync
  bats                     the Bash Automated Testing System (sudo apt install bats): tests are @test "name" { run ./script args; [ "$status" -eq 2 ]; [[ $output == *"usage"* ]]; } blocks, with setup and teardown functions and nicer reports, and run, $status and $output replace the hand-written harness. bats is not installed here either, so no output is shown; the hand-written harness above does the same job and has no dependency
  bash -x script.sh        traces each command (see the Error Handling page); to see the script as the tests see it, run a single test file with bash -x and read the failing assertion
  A CI job: on every commit run bash -n, shellcheck and the test script, so the failure paths stay green.

Three design ideas make this script safe to run from a scheduler, and each is worth copying:
  flock on a file descriptor: exec 9>"$dest_dir/.lock"; flock -n 9 || die ... opens the lock file on descriptor 9 for the life of the script and takes the lock with -n, "do not wait". The kernel drops the lock when the process ends, whatever the reason, so there are no stale locks. (A pid file that a crash leaves behind has to be checked for a live process at the next start.) Keep the lock in the folder you are protecting, not in a shared /tmp name that another user could create first.
  Build in a temporary name, rename when complete: rsync writes to .partial-STAMP and only mv puts it at its final name. A rename inside one filesystem is atomic, so anyone reading the snapshot folder sees either a complete snapshot or none, never half of one, and the EXIT trap deletes the partial folder if anything fails. The same idea works for a single file: write to a temp file in the same folder, then mv it over the real name. One small honesty point: ln -sfn removes and recreates the link, so a reader that looks at "latest" in the same millisecond could briefly find it missing. If that matters, create the new link under another name and rename it over the old one (ln -s TARGET latest.new && mv -T latest.new latest).
  Idempotence: running the script twice should leave the system in the same state as running it once. Here the second run adds a snapshot (that is the point of snapshots), but the content is identical, the lock and the .partial handling mean an interrupted run leaves nothing behind, and prune_old brings the count back to N. Scripts that append to files, add cron entries or create users need an explicit "already there?" test, or they grow a copy on every run.
A leftover .partial-* folder can still exist if the script was killed with kill -9 (no trap runs). Making the next run delete stale partial folders at its start, after it has the lock, is a cheap extra: find "$dest_dir" -maxdepth 1 -name '.partial-*' -exec rm -rf -- {} + (the lab did not add it, to keep the example short).

The exit codes and the log make the script scheduler-friendly. A crontab entry (see the cron page for the environment cron gives you; the script sets its own PATH needs by using only standard tools, and every path is absolute):
  30 2 * * * /usr/local/bin/backup-site.sh -s /srv/site -d /backup/site -l /var/log/backup-site.log
Output goes to the log file and to stderr, which cron mails if MAILTO is set, so a failure produces a message and a normal run does not (the log lines go to stderr as well; with a quiet job, send stderr to /dev/null and rely on the log file and the exit code instead, or log only WARN and above when stderr is not a terminal).
As a systemd service and timer (not run in the lab; shown as a starting point):
  # /etc/systemd/system/backup-site.service
  [Unit]
  Description=Snapshot the web site
  [Service]
  Type=oneshot
  ExecStart=/usr/local/bin/backup-site.sh -s /srv/site -d /backup/site
  # exit codes 3 (lock held) are not failures worth an alert:
  SuccessExitStatus=3
  # /etc/systemd/system/backup-site.timer
  [Timer]
  OnCalendar=*-*-* 02:30:00
  Persistent=true
  RandomizedDelaySec=10min
  [Install]
  WantedBy=timers.target
With systemd the stderr log lines land in the journal automatically (journalctl -u backup-site), TimeoutStopSec= sets how long TERM is given before KILL (remember the six-second delay in the TERM test), and a non-zero exit status marks the unit as failed so monitoring can alert on it. The meaningful exit codes pay off here: SuccessExitStatus and monitoring rules can treat "locked" differently from "rsync failed".

Loading the file in a test runs the whole job              the main guard is missing: if [[ ${BASH_SOURCE[0]} == "$0" ]]; then main "$@"; fi
A test that calls a function exits the whole test run      the function calls exit (die): run it in a subshell ( func args ) and read $?
Tests pass on your machine, fail in CI                     they depend on the current folder, the time zone, the locale, a real network or files in your home: use mktemp -d, set LC_ALL=C and a fixed TZ, and mock the network
Tests are slow because of sleep                            make the timestamp finer, or inject the clock (a function that returns the time that tests can replace), instead of sleeping for real
The ERR trap prints a strange extra line at exit           a trap function returned non-zero: end EXIT-trap functions with return 0 (or exit explicitly)
The cleanup did not run                                    the script was killed with SIGKILL (kill -9) or the machine lost power; no trap can help: make the next run clean up stale state
The signal handler runs late                               bash waits for the foreground command: run long commands in the background and wait, or accept the delay
Two runs overlap anyway                                    the lock file is on a different path for each run, or it is released too early: one fixed path, exec 9>file once, and keep the descriptor open until the end
The lock seems stuck after a crash                         with flock there is none: the kernel dropped it. If you used a pid file instead, check that the pid is alive (kill -0 PID)
--dry-run still changed something                          a command was not wrapped in run(): list every command that writes, deletes or sends, and route it through the wrapper; test with an assertion on the folder
The test passes but the job fails in cron                  cron's environment (minimal PATH, no HOME tricks, no terminal): run the script as cron does, env -i HOME=$HOME PATH=/usr/bin:/bin ./script ...
Everything looks fine and the backup is empty              nothing tests the content: always compare the result with the source (diff -r), as the end-to-end test here does

Start every serious script from a template like this one and delete what you do not need, instead of adding safety to a script after it has hurt you.
Wrap every command that changes state in one function, so --dry-run is truthful and a mock or a log line is one edit.
Give each failure its own exit code and its own sentence, and put both in --help.
Make the exit trap clean up and nothing else, and end it with return 0. Make the signal traps exit with 128 plus the signal number.
Build output under a temporary name and rename it into place; take a flock lock; never rely on "it will not be run twice".
Put the logic in functions behind a main guard, and write the tests for the failure paths first: bad input, missing source, a held lock, a failing command, a signal, a full disk (/dev/full).
Break your own script on purpose and watch the test fail: a test you have never seen fail is a test you cannot trust.
Run bash -n, ShellCheck and the tests on every change, in a commit hook or in CI, and keep the whole thing boring.