⏱️ Scheduling and Monitoring Scripts

Scheduling and monitoring your scripts: runners, locks, heartbeats and alerts
./run-job.sh backup --timeout 3600 --ping URL -- ./backup.sh

Writing a script that does the job is half the work. The other half is making sure it runs when it should, runs only once at a time, cannot hang forever, leaves a trail you can read, and, above all, that somebody finds out when it fails or when it silently stops running. Scheduled scripts fail quietly: nobody is watching, the output goes nowhere, and the first sign of trouble is the day you need the backup that has not existed for three months.
This page is about the layer around the script: a small RUNNER that wraps any command with a lock, a timeout, logging, a heartbeat and an alert; how to detect both a failure and silence; how to hook it into cron or a systemd timer; and how to test it. The cron and the systemd Units and Timers pages explain the schedulers themselves; the Production Script Template page shows how to write the job. Here is what connects them.
Everything below was run for real on Ubuntu 24.04 (bash 5.2, systemd 255) and the output is real. The "monitoring service" in the tests is a tiny HTTP listener on 127.0.0.1 that records every request it receives, so you can see exactly what a hosted heartbeat service would receive, with nothing sent anywhere. Not run, and marked so: anything involving real mail (neither sendmail nor mail is installed on the lab machine), external monitoring services, Prometheus, the catch-up behaviour of anacron and of systemd's Persistent=, RandomizedDelaySec, and the at command (not installed). The machine's name is shown as server1.

  The job never started: the scheduler is not running, the machine was off, a wrong schedule, a unit that is not enabled, a crontab replaced by accident
  It started and failed: a full disk, an expired password, an unreachable server, a changed API; the error went to a mailbox nobody reads or to /dev/null
  It started and hung: a stuck network call or a prompt waiting for input; the next runs pile up behind it or are skipped
  It overlapped itself: a slow run is still going when the next one starts, and the two corrupt each other's data
  It worked in your terminal and not in cron: a different environment (PATH, HOME, no terminal)
  It "succeeded" and did nothing: an exit status of 0 from a script that swallowed its errors, or a backup of an empty folder
  It slowly got worse: the run that took 5 minutes now takes 5 hours, and still "works"
  Nobody knows who owns it: the person who wrote it left, and the alert goes to nobody
The cure for the first group is monitoring that notices ABSENCE; for the others, a runner that records what happened and says so out loud.

Most people only monitor the first. A job that fails can report its own failure (the runner below does). A job that does not run cannot report anything, which is exactly when you need to know. So you need both:
  Failure alerts: pushed by the job when it ends badly (exit status, timeout, skipped)
  Absence alerts: raised by something OUTSIDE the job when an expected success does not arrive in time. This is the "dead man's switch" or heartbeat: the job pings a monitor when it succeeds, and the monitor raises the alarm if no ping arrives within the expected period plus a grace time
Absence alerts catch every item in the list above, including the scheduler being down or the whole machine being dead. Failure alerts alone cannot. If you only do one thing, do the heartbeat. Hosted services that work this way exist (Healthchecks, Cronitor, Dead Man's Snitch and similar), and a self-hosted monitoring system can do the same with a "last success" timestamp (section 9). The lab used a local listener in place of such a service, so the requests below are real, but no external service was contacted.

Use cron for a simple recurring one-liner that must run everywhere; use a systemd timer on a systemd machine when you want logs in the journal, no overlap, dependencies, and catch-up after downtime (the cron page's section 12 and the systemd Units and Timers page compare them in detail). Whichever you pick, CHECK the expression before you rely on it. systemd can tell you when an OnCalendar expression will fire next (no root needed):
$ systemd-analyze calendar "*-*-* 02:30:00" --iterations=2 | grep -E 'Normalized|Next'
Normalized form: *-*-* 02:30:00
Next elapse: Sat 2026-10-03 02:30:00 EDT
$ systemd-analyze calendar "Mon..Fri *-*-* 09:00" --iterations=2 | grep -E 'Normalized|Next'
Normalized form: Mon..Fri *-*-* 09:00:00
Next elapse: Mon 2026-10-05 09:00:00 EDT
$ systemd-analyze calendar "*:0/15" --iterations=2 | grep -E 'Normalized|Next'
Normalized form: *-*-* *:00/15:00
Next elapse: Fri 2026-10-02 18:45:00 EDT
$ systemd-analyze calendar "Sat *-*-1..7 03:00" --iterations=2 | grep -E 'Normalized|Next'
Normalized form: Sat *-*-01..07 03:00:00
Next elapse: Sat 2026-10-03 03:00:00 EDT
$ systemd-analyze calendar "weekly" --iterations=2 | grep -E 'Normalized|Next'
Normalized form: Mon *-*-* 00:00:00
Next elapse: Mon 2026-10-05 00:00:00 EDT
The "Normalized form" shows what systemd understood (note that "weekly" means Monday at midnight, and that the weekday-and-day-range expression is the classic "first Saturday of the month"). cron syntax is NOT accepted, and the error message does not tell you why:
$ systemd-analyze calendar "*/5 * * * *"
Failed to parse calendar specification '*/5 * * * *': Invalid argument
$ systemd-analyze calendar "Mon..Fry 09:00"
Failed to parse calendar specification 'Mon..Fry 09:00': Invalid argument
The first is cron syntax pasted into a timer (the systemd equivalent of every five minutes is OnCalendar=*:0/5); the second is a typo. Always run the check, and write the "Next elapse" in the ticket so the next person can verify the intent.
Two timer facts that surprised the lab. First, a timer's default accuracy is one MINUTE: systemd may fire it up to a minute late to save wake-ups, so a job meant for 03:00:00 can run at 03:00:47. Second, in the first attempt to test a "3 seconds from now" timer, nothing had happened when the lab checked 5 seconds later; the default accuracy explains it:
$ systemd-analyze timespan 1min | grep -E 'Original|μs'
Original: 1min
      μs: 60000000
For jobs that must start on time set AccuracySec=1s in the timer (and for jobs on many machines at once, the opposite: RandomizedDelaySec spreads the load, not run here). And a timer does not pile up copies of a slow job. A timer was set to fire every second for a job that takes four seconds; in seven seconds the journal shows the job started twice (once at the start, once after the first finished) and never overlapped itself:
$ systemd-run --user --on-active=1s --on-unit-active=1s --timer-property=AccuracySec=100ms --unit=demo-overlap bash -c 'echo job-start; sleep 4; echo job-end'
(after 7 seconds, from the journal)
      2 job-start
      1 job-end
(The second run was still in progress when the lab stopped the test, so one job-end is missing.) A systemd timer does not start a service that is still running; cron has no such rule, which is why the runner below takes its own lock.

This is the whole runner (61 lines), exactly as tested. It runs any command with a lock, a timeout, heartbeat pings, a status file, and a rate-limited alert. Read it first, then the notes.
$ cat run-job.sh
#!/usr/bin/env bash
# run-job.sh: run a scheduled command with a lock, a timeout, logging, heartbeat pings and rate-limited alerts
# usage: run-job.sh NAME [--timeout SECS] [--ping URL] [--alert-url URL] [--alert-every SECS] [--state DIR] -- COMMAND [ARGS...]
# exit:  the command's own status; 124 on timeout; 75 when skipped because the previous run is still going
set -Eeuo pipefail

name=${1:?usage: run-job.sh NAME [options] -- COMMAND...}; shift
timeout_s=3600 ping="" alert_url="" alert_every=3600 state=${XDG_STATE_HOME:-$HOME/.local/state}/run-job
while (( $# )); do
  case $1 in
    --timeout)     timeout_s=$2; shift 2 ;;
    --ping)        ping=$2; shift 2 ;;
    --alert-url)   alert_url=$2; shift 2 ;;
    --alert-every) alert_every=$2; shift 2 ;;
    --state)       state=$2; shift 2 ;;
    --)            shift; break ;;
    *) echo "run-job.sh: unknown option $1" >&2; exit 2 ;;
  esac
done
(( $# )) || { echo "run-job.sh: no command given" >&2; exit 2; }
mkdir -p "$state"

log()  { local line; line="$(date '+%F %T') $name: $*"; echo "$line" >&2; logger -t "job-$name" -- "$*" 2>/dev/null || true; }
ping_url() {   # a failed ping must never fail the job
  [[ -n $ping ]] || return 0
  curl -fsS -m 5 --retry 1 -o /dev/null "$@" 2>/dev/null || log "WARN: could not reach the monitor ($1)"
}

exec 9>"$state/$name.lock"
if ! flock -n 9; then log "SKIPPED: the previous run is still going"; ping_url "$ping/log" --data-binary "skipped: previous run still going"; exit 75; fi

start=$(date +%s); out=$(mktemp)
trap 'rm -f -- "$out"' EXIT
log "START: $*"; ping_url "$ping/start"
set +e; timeout -k 5 "$timeout_s" "$@" >"$out" 2>&1; rc=$?; set -e
end=$(date +%s); dur=$(( end - start ))
case $rc in 0) result=success ;; 124) result=timeout ;; *) result=failure ;; esac
tail -n 20 "$out" | while IFS= read -r l; do log "  | $l"; done

# one JSON status file per job, replaced atomically (a monitor or a human can read it any time)
jq -n --arg job "$name" --arg host "$(hostname -s)" --arg result "$result" --argjson rc "$rc" \
      --argjson start "$start" --argjson end "$end" --argjson dur "$dur" \
      '{job:$job, host:$host, result:$result, exit:$rc, started:$start, ended:$end, duration_s:$dur}' > "$state/$name.status.tmp"
mv -- "$state/$name.status.tmp" "$state/$name.status.json"

alert() {   # alert KIND MESSAGE: POST a JSON message to a webhook
  [[ -n $alert_url ]] || return 0
  local payload; payload=$(jq -n --arg t "[$(hostname -s)] $name: $1 - $2" '{text:$t}')
  curl -fsS -m 5 -H 'Content-Type: application/json' -d "$payload" -o /dev/null "$alert_url" 2>/dev/null || log "WARN: could not send the alert"
}
if [[ $result == success ]]; then
  date +%s > "$state/$name.last_success"
  log "OK: finished in ${dur}s"; ping_url "$ping"
  if [[ -e $state/$name.alerted ]]; then alert RECOVERED "succeeded again after failing"; rm -f -- "$state/$name.alerted"; fi
else
  log "FAIL: $result (exit $rc) after ${dur}s"; ping_url "$ping/fail" --data-binary "@$out"
  last=0; [[ -e $state/$name.alerted ]] && last=$(<"$state/$name.alerted")
  if (( end - last >= alert_every )); then alert FAILED "$result (exit $rc) after ${dur}s"; echo "$end" > "$state/$name.alerted"
  else log "alert suppressed (last one $(( end - last ))s ago, limit ${alert_every}s)"; fi
fi
exit "$rc"
How it works, part by part:
  Options: a job NAME (it names the lock, status and state files) and a few options before --, then the command after --. Defaults: a one-hour timeout and a state folder under ~/.local/state/run-job. It exits 2 for wrong usage, like every well-behaved tool.
  log(): every message goes to stderr with a time stamp AND to the journal (logger -t job-NAME), so you can read it live and later. The logger call is allowed to fail (|| true): losing a log line must never fail the job.
  ping_url(): the heartbeat. It uses curl with a 5-second limit and one retry (-fsS: fail on HTTP errors, no progress meter, show errors), and a failure to reach the monitor only logs a warning. A monitoring outage must never break the thing being monitored.
  The lock: exec 9>FILE opens a lock file on descriptor 9 and flock -n 9 takes the lock without waiting. If another run holds it, this one logs "SKIPPED", tells the monitor, and exits 75 (EX_TEMPFAIL, "try again later"). The kernel releases the lock when the process ends for ANY reason, even kill -9, so there is no stale lock to clean up.
  The run: set +e around timeout -k 5 "$timeout_s" "$@" so the exit status can be examined instead of aborting the wrapper; the command's output goes to a temp file (removed by the EXIT trap) so the last lines can be logged and attached to the failure ping. timeout sends TERM at the limit and KILL 5 seconds later if the command ignores TERM, and returns 124 for a timeout.
  The result: 0 is success, 124 timeout, anything else failure. A JSON status file (job, host, result, exit, start, end, duration) is written with jq and moved into place atomically, so a monitor reading it never sees a half-written file.
  Success: the time of the last success is recorded (this is what the staleness check reads), the success ping is sent, and if the previous run had alerted, a RECOVERED message goes out and the alert state is cleared.
  Failure: the failure ping carries the job's output as its body; the alert is sent only if no alert was sent in the last alert_every seconds (default an hour), so a job that fails every minute produces one alert, not sixty.
  The exit status of the wrapper is the exit status of the command (or 124, or 75), so cron, systemd and your own scripts see the truth.
It needs bash, jq, curl, flock, timeout and logger (all present on a normal Ubuntu server; the lab checked that they are in /usr/bin).

All seven ran against the local listener on port 18200. After each run the lab printed what the "monitor" received. Time stamps are real; the host name is shown as server1.
1. A successful run: the monitor gets the start ping and the success ping, and the job leaves a status file and a last-success time:
$ ./run-job.sh backup --state state --ping http://127.0.0.1:18200/ping/backup --alert-url http://127.0.0.1:18200/alert -- bash -c 'echo "copied 120 files"; sleep 0.3'
2026-10-02 18:35:11 backup: START: bash -c echo "copied 120 files"; sleep 0.3
2026-10-02 18:35:11 backup:   | copied 120 files
2026-10-02 18:35:11 backup: OK: finished in 0s
[exit 0]
  monitor received:
      GET /ping/backup/start
      GET /ping/backup
$ jq -c . state/backup.status.json
{"job":"backup","host":"server1","result":"success","exit":0,"started":1790980511,"ended":1790980511,"duration_s":0}
(The duration shows 0 seconds because the lab's whole-second clock rounded the 0.3 s sleep down; real jobs are longer.)
2. A failing run (exit 3): the failure ping carries the job's output, and an alert goes out:
$ ./run-job.sh backup ... -- bash -c 'echo "connecting to db1"; echo "error: connection refused" >&2; exit 3'
2026-10-02 18:35:11 backup: START: bash -c echo "connecting to db1"; echo "error: connection refused" >&2; exit 3
2026-10-02 18:35:11 backup:   | connecting to db1
2026-10-02 18:35:11 backup:   | error: connection refused
2026-10-02 18:35:11 backup: FAIL: failure (exit 3) after 0s
[exit 3]
  monitor received:
      GET /ping/backup/start
      POST /ping/backup/fail connecting to db1 | error: connection refused |
      POST /alert { |   "text": "[server1] backup: FAILED - failure (exit 3) after 0s" | }
3. The same job fails again a second later: the failure ping is sent again (the monitor must know), but the alert is suppressed:
2026-10-02 18:35:12 backup: FAIL: failure (exit 3) after 0s
2026-10-02 18:35:12 backup: alert suppressed (last one 1s ago, limit 3600s)
  monitor received:
      GET /ping/backup/start
      POST /ping/backup/fail
4. The job works again: a single RECOVERED message, so the person who was alerted knows it is over:
  monitor received:
      GET /ping/backup/start
      GET /ping/backup
      POST /alert { |   "text": "[server1] backup: RECOVERED - succeeded again after failing" | }
5. A job that hangs (the command sleeps 30 seconds, the limit is 1 second): killed, reported as a timeout, exit status 124:
$ ./run-job.sh backup ... --timeout 1 -- bash -c 'echo started; sleep 30'
2026-10-02 18:35:12 backup: START: bash -c echo started; sleep 30
2026-10-02 18:35:13 backup:   | started
2026-10-02 18:35:13 backup: FAIL: timeout (exit 124) after 1s
[exit 124]
$ jq -c '{result,exit,duration_s}' state/backup.status.json
{"result":"timeout","exit":124,"duration_s":1}
6. The previous run is still going (a three-second job was started first): the second invocation is skipped with exit 75, and the monitor is told:
2026-10-02 18:35:14 backup: SKIPPED: the previous run is still going
[exit 75]
  monitor received:
      GET /ping/backup/start
      POST /ping/backup/log skipped: previous run still going
      GET /ping/backup
(The start and the final success ping belong to the first, slow run; the log ping is the skipped one.)
7. The monitor is unreachable (the lab pointed it at a closed port): the job still runs and succeeds, with a warning:
2026-10-02 18:35:16 backup: START: bash -c echo work done
2026-10-02 18:35:16 backup: WARN: could not reach the monitor (http://127.0.0.1:1/ping/backup/start)
2026-10-02 18:35:16 backup:   | work done
2026-10-02 18:35:16 backup: OK: finished in 0s
2026-10-02 18:35:16 backup: WARN: could not reach the monitor (http://127.0.0.1:1/ping/backup)
[exit 0]
These cover the failure modes that matter: normal success, failure with output, repeated failure without an alert storm, recovery, hang, overlap, and a broken monitor.

Why a lock: two copies of a backup, an import or a cleanup writing to the same place corrupt it. Why flock: it locks an open file descriptor, so the kernel frees the lock when the process dies, with no stale lock files and no race between "check" and "create" (the cron page's section 9 shows the one-line form for a crontab). Why exit 75: it is the standard "temporary failure" status (EX_TEMPFAIL from sysexits.h), so a scheduler can treat "skipped because busy" differently from a real failure. systemd can be told so, and the lab tested it:
$ systemd-run --user --wait --collect -p SuccessExitStatus=75 --unit=demo-skip bash -c "exit 75"
Running as unit: demo-skip.service; invocation ID: ...
Finished with result: success
Main processes terminated with: code=exited/status=75
$ systemd-run --user --wait --collect --unit=demo-fail2 bash -c "exit 75"        (without SuccessExitStatus)
Finished with result: exit-code
Main processes terminated with: code=exited/status=75
The same exit status 75 was a success with SuccessExitStatus=75 and a failure without it. Put SuccessExitStatus=75 in the unit that runs the runner so that a skipped run does not trigger OnFailure alerts, and decide how long a skip may go on: one skip is normal, but "skipped for six hours in a row" means the first run is stuck, and the timeout (next section) and the staleness check (section 9) are what catch it.
Where to put the lock file: in a directory only the job's user can write (the state folder, or /run/lock/NAME for system jobs), never in a shared /tmp name another user could create first.

Every unattended job needs an upper bound. Without one, a hang holds the lock and every later run is skipped for ever, and nothing alerts because "it is still running". Two ways, both tested:
  timeout(1): timeout -k 5 3600 command sends TERM after 3600 seconds and KILL 5 seconds after that, and returns 124. It is what the runner uses; it works under cron and in any script.
  systemd: RuntimeMaxSec= in the service unit. A job that would run for 30 seconds was given 2:
$ systemd-run --user --wait --collect -p RuntimeMaxSec=2 --unit=demo-limit sleep 30
Finished with result: timeout
Main processes terminated with: code=killed/status=TERM
Service runtime: 2.172s
$ timeout -k 1 2 sleep 30; echo "exit $?"
exit 124
Both stopped the job at 2 seconds. systemd reports "result: timeout" and, by its documented default (KillMode=control-group, not tested separately here), stops every process in the unit's control group, the job and any helper processes it started; timeout(1) signals the command it started, so check what your job's children do when their parent is killed. Choose the limit from data: two or three times the longest normal run (the status file's duration_s records it). Too tight a limit kills a healthy job on a busy night; too loose a limit defeats the purpose.

Good job logs answer three questions after the fact: did it run, what did it do, how did it end. The runner logs the start, the last 20 lines of the job's output, the result and the duration, to the terminal (for cron's mail) and to the journal under the tag job-NAME:
$ journalctl -t job-backup -o short-iso --no-pager | tail -n 4
2026-10-02T18:35:16-04:00 server1 job-backup: START: bash -c echo work done
2026-10-02T18:35:16-04:00 server1 job-backup: WARN: could not reach the monitor (http://127.0.0.1:1/ping/backup/start)
2026-10-02T18:35:16-04:00 server1 job-backup:   | work done
2026-10-02T18:35:16-04:00 server1 job-backup: OK: finished in 0s
$ journalctl -t job-backup --no-pager -o cat -g 'FAIL|SKIPPED'
FAIL: failure (exit 3) after 0s
FAIL: failure (exit 3) after 0s
FAIL: timeout (exit 124) after 1s
SKIPPED: the previous run is still going
One tag per job means a single command lists a job's whole history, and -g filters by text (see the journalctl page for more). For a job run by systemd, standard output and error go to the journal automatically, with the unit name as the tag (journalctl -u NAME); run by cron, they go to cron's mail unless you redirect them (see the cron page, section 6).
If a script must send everything it prints to the journal, the common idiom is exec > >(logger -t NAME) 2>&1 at the top. The process substitution runs in the background and the shell does not wait for it, so in theory the last lines can be lost if the script exits first. The lab ran a three-line script 30 times with the idiom and 30 times with a pipeline, and in both cases every line arrived (a race that did not show up here is still a race, and a loaded machine behaves differently). The pipeline form is deterministic, because the shell waits for every command in it:
$ { echo "line 1"; echo "line 2"; echo "final line"; } 2>&1 | logger -t NAME
Keep the machine-readable record too: the status JSON file (last run, result, exit status, duration) is cheap, atomic, and readable by anything, which brings us to monitoring. Do not log secrets (command lines with passwords, tokens in URLs): the runner logs the command it runs, so pass secrets through the environment or a file, not as arguments.

Heartbeat pings, as in the scenarios above, are the simplest monitor: the job tells an outside service "I started", "I succeeded" or "I failed" by requesting a URL. The conventions used by hosted services are /start, the bare URL for success, /fail for failure and sometimes /log for a message; the service raises an alert when the success ping does not arrive within the period you set plus a grace time, or when it gets /fail. The lab's local listener showed exactly these requests (scenarios 1 to 6). Three points: send the start ping too (the service can then measure duration and tell "never started" from "started and hung"); attach the failure output as the body (scenario 2: the monitor received "connecting to db1 | error: connection refused"); and never let a failed ping fail the job (scenario 7). The request must have a short timeout, since a hung monitor would otherwise hang the job.
Staleness checks are the same idea without a service: any monitoring system that can run a command can check how old the last success is. This is a short checker, tested for real:
$ cat check-stale.sh
#!/bin/bash
# check-stale.sh FILE MAX_AGE_SECONDS: exit 0 if FILE exists and is newer than MAX_AGE, 2 if stale, 3 if missing
f=${1:?usage: check-stale.sh FILE MAX_AGE_SECONDS}; max=${2:?}
[[ -e $f ]] || { echo "CRITICAL: $f does not exist (the job has never succeeded)"; exit 3; }
age=$(( $(date +%s) - $(stat -c %Y -- "$f") ))
if (( age > max )); then echo "STALE: $f is ${age}s old (limit ${max}s)"; exit 2; fi
echo "OK: $f is ${age}s old (limit ${max}s)"
$ ./check-stale.sh state/backup.last_success 86400; echo "[exit $?]"
OK: state/backup.last_success is 18s old (limit 86400s)
[exit 0]
$ touch -d '3 days ago' state/old.last_success; ./check-stale.sh state/old.last_success 86400; echo "[exit $?]"
STALE: state/old.last_success is 259200s old (limit 86400s)
[exit 2]
$ ./check-stale.sh state/never.last_success 86400; echo "[exit $?]"
CRITICAL: state/never.last_success does not exist (the job has never succeeded)
[exit 3]
The exit codes (0, 2, 3) follow the Nagios convention (0 OK, 1 warning, 2 critical, 3 unknown), so the same script can be a check in Nagios, Icinga, or a cron job on ANOTHER machine that mails you; the point is that something outside the job decides. Set the limit to the schedule plus a margin: for a nightly job, 26 or 27 hours, not 24.
The status JSON adds the last RUN (success or not) and how long it took:
$ jq -r '"last run: \(.result) (exit \(.exit)) \(now - .ended | floor)s ago, took \(.duration_s)s"' state/backup.status.json
last run: success (exit 0) 18s ago, took 0s
$ jq -e '.duration_s < 600' state/backup.status.json >/dev/null && echo "duration within 600s" || echo "SLOW"
duration within 600s
jq -e makes the comparison an exit status (see the jq page), so a slow-job check is one line. A job whose duration doubles is often about to fail; watching the trend catches it a week early.
With Prometheus (not run here) the usual way is the node_exporter textfile collector: the job writes a file such as job_last_success_timestamp_seconds{job="backup"} 1790980511 and job_duration_seconds{job="backup"} 312 into the collector's directory (write to a temp name and rename, like the status file), and an alert rule fires when time() - job_last_success_timestamp_seconds exceeds the limit. The principle is the same: expose the last success time and alert on its age.
Finally, a warning about what is being monitored: "the backup job succeeded" is not "the backup can be restored". Test restores on a schedule and monitor THAT too.

The runner posts a JSON message to a webhook (the format used by Slack-style incoming webhooks and many chat tools). The lab built the message with jq, never by pasting text (see the jq page for why), and the listener received this body:
POST /alert { |   "text": "[server1] backup: FAILED - failure (exit 3) after 0s" | }
(the receiver shows the pretty-printed body on one line, with " | " standing for the line breaks). Rules for alerts that get acted on:
  Say WHAT (the job), WHERE (the host), HOW (failed, timed out, skipped, stale) and WHEN, and give the next step: the journalctl command that shows the log, or a link to the runbook. "backup failed" at 03:00 is a bad message; "backup on web1 failed (exit 3) at 03:00: journalctl -t job-backup" is a useful one
  One alert per incident, not per failure. The runner's rate limit (scenario 3) and its RECOVERED message (scenario 4) are the minimum; a good alerting system also groups related alerts and escalates if nobody acknowledges
  Alert on absence (heartbeat or staleness), not only on failure
  Route by importance: a failed nightly report is a ticket, a failed backup is a page; too many alerts train people to ignore them all
  Test the alert path itself, regularly: run a job that is meant to fail and check that the right person receives the message, on the right channel. An alert that goes to a mailbox nobody reads is the same as no alert
  Name an owner for every job (in the job's comment, the runbook or the alert text)
  Mail: cron mails output to MAILTO if a mail system is installed (see the cron page). Neither mail nor sendmail is installed on the lab machine, so a mail alert was not tested here; if you use mail, check that the server can really send (and that the message does not land in spam) before you depend on it
Be careful with the webhook URL itself: it is a secret (anyone who has it can post to your channel). Keep it in a file or the environment with restricted permissions, not on the command line (which every user can read in ps) and not in a repository.

A systemd timer starts a service, so the runner goes in the service. The lab started the runner from a transient timer ("in 2 seconds, run this command") with AccuracySec=1s and watched the listener:
$ systemd-run --user --on-active=2s --timer-property=AccuracySec=1s --unit=demo-timed --collect ./run-job.sh timed --state state_t --ping http://127.0.0.1:18200/ping/timed -- bash -c 'echo "hello from the timer"'
Running timer as unit: demo-timed.timer
Will run service as unit: demo-timed.service
$ systemctl --user list-timers --no-pager | grep -E 'NEXT|demo-timed'
NEXT                        LEFT LAST                         PASSED UNIT
Fri 2026-10-02 18:36:31 EDT   1s -                                 - demo-timed.timer
(after the timer fired) monitor received:
   GET /ping/timed/start
   GET /ping/timed
$ journalctl -t job-timed --no-pager -o cat
START: bash -c echo "hello from the timer"
  | hello from the timer
OK: finished in 0s
The timer started the service, the service ran the wrapper, and the monitor received the start and success pings; the log is in the journal under the wrapper's own tag. As permanent files (not run in the lab) the pair looks like this, with the runner as the command, the exit status 75 treated as success, and a hook for the case where even the runner fails:
  # /etc/systemd/system/backup-site.service
  [Unit]
  Description=Nightly site backup (via run-job)
  OnFailure=alert@%n.service
  [Service]
  Type=oneshot
  ExecStart=/usr/local/bin/run-job.sh backup --timeout 3600 --ping https://monitor.example/ping/UUID -- /usr/local/bin/backup-site.sh -s /srv/site -d /backup/site
  SuccessExitStatus=75
  # /etc/systemd/system/backup-site.timer
  [Timer]
  OnCalendar=*-*-* 02:30:00
  AccuracySec=1min
  RandomizedDelaySec=10min
  Persistent=true
  [Install]
  WantedBy=timers.target
Persistent=true makes a timer that was missed because the machine was off run once at the next boot (not run here; it is the systemd answer to anacron). AccuracySec and RandomizedDelaySec here say "any time in a ten-minute window after 02:30", which is right for a backup and wrong for something that must be on time.
OnFailure= is the hook for failures the runner cannot report: a unit that fails to start, a wrong path, a crash of the wrapper itself. It starts another unit when this one fails. The lab tested it with a template unit that writes a line to a file:
$ cat alert@.service
[Unit]
Description=Alert: %i failed
[Service]
Type=oneshot
ExecStart=/bin/bash -c 'echo "$(date +%%T) ALERT: unit %i failed" >> /path/to/alerts.log'
$ systemd-run --user --wait --collect --unit=demo-onfail -p OnFailure=alert@demo-onfail.service bash -c 'echo "boom: disk full" >&2; exit 3'
Finished with result: exit-code
Main processes terminated with: code=exited/status=3
$ cat alerts.log
18:36:06 ALERT: unit demo-onfail failed
$ journalctl --user -u demo-onfail --no-pager -o cat
Started demo-onfail.service - /usr/bin/bash -c "echo \"boom: disk full\" >&2; exit 3".
boom: disk full
demo-onfail.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
demo-onfail.service: Failed with result 'exit-code'.
The failing job started the alert unit, whose name carries the failed unit (%i, the instance part of alert@NAME.service). A job that SUCCEEDS does not trigger it (the lab's second run left alerts.log empty). The failed job's own output ("boom: disk full") is in the journal for the failed unit, so the alert can say where to look. In a real alert@.service, replace the echo with your webhook call or mail command. A systemd unit file typo is easy to make and silent until the first run; check files before you install them:
$ systemd-analyze verify good.service        (a correct unit: no output, exit status 0)
$ systemd-analyze verify bad.service
bad.service:6: Failed to parse sec value, ignoring: banana
bad.service: Command /nonexistent/backup.sh is not executable: No such file or directory
verify caught both a bad value and a missing program before any timer ever ran. Run it for every new or changed unit (for system units without --user).

A job that was only ever run by hand is untested. Check each of these before you trust a schedule:
  Run it as the scheduler will: with the scheduler's environment. cron gives a minimal one (the cron page, section 4). The runner was tried with an empty environment and a short PATH, no terminal and no input, as cron would run it:
$ env -i HOME="$HOME" PATH=/usr/bin:/bin ./run-job.sh cronlike --state state_c -- bash -c 'echo "PATH inside the job: $PATH"; command -v jq curl flock timeout logger | wc -l | sed "s/^/tools found: /"' < /dev/null
cronlike: START: ...
cronlike:   | PATH inside the job: /usr/bin:/bin
cronlike:   | tools found: 5
cronlike: OK: finished in 0s
It worked, and all five tools it needs were found in /usr/bin. And what a missing tool looks like when PATH is wrong:
$ env -i HOME="$HOME" PATH=/nonexistent ./run-job.sh cronlike2 --state state_c -- true < /dev/null
/usr/bin/env: 'bash': No such file or directory
[exit 127]
With a broken PATH even the interpreter of the runner could not be found: exit status 127, no log, no ping. This is the case where ONLY the external heartbeat helps: the job could not even report that it failed. For a systemd unit, test with systemctl start NAME.service (a service started by hand behaves like one started by the timer) and systemd-run --user --wait (a transient copy), and read the result with systemctl status and journalctl -u NAME.
  Force the failure paths: a command that exits non-zero, one that hangs (to see the timeout), one that is slow (to see the overlap skip), a monitor that is unreachable (scenario 7). Use fake commands placed early in PATH to simulate a full disk or a dead network (the Production Script Template page shows the technique).
  Test the alerts end to end: break something on purpose and watch the message arrive where a human will read it, then fix it and see the RECOVERED message.
  Test the silence: stop the scheduler, or disable the timer, and check that the heartbeat monitor or the staleness check alerts after the grace time. This is the test that most setups never do.
  Test the restore, not only the backup. Put it in the calendar.
  Write down the schedule with its check (systemd-analyze calendar "..." output) and the owner, and review the list of scheduled jobs every few months: remove the ones nobody can explain.

The job never runs                                         timer not enabled or not started (systemctl list-timers --all), cron service down, wrong user's crontab, a bad OnCalendar (check with systemd-analyze calendar), a unit with a typo (systemd-analyze verify)
A job is "late" or runs at a slightly different time       systemd's default AccuracySec is one minute: set AccuracySec=1s if the time matters; RandomizedDelaySec adds delay on purpose
systemd-analyze calendar says "Invalid argument"           the expression is cron syntax or has a typo: systemd needs forms like *-*-* 02:30:00 or *:0/15
Two copies of the job run at once                          cron has no overlap protection: take a lock with flock -n (the runner does); systemd does not start a second copy of a running service
Every later run is skipped for hours                       the first run hangs and holds the lock: add a timeout; alert on staleness, not just on failure
The job works by hand and fails from cron or a timer       a different environment: test with env -i and a short PATH, use absolute paths, no terminal, no prompt, no ssh agent
No alert although the job is failing                       the alert goes to a mailbox nobody reads, a webhook URL expired, or the alert channel was never tested: send a test alert every month
Alerts every minute (alert storm)                          no rate limit or grouping: send one alert per incident, then a recovery message (scenarios 3 and 4)
The heartbeat monitor alerts although the job ran          the grace time is shorter than the job's runtime or the timer's delay: set period = schedule interval, grace = longest normal run plus the accuracy window
The job fails because the monitor is down                  the ping is not allowed to fail the job: use a short timeout and || (the runner's ping_url, scenario 7)
Exit status 0 but nothing was done                         the script swallowed its errors: use set -Eeuo pipefail and check the result (the number of files, the size of the backup, the age of the newest record) before reporting success
Missed runs after a reboot are never made up               cron does not catch up: use systemd Persistent=true, or anacron on machines that are not always on
The log of a cron job is empty                             its output was discarded (> /dev/null) or mailed to a user with no mailbox: log to the journal with logger or use the runner
A run is "skipped" every time                              a stale process still holds the lock (check with fuser or lsof on the lock file), or the previous run really is stuck: look at what it is doing

Monitor absence, not just failure: a heartbeat or a staleness check catches the scheduler being down, the machine being off and the job never starting.
Wrap every unattended job in a runner: a lock, a timeout, logging, a status file, a failure ping and a rate-limited alert. Write it once and use it everywhere.
Let the ping and the alert fail without failing the job, and give every network call a short timeout.
Use exit status 75 for "skipped, still running", teach systemd with SuccessExitStatus=75, and let a timeout and a staleness check catch a stuck run.
Check every schedule with systemd-analyze calendar and every unit with systemd-analyze verify before relying on them; remember the one-minute default accuracy of timers.
One alert per incident with host, job, reason, time and the next step; one recovery message; test the alert path monthly.
Log to the journal under one tag per job, keep a machine-readable status file, and watch the duration trend as well as the result.
Test as the scheduler runs it (a minimal environment, no terminal), force every failure path, stop the scheduler on purpose to see the silence alert, and test restores, not just backups.
Give every job an owner and a place in a list that is reviewed. A scheduled job nobody owns is a failure waiting to be noticed too late.