Practical recipes for engineers and sysadmins: heredocs, tee, dates, text tools, loops, scripts, logs, SSH and more
▶Prefer to watch? This short video runs this page's main commands on a real Ubuntu 24.04 machine, with on-screen captions and no voiceover. Click to play, or watch it on YouTube.
Command Line Handbook: practical recipes for engineers and sysadmins
man bash (and when you forget a flag: command --help | less)
This is the handbook for the commands you reach for every day and the little tricks that save the other twenty minutes: writing files with here documents and tee, dates and timestamps, redirection, counting and sorting, running things on many files, safe scripts, logs, network checks, hashing, JSON, and background jobs.
It is written for people who are new to the shell and for people who have used it for twenty years and still look things up. Each recipe has a one-line "why" and a real example.
Every example below was actually run (on Ubuntu 24.04 with bash 5.2 and GNU tools, in a scratch folder, with the clock set to UTC) and the output you see is what the machine printed, not what the author remembered. Examples that cannot be run safely here (anything that changes the system, needs root, or prints your own machine's details) are shown without output, or with output marked as an illustration in a comment.
The sheets for grep, awk, sed, find, tar, curl, tmux and the shells go deeper on their own tools; this handbook is the glue between them.
In the examples, a line starting with "$ " is what you type, "> " is the shell asking for the rest of a multi-line command (you do not type it), and everything else is the output. /home/demo is the scratch home folder used for the examples.
Platforms: GNU/Linux first. macOS and BSD ship older, different versions of date, sed, xargs and others: where a recipe differs you will see a note. On macOS, brew install coreutils gnu-sed gives you the GNU versions (as gdate, gsed and so on).
I want to ... (the task index)
Write a multi-line file from a script or the prompt
Here documents: cat > file <<'EOF'
Write a file that needs root
tee with sudo: sudo tee /etc/file >/dev/null <<'EOF'
Feed a block of text to any command
Here documents and here strings: python3 - <<'PY'
See output AND save it
tee: cmd | tee file
Throw away errors, or capture them separately
Redirection: cmd 2>/dev/null cmd > out 2> err
Compare the output of two commands
Process substitution: diff <(cmd1) <(cmd2)
Put a timestamp in a filename or a log line
Dates: $(date +%Y%m%d-%H%M%S)
Convert epoch seconds to a date, and back
Dates: date -d @1700000000 date -d '...' +%s
Work out "90 days ago", "next Friday", "last day of the month"
Dates: date -d '2026-03-08 -90 days'
Count the days between two dates
Dates: $(( (end - start) / 86400 ))
Show a time in other time zones
Dates: TZ=Asia/Tokyo date
Find the biggest files and folders
Files and disk: du -ah . | sort -rh | head
Find files changed in a time window
Files and disk: find . -newermt '2026-01-01'
See permissions as a number, or the real path of a link
Files and disk: stat -c '%a %n' readlink -f
Count how often things appear (top IPs, errors, users)
Text toolkit: sort | uniq -c | sort -rn
Turn messy output into neat columns
Text toolkit: column -t
Remove duplicates but keep the original order
Text toolkit: awk '!seen[$0]++'
Run a command on many files, safely, in parallel
xargs, find and loops: find ... -print0 | xargs -0 -P4
Retry until it works, wait for a port, process a CSV
Loops and batch jobs: until cmd; do sleep 1; done
Make a script safe to run twice (lock, cleanup, errors)
Script recipes: flock trap set -euo pipefail
See what is using the CPU, memory, disk or a port
System inspection: ps ss -tulpn du
Read the logs of the last hour for one service
Logs: journalctl -u nginx --since '1 hour ago'
Copy files between machines
SSH and transfers: rsync -avh src/ host:dst/
Hash a file, encode or decode, make a random password
Hashes, encoding and random data: sha256sum base64 openssl rand
Pick values out of JSON
JSON and CSV: jq -r '.items[].name'
Run something later, or keep it running after logout
Background and scheduled work: nohup at systemd-run flock
Avoid the classic "oops" moments
Safety habits: -- "$var" ${var:?} echo first
Here documents (heredocs): multi-line text without a text editor
A here document gives a command several lines of text as its standard input. You write the lines right in the script or at the prompt, ending with a line that contains only the delimiter word (EOF by convention, but any word works).
Unquoted delimiter: variables and command substitutions ARE expanded
$ cat <<EOF
> Hello from $USER
> Your home folder is $HOME
> 2 + 2 = $((2 + 2))
> EOF
Hello from demo
Your home folder is /home/demo
2 + 2 = 4
Quoted delimiter: the text is taken literally (this is almost always what you want for config files and scripts)
$ cat <<'EOF'
> No expansion here: $USER $(date) `whoami` $((2 + 2))
> EOF
No expansion here: $USER $(date) `whoami` $((2 + 2))
Rule of thumb: quote the delimiter ('EOF') unless you deliberately want the shell to fill in values. Then a stray $ or backtick in your text cannot surprise you.
Write a file
$ cat > app.conf <<'EOF'
> listen_port = 8080
> log_level = info
> EOF
$ cat app.conf
listen_port = 8080
log_level = info
Append to a file (>> instead of >)
$ cat >> app.conf <<'EOF'
> timeout = 30
> EOF
$ cat app.conf
listen_port = 8080
log_level = info
timeout = 30
Mix literal text and a few values: expand some, protect the rest with a backslash
$ host=web01; cat <<EOF
> name: $host
> prompt: \$ (a literal dollar sign)
> today: $(date -d '2026-03-08' +%F)
> EOF
name: web01
prompt: $ (a literal dollar sign)
today: 2026-03-08
Indent a heredoc inside a function or an if: <<- strips ALL leading TAB characters from every line (tabs only, never spaces)
$ backup_note() {
> cat <<-'EOF'
> Backup finished.
> Still flush left: tabs of any depth are removed.
> Four spaces after the tabs are kept.
> EOF
> }
$ backup_note | cat -A
Backup finished.$
Still flush left: tabs of any depth are removed.$
Four spaces after the tabs are kept.$
In the example above the indentation before each line is made of TABS. cat -A shows tabs as ^I and line ends as $, which is the quickest way to see invisible characters (a tab that you thought was spaces is a classic reason for "my heredoc will not close").
Feed a heredoc to any command, not just cat
$ sort <<'EOF'
> pear
> apple
> fig
> EOF
apple
fig
pear
$ wc -l <<'EOF'
> one
> two
> three
> EOF
3
$ python3 - <<'PY'
> import json
> print(json.dumps({"service": "web", "ok": True}))
> PY
{"service": "web", "ok": true}
The "-" after python3 means "read the program from standard input". The same pattern works for bash -s, psql -f -, sqlite3, mysql, mail and many more.
Heredoc into a pipeline
$ cat <<'EOF' | tr 'a-z' 'A-Z' | sed 's/^/| /'
> quiet text
> becomes loud text
> EOF
| QUIET TEXT
| BECOMES LOUD TEXT
Heredoc into a variable (the command substitution keeps your text in one piece)
$ msg=$(cat <<'EOF'
> Line one
> Line two
> EOF
> )
$ echo "$msg" | wc -l
2
Run a block of commands in another shell: here a local bash, but the same works over ssh
$ bash -s <<'EOF'
> echo "running in a child shell"
> x=$((6 * 7))
> echo "x is $x"
> EOF
running in a child shell
x is 42
$ ssh web01 'bash -s' <<'EOF'
> hostname
> uptime -p
> EOF
Here strings (<<<): one string as standard input, no quoting acrobatics
$ wc -w <<< "one two three four"
4
$ read -r first second rest <<< "alpha beta gamma delta"; echo "$first | $second | $rest"
alpha | beta | gamma delta
$ IFS=, read -r a b c <<< "10,20,30"; echo "$((a + b + c))"
60
$ base64 <<< "hello"
aGVsbG8K
Note: a here string adds a newline at the end (that is why the base64 output above is "aGVsbG8K", not "aGVsbG8="). Use printf '%s' "hello" | base64 when the exact bytes matter.
Writing a file that needs root: the classic mistake and the fix
The redirection (> file) is done by YOUR shell, before sudo starts, so sudo cat > /etc/file can never work. Let a command that runs as root do the writing: tee.
$ cat > /etc/motd <<'EOF'
> Authorized use only.
> EOF
bash: /etc/motd: Permission denied
$ sudo tee /etc/motd >/dev/null <<'EOF'
> Authorized use only.
> EOF
tee writes its input to the file(s) you name AND to its own standard output. The >/dev/null hides that second copy. Add -a to append instead of overwrite (sudo tee -a /etc/file).
Same thing without a heredoc:
$ echo 'vm.swappiness=10' | sudo tee /etc/sysctl.d/99-swappiness.conf
$ sudo sh -c 'cat > /etc/motd' <<'EOF'
> also works: the shell started by sudo does the redirection
> EOF
Always look at the file afterwards (cat, or sudo -e / visudo for sensitive ones) and keep a backup of anything you overwrite: sudo cp -a /etc/file /etc/file.bak
tee in a sandbox (what it really does)
$ tee copy.txt <<'EOF'
> shown on the screen AND saved
> EOF
shown on the screen AND saved
$ cat copy.txt
shown on the screen AND saved
$ printf 'first\n' | tee log.txt >/dev/null; printf 'second\n' | tee -a log.txt >/dev/null; cat log.txt
first
second
Heredoc gotchas
1. The closing delimiter must be alone on its line, exactly as written: no trailing spaces, no leading spaces (unless you used <<- with tabs). If the shell says "here-document delimited by end-of-file", this is almost always why.
2. Unquoted delimiter = expansion. A password containing $ or ` in an unquoted heredoc is silently changed.
3. Windows line endings (\r\n) in a script make the delimiter "EOF\r", which never matches. Fix the file: tr -d '\r' < in.sh > out.sh (or dos2unix).
4. Pasting indented heredocs into a terminal: the Tab characters can trigger completion. Use spaces, or write the text into a file with an editor and run the file.
5. Choose a delimiter that cannot appear in the text, for example END_OF_CONFIG, when you are generating other scripts.
6. A heredoc inside $( ... ) is fine, but put the closing parenthesis on its own line after the delimiter (as in the variable example above).
tee, redirection and process substitution
Every command has three standard streams: 0 = input (stdin), 1 = normal output (stdout), 2 = error messages (stderr). Redirection decides where each one goes.
See output and save it at the same time: tee
$ printf 'build ok\nall tests passed\n' | tee result.txt
build ok
all tests passed
$ cat result.txt
build ok
all tests passed
Add -a to append, and name several files to write several copies
$ echo "run 2" | tee -a result.txt copy1.txt copy2.txt >/dev/null; cat result.txt copy2.txt
build ok
all tests passed
run 2
run 2
tee in the middle of a pipeline keeps a full copy while the rest of the pipeline filters
$ printf 'INFO start\nERROR disk full\nINFO done\n' | tee full.log | grep ERROR
ERROR disk full
$ cat full.log
INFO start
ERROR disk full
INFO done
Keep errors too: 2>&1 sends stderr to the same place as stdout, so tee sees both
$ f 2>&1 | tee both.log
to stdout
to stderr
$ cat both.log
to stdout
to stderr
Which stream goes where
$ f 2>/dev/null
to stdout
$ f >/dev/null
to stderr
$ f > out.txt 2> err.txt; echo "out.txt: $(cat out.txt)"; echo "err.txt: $(cat err.txt)"
out.txt: to stdout
err.txt: to stderr
$ f > all.txt 2>&1; cat all.txt
to stdout
to stderr
$ f &> all2.txt; cat all2.txt
to stdout
to stderr
The last two are the same (&> is the bash shortcut). Throw everything away: command > /dev/null 2>&1 (or &> /dev/null)
ORDER MATTERS: redirections are read left to right.
$ f 2>&1 >/dev/null | sed 's/^/pipe saw: /'
pipe saw: to stderr
$ f >/dev/null 2>&1 | sed 's/^/pipe saw: /'
In the first line stderr is pointed at the pipe first, THEN stdout goes to /dev/null: only the error reaches the pipe. In the second, stdout goes to /dev/null first and stderr then follows it: nothing is left.
Print your own error messages to stderr so they do not pollute data
$ echo "usage: backup.sh <dir>" >&2
usage: backup.sh <dir>
$ { echo "data line"; echo "warning line" >&2; } 2>/dev/null
data line
Overwrite, append, truncate, and protect yourself
$ echo one > t.txt; echo two >> t.txt; cat t.txt
one
two
$ : > t.txt; wc -c < t.txt
0
That ": > file" idiom empties a file without deleting it (handy for logs that a service keeps open).
$ set -o noclobber; echo new > t.txt; echo "exit status $?"
bash: t.txt: cannot overwrite existing file
exit status 1
$ echo forced >| t.txt; cat t.txt; set +o noclobber
forced
With noclobber on, > refuses to overwrite an existing file; >| overrides it deliberately. It stops the most common "I overwrote my notes" accident.
Send the output of a whole block or script somewhere
$ { echo "step 1"; echo "step 2"; } > steps.txt; cat steps.txt
step 1
step 2
$ ( echo "in a subshell"; cd /; pwd ) ; pwd
in a subshell
/
/home/demo
$ bash -c 'exec > script.log 2>&1; echo "this line goes to the log"; echo "and so does this error" >&2'; cat script.log
this line goes to the log
and so does this error
At the top of a script, exec > >(tee -a script.log) 2>&1 shows everything on screen AND logs it.
Extra file descriptors (3 and up): keep a log open without repeating it on every line
$ exec 3> debug.log; echo "first" >&3; echo "second" >&3; exec 3>&-; cat debug.log
first
second
Process substitution: use the output of a command where a file name is expected
$ diff <(sort a.txt) <(sort b.txt)
1d0
< apple
3a3
> date
# (exit status 1)
$ comm -12 <(sort a.txt) <(sort b.txt)
banana
cherry
$ comm -23 <(sort a.txt) <(sort b.txt)
apple
(diff exits with status 1 when the inputs differ, 0 when they are the same: handy in scripts.) comm needs sorted input: -12 shows lines in both files, -23 shows lines only in the first, -13 only in the second.
$ paste <(cut -d' ' -f1 <<< "a1 a2") <(cut -d' ' -f2 <<< "a1 a2")
a1 a2
Duplicate a stream into a side command and still pass it on: cmd | tee >(gzip > copy.gz) | next
$ printf 'one\ntwo\n' | tee >(wc -l > count.txt) | tr a-z A-Z; sleep 0.2; cat count.txt
ONE
TWO
2
The pipe-into-while trap (variables vanish)
Each command in a pipeline runs in its own subshell, so a variable set inside "| while read" is lost afterwards. Feed the loop from a process substitution instead.
$ n=0; printf 'a\nb\nc\n' | while read -r line; do n=$((n + 1)); done; echo "n is $n"
n is 0
$ n=0; while read -r line; do n=$((n + 1)); done < <(printf 'a\nb\nc\n'); echo "n is $n"
n is 3
$ n=0; while read -r line; do n=$((n + 1)); done <<< $'a\nb\nc'; echo "n is $n"
n is 3
Exit status in pipelines
$ false | true; echo "status: $? all statuses: ${PIPESTATUS[*]}"
status: 0 all statuses: 1 0
$ set -o pipefail; false | true; echo "status: $?"; set +o pipefail
status: 1
Without pipefail a pipeline's status is the LAST command's, so "cmd_that_failed | tee log" looks successful. Scripts should use set -o pipefail (see the script recipes).
Dates and timestamps (GNU date)
date prints the time, converts between formats and does calendar arithmetic. Everything here uses a FIXED date so the output never changes; on your machine use date alone (or date -d "now") for the current time.
Format codes: build any timestamp you need (this table was produced by running each code)
$ for f in %F %T %Y %m %d %H %M %S %y %e %j %u %a %A %b %B %p %I %U %V %Z %z %s %D %R; do printf '%-3s %s\n' "$f" "$(date -d "$d" "+$f")"; done
%F 2026-03-08
%T 14:05:09
%Y 2026
%m 03
%d 08
%H 14
%M 05
%S 09
%y 26
%e 8
%j 067
%u 7
%a Sun
%A Sunday
%b Mar
%B March
%p PM
%I 02
%U 10
%V 10
%Z UTC
%z +0000
%s 1772978709
%D 03/08/26
%R 14:05
Common combinations
$ date -d "$d" '+%Y-%m-%d'
2026-03-08
$ date -d "$d" '+%Y%m%d-%H%M%S'
20260308-140509
$ date -d "$d" '+%F %T'
2026-03-08 14:05:09
$ date -d "$d" '+%A, %B %e, %Y'
Sunday, March 8, 2026
$ date -d "$d" '+%d/%m/%Y %I:%M %p'
08/03/2026 02:05 PM
%F is %Y-%m-%d and %T is %H:%M:%S. The ISO order (year first, big units before small ones) sorts correctly as plain text: that is why logs and backup names should use it.
For "now" (output differs on every run, so shown here without a fixed result):
$ date +%F_%H%M%S
2026-10-02_093015 (illustration: yours is the current time)
$ date +%s
1790947815 (illustration for 2026-10-02 13:30:15 UTC: seconds since 1970-01-01 00:00:00 UTC, the "epoch")
ISO 8601 and UTC
$ date -d "$d" -Iseconds
2026-03-08T14:05:09+00:00
$ date -u -d "$d" +%Y-%m-%dT%H:%M:%SZ
2026-03-08T14:05:09Z
$ date -d "$d" -Idate
2026-03-08
Use -u (UTC) in logs, scripts and anything shared between servers. Convert to local time only when a human reads it.
Epoch seconds: convert both ways
$ date -d @1700000000
Tue Nov 14 22:13:20 UTC 2023
$ date -u -d @1700000000 '+%F %T'
2023-11-14 22:13:20
$ date -u -d '2023-11-14 22:13:20' +%s
1700000000
$ date -d '2026-03-08 14:05:09 UTC' +%s
1772978709
Milliseconds and nanoseconds (GNU date): date +%s%3N and date +%s%N. Epoch seconds from other systems that count milliseconds need dividing: date -d @$((1700000000123 / 1000)).
The same, with no extra process, in bash itself (bash 4.2 and newer)
$ printf '%(%F %T)T\n' 1700000000
2023-11-14 22:13:20
$ printf '%(%F %T)T\n' -1
2026-10-02 09:30:15 (-1 means "now")
Calendar arithmetic with plain words
Use a date WITHOUT a time of day as the starting point for sums: it is the safe form.
$ date -d "$day +1 day" +%F
2026-03-09
$ date -d "$day -90 days" +%F
2025-12-08
$ date -d "$day +2 weeks" +%F
2026-03-22
$ date -d "$day +1 month" +%F
2026-04-08
$ date -d "$day +1 year" +%F
2027-03-08
$ date -d "$day +36 hours" '+%F %T'
2026-03-09 12:00:00
$ date -d "2026-02-01 +1 month -1 day" +%F
2026-02-28
The one before last is the standard way to get the last day of a month: go to the first of the NEXT month, then back one day.
THE SIGN TRAP: once a time of day is in the string, GNU date reads "+5" or "-90" as a TIME ZONE offset, not as an amount. The result is wrong, and often only slightly wrong, which is the worst kind. Real output:
$ date -d "2026-03-08 14:05:09 +1 day" '+%F %T'
2026-03-09 13:05:09
$ date -d "2026-03-08 14:05:09 -90 days" '+%F %T'
date: invalid date ‘2026-03-08 14:05:09 -90 days’
# (exit status 1)
The first answer should have been 2026-03-09 14:05:09 (the hour moved), and the second is an error. Three safe ways to write it:
$ date -d "2026-03-08 14:05:09 5 days ago" '+%F %T'
2026-03-03 14:05:09
$ date -d "2026-03-08 14:05:09 UTC +5 days" '+%F %T'
2026-03-13 14:05:09
$ date -d "@$(( $(date -d '2026-03-08 14:05:09' +%s) + 5*86400 ))" '+%F %T'
2026-03-13 14:05:09
(No sign at all, a zone name before the sign, or plain epoch arithmetic. The last one is the most predictable, and it is what scripts that care about exact seconds should use.)
Month arithmetic gotcha: January 31 plus one month does not exist in February, so GNU date overflows into March
$ date -d "2026-01-31 +1 month" +%F
2026-03-03
Invalid dates are errors, which also makes date a quick calendar validator
$ date -d "2026-02-29" +%F
date: invalid date ‘2026-02-29’
# (exit status 1)
$ date -d "2024-02-29" +%F
2024-02-29
Words that work relative to NOW: now, today, yesterday, tomorrow, noon, midnight, "3 days ago", "next month", "last year", "friday", "next friday", "last monday", "1 hour ago", "tomorrow 03:30".
$ date -d "next friday" +%F
2026-10-09 (illustration, when run on a Friday 2026-10-02; the answer depends on today)
Weekday words only work relative to the current moment: combined with an explicit date they are ignored. To find "the next Friday after a given date", do the sum:
$ base=2026-03-08; dow=$(date -d "$base" +%u); delta=$(( (5 - dow + 7) % 7 )); [ "$delta" -eq 0 ] && delta=7; date -d "$base +$delta days" '+%F %A'
2026-03-13 Friday
(%u counts Monday as 1 ... Sunday as 7; change the 5 to 1 for the next Monday, and so on.)
Day of week, week number, day of year
$ date -d "2026-12-25" '+%A (day %u of the week), ISO week %V, day %j of the year'
Friday (day 5 of the week), ISO week 52, day 359 of the year
How long between two dates (or times)
$ start=$(date -d '2026-03-08' +%s); end=$(date -d '2026-12-25' +%s); echo "$(( (end - start) / 86400 )) days"
292 days
$ t1=$(date -d '2026-03-08 09:15:00' +%s); t2=$(date -d '2026-03-08 17:42:30' +%s); echo "$(( t2 - t1 )) seconds"
30450 seconds
Seconds to a readable duration: a small function
$ human() { local s=$1; printf '%dd %02dh %02dm %02ds\n' $((s/86400)) $((s%86400/3600)) $((s%3600/60)) $((s%60)); }; human 200000; human 59; human 8130
2d 07h 33m 20s
0d 00h 00m 59s
0d 02h 15m 30s
$ date -u -d @8130 +%T
02:15:30
The last line is a trick for durations shorter than one day: format the seconds as a time of day.
Running time of your own commands: the SECONDS variable counts since the shell started, and time measures a command
$ SECONDS=0; sleep 1; echo "elapsed: ${SECONDS}s"
elapsed: 1s
$ timeout 1 sleep 5; echo "timeout exit status: $? (124 means it was stopped for running too long)"
timeout exit status: 124 (124 means it was stopped for running too long)
Time zones: convert without leaving the terminal
$ for z in UTC America/New_York Europe/London Asia/Tokyo Australia/Sydney; do printf '%-18s' "$z"; TZ=$z date -d '2026-06-01 15:00 UTC' '+%F %H:%M %Z'; done
UTC 2026-06-01 15:00 UTC
America/New_York 2026-06-01 11:00 EDT
Europe/London 2026-06-01 16:00 BST
Asia/Tokyo 2026-06-02 00:00 JST
Australia/Sydney 2026-06-02 01:00 AEST
$ t='2026-06-01 15:00'; for z in America/New_York Europe/Berlin Asia/Kolkata; do printf '%-18s' "$z"; TZ=$z date -d "TZ=\"America/New_York\" $t" '+%F %H:%M %Z'; done
America/New_York 2026-06-01 15:00 EDT
Europe/Berlin 2026-06-01 21:00 CEST
Asia/Kolkata 2026-06-02 00:30 IST
The second example answers "what time is 3pm New York for my colleagues": the TZ="..." inside the date string says which zone the INPUT is in.
Daylight saving changes, shown by moving one second across the US spring-forward moment (2026-03-08 07:00 UTC)
$ for t in '06:59:59' '07:00:00'; do TZ=America/New_York date -d "2026-03-08 $t UTC" '+%F %T %Z'; done
2026-03-08 01:59:59 EST
2026-03-08 03:00:00 EDT
Set the zone for one command with TZ=..., for the shell with export TZ=..., and for the whole machine with timedatectl set-timezone (as root). List zone names with timedatectl list-timezones.
$ timedatectl
Local time: Fri 2026-10-02 09:30:15 EDT (illustration: the output shows the zone, whether the clock is synchronized, and NTP status)
Timestamps on files
$ ls -l --time-style=long-iso logs
total 0
-rw-rw-r-- 1 demo demo 0 2026-02-20 14:30 mid.log
-rw-rw-r-- 1 demo demo 0 2026-03-05 23:59 new.log
-rw-rw-r-- 1 demo demo 0 2026-01-10 08:00 old.log
$ stat -c '%y %n' logs/*
2026-02-20 14:30:00.000000000 +0000 logs/mid.log
2026-03-05 23:59:59.000000000 +0000 logs/new.log
2026-01-10 08:00:00.000000000 +0000 logs/old.log
$ stat -c '%Y %n' logs/new.log
1772755199 logs/new.log
$ date -r logs/mid.log '+%F %T'
2026-02-20 14:30:00
%y is the last modification time in human form, %Y the same in epoch seconds; %x/%w exist for access and creation time where the file system records them.
Find files in a date window
$ find logs -type f -newermt '2026-02-01' ! -newermt '2026-03-01'
logs/mid.log
$ find logs -type f -newer logs/mid.log
logs/new.log
$ find logs -type f -mtime +30
(The last form depends on today's date, so no result is shown.) -mtime +30 means "modified MORE than 30 days ago", -mtime -7 "within the last 7 days", -mmin -30 "within the last 30 minutes". Use -newermt with a real date when you mean a calendar window.
Set or copy times
$ touch -d '2026-03-08 14:05:09' fixed.txt; touch -r fixed.txt copy-of-time.txt; stat -c '%y' copy-of-time.txt
2026-03-08 14:05:09.000000000 +0000
$ touch -t 202601151030.00 stamp.txt; stat -c '%y' stamp.txt
2026-01-15 10:30:00.000000000 +0000
Rename files with their modification date
$ for f in logs/*.log; do d=$(date -r "$f" +%Y%m%d); echo "mv $f logs/${d}_$(basename "$f")"; done
mv logs/mid.log logs/20260220_mid.log
mv logs/new.log logs/20260305_new.log
mv logs/old.log logs/20260110_old.log
(The loop only prints the mv commands. When they look right, remove the echo and the quotes around the command.)
Dates inside text and logs
Convert an epoch column to readable dates (the portable way: call date for each line)
$ while read -r ts what who; do printf '%s %s %s\n' "$(date -u -d @"$ts" '+%F %T')" "$what" "$who"; done < events.txt
2023-11-14 22:13:20 login alice
2023-11-14 23:13:20 logout alice
2023-11-15 23:13:20 login bob
awk can do the same without starting a process per line, if you use gawk (strftime). The default awk on Ubuntu is mawk: check what your system has.
$ awk --version 2>&1 | head -1
mawk 1.3.4 20240123
Sort log lines by an ISO timestamp: plain sort works; sort by an epoch column: sort -n
$ printf '2026-03-08T14:05:09 b\n2025-12-31T23:59:59 a\n2026-03-08T09:00:00 c\n' | sort
2025-12-31T23:59:59 a
2026-03-08T09:00:00 c
2026-03-08T14:05:09 b
Get yesterday's log file name in a script
$ f=/var/log/app/app-$(date -d yesterday +%F).log
Systemd logs speak dates too:
$ journalctl --since "2026-03-08 09:00" --until "2026-03-08 10:00"
$ journalctl --since "1 hour ago" -u nginx
Wait, repeat and schedule
$ watch -n 5 'date +%T; uptime'
$ sleep 0.5 # fractions are fine in GNU sleep; sleep 2m, sleep 1h also work
$ date -d 'tomorrow 03:30' +%s # an epoch to feed to at, systemd-run or a script that sleeps until then
Time-related gotchas
1. Never put local time without a zone into logs or databases: during daylight-saving changes there is an hour that happens twice and an hour that does not exist. Use UTC (date -u, -Iseconds, or a trailing Z).
2. date -d "yyyy-mm-dd" without a time means 00:00 in the CURRENT zone: set TZ=UTC date -d ... when you mean UTC.
3. Month and year arithmetic overflows (January 31 + 1 month). Compute from the first of the month, or do arithmetic in epoch seconds.
4. %U (week starting Sunday), %W (week starting Monday) and %V (ISO week) differ around New Year. For ISO weeks always pair %V with %G (ISO year), not %Y.
5. macOS/BSD date uses different options: date -r SECONDS for epoch, date -v+1d for arithmetic, date -j -f to parse. Install GNU coreutils (gdate) to use the commands in this section unchanged.
6. Two machines "five minutes apart" cause surprising failures (TLS certificates, Kerberos, TOTP, make): check with timedatectl and keep them on NTP (chrony or systemd-timesyncd).
Files, folders and disk space
What is big? (the question behind most "disk full" tickets)
$ du -sh project/* | sort -h
0 project/run
8.0K project/docs
28K project/src
3.7M project/logs
$ du -ah project | sort -rh | head -5
3.7M project/logs
3.7M project
2.9M project/logs/big.log
784K project/logs/mid.log
28K project/src
$ ls -lhS project/logs
total 3.7M
-rw-r----- 1 demo demo 2.9M Jan 15 2026 big.log
-rw-rw-r-- 1 demo demo 782K Jan 15 2026 mid.log
$ find project -type f -size +500k -exec ls -lh {} +
-rw-r----- 1 demo demo 2.9M Jan 15 2026 project/logs/big.log
-rw-rw-r-- 1 demo demo 782K Jan 15 2026 project/logs/mid.log
sort -h understands 4.0K, 2.9M, 1G. For a whole file system: df -h (space) and df -i (inodes: a disk can be "full" with plenty of space if it has run out of inodes). Interactive explorers: ncdu (if installed).
$ df -hT -x tmpfs -x devtmpfs
$ sudo du -xh / 2>/dev/null | sort -rh | head -20 # -x stays on one file system
Look at a file without opening an editor
$ head -n 3 project/docs/README.md; tail -n 1 project/src/main.py
# Notes
print("hi")
$ wc project/src/main.py
1 1 12 project/src/main.py
$ file project/src/main.py project/src/app.bin project/run
project/src/main.py: ASCII text
project/src/app.bin: data
project/run: symbolic link to src/main.py
$ cat -A project/src/main.py
print("hi")$
$ nl -ba project/src/main.py
1 print("hi")
Useful variants: head -c 100 (first 100 bytes), tail -n +2 (everything from line 2), head -n -1 (everything except the last line), tac (reverse the lines), rev (reverse each line), less +F file (follow like tail -f, Ctrl+C to stop following, q to quit).
Permissions, owner and size as numbers (stat prints exactly what you ask for)
$ stat -c '%a %A %U:%G %s bytes %n' project/logs/big.log project/src/main.py
640 -rw-r----- demo:demo 3000000 bytes project/logs/big.log
755 -rwxr-xr-x demo:demo 12 bytes project/src/main.py
$ stat -c '%F' project/src project/run
directory
symbolic link
$ stat -c '%N' project/run
'project/run' -> 'src/main.py'
Format letters worth knowing: %a octal mode, %A mode as text, %U owner, %G group, %s size in bytes, %F file type, %y modified, %i inode, %h hard links.
Make a file executable, or private
$ chmod u+x,go-rwx project/logs/mid.log; stat -c '%a %n' project/logs/mid.log
700 project/logs/mid.log
$ chmod 600 project/logs/mid.log; stat -c '%a %n' project/logs/mid.log
600 project/logs/mid.log
Paths: where is it really, and what is its name?
$ readlink -f project/run
/home/demo/project/src/main.py
$ realpath --relative-to=project/docs project/src/main.py
../src/main.py
$ basename /var/log/nginx/access.log; dirname /var/log/nginx/access.log; basename /var/log/nginx/access.log .log
access.log
/var/log/nginx
access
$ f=project/logs/big.log; echo "${f##*/} ${f%/*} ${f%.*} ${f##*.}"
big.log project/logs project/logs/big log
The last line uses bash parameter expansion (no extra process): ${f##*/} name, ${f%/*} folder, ${f%.*} without extension, ${f##*.} extension.
$ cd project/src && pwd -P && cd - >/dev/null; pwd
/home/demo/project/src
/home/demo
In a script, the folder the script itself lives in is: script_dir=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)
Create, copy, move and link without regrets
$ mkdir -p demo/{src,docs,tests}/archive; ls -R demo | head -12
demo:
docs
src
tests
demo/docs:
archive
demo/docs/archive:
demo/src:
archive
$ cp project/docs/README.md project/docs/README.md.bak; ls project/docs
README.md
README.md.bak
$ cp project/docs/README.md{,.orig}; ls project/docs
README.md
README.md.bak
README.md.orig
$ cp -a project project-copy && echo "copied with permissions and times kept"
copied with permissions and times kept
$ mv -n project/docs/README.md.orig project/docs/README.md.bak; ls project/docs
mv: not replacing 'project/docs/README.md.bak'
README.md
README.md.bak
README.md.orig
$ install -d -m 750 secure/inbox; stat -c '%a %n' secure secure/inbox
755 secure
750 secure/inbox
install -d creates the folder (and parents) in one step and sets the mode on the LAST one only: the parent kept the default. Use it in scripts instead of mkdir + chmod (install -D -m 600 src dest also copies a file and creates the folders).
$ ln -s project/docs/README.md latest.md; readlink latest.md
project/docs/README.md
$ mktemp -d | sed 's/[^/]*$/XXXXXXXXXX/'
/tmp/XXXXXXXXXX
cp -a keeps permissions, owner and times (and copies folders and links as they are). mv -n never overwrites; cp --update=none (older cp, macOS: cp -n) likewise. cp --backup=numbered makes numbered backups of anything it would overwrite. A brace like file{,.bak} expands to "file file.bak": the shortest way to back up before editing.
mktemp makes a unique, private temporary file (or a folder with -d) and prints its name. The sed above only hides the random part for this page.
$ tmp=$(mktemp); echo "scratch" > "$tmp"; cat "$tmp"; rm -f "$tmp"
scratch
Compare files and folders
$ diff one.txt two.txt
2c2
< beta
---
> BETA
3a4
> delta
# (exit status 1)
$ diff -u one.txt two.txt
--- one.txt 2026-01-15 10:30:00.000000000 +0000
+++ two.txt 2026-01-15 10:30:00.000000000 +0000
@@ -1,3 +1,4 @@
alpha
-beta
+BETA
gamma
+delta
# (exit status 1)
$ diff -y --suppress-common-lines one.txt two.txt
beta | BETA
> delta
# (exit status 1)
$ cmp one.txt two.txt
one.txt two.txt differ: byte 7, line 2
# (exit status 1)
$ diff -rq project project-copy && echo "folders are identical"
folders are identical
$ sha256sum one.txt two.txt
4fdbc441ea7b546100e086ac1e4fc5ae6749b7314311c99db05be450eca12996 one.txt
2d1a8745bdad293ad22e1bd43a730ea6a6e79dfb25ee4de382dee96633029ff6 two.txt
diff -u is the format used for patches (diff -u old new > fix.patch; patch -p0 < fix.patch). cmp compares byte by byte and is the right tool for binary files. A checksum comparison (sha256sum) is the way to compare big files on two different machines.
$ diff <(ls project) <(ls project-copy) && echo "same listing"
same listing
Cut big files into pieces and put them back
$ seq 1 10 > numbers.txt; split -l 4 -d numbers.txt part_; ls part_*; cat part_* | wc -l
part_00
part_01
part_02
10
$ truncate -s 1M sparse.img; stat -c '%n is %s bytes long' sparse.img; du -h sparse.img
sparse.img is 1048576 bytes long
0 sparse.img
truncate makes a sparse file of the stated size instantly (the size is 1 MiB, du shows what is really stored: 0). Handy for test disks, swap and quota tests.
Delete carefully
$ echo project/logs/*.log
project/logs/big.log project/logs/mid.log
$ rm -v project/logs/mid.log
removed 'project/logs/mid.log'
$ find project -type f -name '*.bin' -print
project/src/app.bin
$ find project -type f -name '*.bin' -delete
Always preview a glob or a find with echo or -print before you attach rm or -delete to it. And: rm -rf "$dir/" with an empty $dir becomes rm -rf /, so guard variables with ${dir:?} (see the safety habits section).
xargs, find and loops: run a command on many things
xargs: turn lines into arguments
$ echo "a b c" | xargs -n1
a
b
c
$ printf '%s\n' 1 2 3 4 5 | xargs -n2
1 2
3 4
5
$ cat list.txt | xargs -I{} echo "processing {}"
processing one
processing two
processing three
$ printf 'x\ny\n' | xargs -I{} sh -c 'echo "[{}] $(echo {} | tr a-z A-Z)"'
[x] X
[y] Y
$ seq 1 6 | xargs -P3 -I{} sh -c 'sleep 0.1; echo "job {}"' | sort
job 1
job 2
job 3
job 4
job 5
job 6
-n limits arguments per call, -I{} puts each input in a chosen spot, -P runs that many at once (parallel, here three at a time: the output order is not guaranteed, so sort it or write results to separate files). GNU xargs -r skips running the command when there is no input at all.
$ printf '' | xargs -r echo "never printed"; echo "done (no output above)"
done (no output above)
Names with spaces: the null-byte way (always use it with find)
$ find pics -name '*.jpeg' | xargs ls 2>&1 | head -3
ls: cannot access 'pics/with': No such file or directory
ls: cannot access 'space.jpeg': No such file or directory
pics/a.jpeg
$ find pics -name '*.jpeg' -print0 | xargs -0 ls
pics/a.jpeg
pics/b.jpeg
pics/with space.jpeg
$ find pics -name '*.jpeg' -exec ls {} +
pics/a.jpeg
pics/b.jpeg
pics/with space.jpeg
The first command breaks on "with space.jpeg" (xargs splits at spaces); -print0 with xargs -0 uses a null byte that cannot appear in a file name. find -exec ... {} + does the same job without xargs.
Run one command per file, or many files per command
$ find pics -name '*.jpeg' -exec echo FILE: {} \; | sort
FILE: pics/a.jpeg
FILE: pics/b.jpeg
FILE: pics/with space.jpeg
$ find pics -name '*.jpeg' -exec sh -c 'echo "one call, $# files"' _ {} +
one call, 3 files
find lists files in directory order, not alphabetical order (pipe through sort when the order matters). With \; the command runs once per file; with + it receives as many files as fit in one call.
Loops over files (quote the variable, always)
$ for f in pics/*.jpeg; do echo "found: $f"; done
found: pics/a.jpeg
found: pics/b.jpeg
found: pics/with space.jpeg
$ for f in pics/*.jpeg; do mv -- "$f" "${f%.jpeg}.jpg"; done; ls pics
a.jpg
b.jpg
c.png
with space.jpg
$ for f in pics/*.jpg; do printf '%s -> %s\n' "$f" "${f%.jpg}.webp"; done
pics/a.jpg -> pics/a.webp
pics/b.jpg -> pics/b.webp
pics/with space.jpg -> pics/with space.webp
$ shopt -s nullglob; for f in pics/*.gif; do echo "gif: $f"; done; echo "no gifs, loop ran zero times"; shopt -u nullglob
no gifs, loop ran zero times
Without nullglob, a pattern that matches nothing stays as the literal text pics/*.gif and the loop runs once with that bogus name. Turn nullglob on in scripts.
Read a file line by line (the safe way)
$ while IFS= read -r line; do echo "line: [$line]"; done < list.txt
line: [one]
line: [two]
line: [three]
$ printf 'a:1\nb:2\n' | while IFS=: read -r key value; do echo "$key has $value"; done
a has 1
b has 2
$ mapfile -t names < list.txt; echo "${#names[@]} lines, second is ${names[1]}"
3 lines, second is two
IFS= keeps leading spaces and -r keeps backslashes: together they make read return the line exactly as written. mapfile -t (also called readarray) loads a whole file into an array.
Parse a CSV (simple cases)
$ tail -n +2 scores.csv | while IFS=, read -r name team score; do echo "$name ($team): $score"; done
ada (red): 90
bob (blue): 75
cy (red): 82
$ awk -F, 'NR > 1 {total[$2] += $3} END {for (t in total) print t, total[t]}' scores.csv | sort
blue 75
red 172
$ cut -d, -f1,3 scores.csv | column -s, -t
name score
ada 90
bob 75
cy 82
Quoted fields with commas inside need a real CSV tool (python3 -m csv, csvkit, or Miller); cut and awk split on every comma.
Counting loops and ranges
$ for i in {1..5}; do printf '%s ' "$i"; done; echo
1 2 3 4 5
$ for i in {01..05}; do printf '%s ' "$i"; done; echo
01 02 03 04 05
$ for i in $(seq 10 10 50); do printf '%s ' "$i"; done; echo
10 20 30 40 50
$ for ((i = 3; i > 0; i--)); do echo "$i..."; done; echo "liftoff"
3...
2...
1...
liftoff
$ echo {a..e}; echo {2024..2026}-{01,02}
a b c d e
2024-01 2024-02 2025-01 2025-02 2026-01 2026-02
Retry until it works, with a limit
$ n=0; until [ "$n" -ge 3 ]; do n=$((n + 1)); echo "attempt $n"; done
attempt 1
attempt 2
attempt 3
$ retry() { local tries=$1; shift; local i; for ((i = 1; i <= tries; i++)); do "$@" && return 0; echo "attempt $i failed" >&2; sleep 0.1; done; return 1; }; retry 3 false; echo "gave up with status $?"
attempt 1 failed
attempt 2 failed
attempt 3 failed
gave up with status 1
$ cnt=0; flaky() { cnt=$((cnt + 1)); [ "$cnt" -ge 3 ]; }; retry 5 flaky && echo "succeeded on attempt $cnt"
attempt 1 failed
attempt 2 failed
succeeded on attempt 3
Wait for a port or a file
$ wait_for_port() { local host=$1 port=$2 tries=$3 i; for ((i = 0; i < tries; i++)); do (exec 3<>"/dev/tcp/$host/$port") 2>/dev/null && return 0; sleep 0.2; done; return 1; }; wait_for_port 127.0.0.1 9 3 || echo "port 9 on localhost is closed after 3 tries"
port 9 on localhost is closed after 3 tries
The (exec 3<>/dev/tcp/host/port) trick is a built-in bash way to test a TCP port without nc. For a file: until [ -f /tmp/ready ]; do sleep 1; done
$ timeout 2 bash -c 'until [ -f never ]; do sleep 0.2; done'; echo "timeout stopped the wait: status $?"
timeout stopped the wait: status 124
Do something every N seconds, and stop after a while
$ for i in 1 2 3; do echo "tick $i"; sleep 0.2; done
tick 1
tick 2
tick 3
$ while true; do date +%T; sleep 5; done # Ctrl+C to stop
$ watch -n 2 -d 'df -h /' # -d highlights what changed
Menus, defaults and confirmations
$ read -r -p "Continue? [y/N] " answer <<< "y"; case ${answer,,} in y|yes) echo "going ahead";; *) echo "cancelled";; esac
going ahead
$ name=${1:-world}; echo "hello, $name"
hello, world
In a real script read -p reads from the keyboard; the here string above only feeds it an answer so the example can run unattended. ${answer,,} lowercases the value (bash 4+).
Script recipes: small building blocks for scripts you can trust
A safe header
$ bash header.sh
script started
result: ok
$ bash -n header.sh && echo "syntax OK"
syntax OK
set -e stops at the first failing command, set -u treats an unset variable as an error (it catches typos like $DIRR), set -o pipefail makes a pipeline fail if ANY part fails. bash -n only checks the syntax without running anything, and bash -x file prints every command as it runs.
What -u and -e do
$ bash -c 'set -u; echo "dir is $DIRR"'; echo "status $?"
bash: line 1: DIRR: unbound variable
status 127
$ bash -c 'set -e; echo before; false; echo "never printed"'; echo "status $?"
before
status 1
$ bash -c 'echo "${DIRR:-/default/path}"; echo "${DIRR:?DIRR must be set}"'; echo "status $?"
/default/path
bash: line 1: DIRR: DIRR must be set
status 127
${var:-default} uses a default, ${var:?message} stops with a message when the variable is empty or unset, ${var:+text} gives text only when it IS set.
Clean up after yourself: mktemp plus trap
$ cat > cleanup.sh <<'EOF'
> #!/usr/bin/env bash
> set -euo pipefail
> workdir=$(mktemp -d)
> trap 'rm -rf "$workdir"; echo "cleaned up"' EXIT
> echo "working in a temporary folder: $(test -d "$workdir" && echo exists)"
> echo data > "$workdir/scratch.txt"
> exit 3
> EOF
$ bash cleanup.sh
working in a temporary folder: exists
cleaned up
# (exit status 3)
A trap on EXIT runs whatever way the script ends: normally, with exit, on an error under set -e, or on Ctrl+C. Other useful traps: ERR (log the failing line) and INT/TERM (stop child processes).
Report where it failed
$ cat > err.sh <<'EOF'
> #!/usr/bin/env bash
> set -Eeuo pipefail
> trap 'echo "failed at line $LINENO: $BASH_COMMAND" >&2' ERR
> echo "step 1"
> ls /does/not/exist
> echo "step 2"
> EOF
$ bash err.sh
step 1
ls: cannot access '/does/not/exist': No such file or directory
failed at line 5: ls /does/not/exist
# (exit status 2)
Only one copy at a time: flock
$ cat > once.sh <<'EOF'
> #!/usr/bin/env bash
> exec 9> /tmp/once-demo.lock
> flock -n 9 || { echo "already running: exiting"; exit 1; }
> echo "got the lock, working..."
> sleep "${1:-0}"
> EOF
$ bash once.sh 1 & sleep 0.3; bash once.sh 0; wait
got the lock, working...
already running: exiting
$ flock -n /var/lock/backup.lock /usr/local/bin/backup.sh # same idea in a cron line: skip the run if the last one is still going
The lock is released automatically when the process ends, even if it crashes: that is the advantage over "touch a lock file".
Arguments and usage text
$ cat > args.sh <<'EOF'
> #!/usr/bin/env bash
> set -euo pipefail
> usage() { cat <<'USAGE'
> Usage: args.sh [-v] [-o FILE] INPUT
> -v verbose
> -o FILE write the result to FILE
> USAGE
> }
> verbose=0; out=/dev/stdout
> while getopts ':vo:h' opt; do
> case $opt in
> v) verbose=1 ;;
> o) out=$OPTARG ;;
> h) usage; exit 0 ;;
> \?) echo "unknown option: -$OPTARG" >&2; usage >&2; exit 2 ;;
> :) echo "option -$OPTARG needs a value" >&2; exit 2 ;;
> esac
> done
> shift $((OPTIND - 1))
> input=${1:?missing INPUT (try -h)}
> [ "$verbose" -eq 1 ] && echo "reading $input" >&2
> echo "processed $input" > "$out"
> EOF
$ bash args.sh -h
Usage: args.sh [-v] [-o FILE] INPUT
-v verbose
-o FILE write the result to FILE
$ bash args.sh
args.sh: line 20: 1: missing INPUT (try -h)
# (exit status 1)
$ bash args.sh -v -o result.txt data.csv; cat result.txt
reading data.csv
processed data.csv
$ bash args.sh -x data.csv
unknown option: -x
Usage: args.sh [-v] [-o FILE] INPUT
-v verbose
-o FILE write the result to FILE
# (exit status 2)
Exit codes: 0 means success, 1-125 failure (2 is conventionally "usage error"), 126 not executable, 127 command not found, 130 stopped with Ctrl+C. Use them on purpose: other tools and cron look at them.
Logging with timestamps
$ log() { printf '%s [%s] %s\n' "$(date -u '+%F %T')" "$1" "${*:2}"; }; log INFO "backup started" | sed 's/^[^ ]* [^ ]* /<time> /'
<time> [INFO] backup started
$ log() { printf '%(%F %T)T [%s] %s\n' -1 "$1" "${*:2}"; }; log WARN "disk at 91%" | sed 's/^[^ ]* [^ ]* /<time> /'
<time> [WARN] disk at 91%
The sed only hides the current time so this page stays stable. In a script, send log lines to stderr (>&2) when stdout carries data, and consider logger -t myscript "message" to write into the system journal.
Check that tools exist, and that you are who you expect
$ for cmd in curl jq nosuchtool; do command -v "$cmd" >/dev/null && echo "have $cmd" || echo "MISSING $cmd"; done
have curl
have jq
MISSING nosuchtool
$ [[ $EUID -eq 0 ]] && echo "root" || echo "not root (EUID=$EUID)"
not root (EUID=1000)
$ script_dir=$(cd -- "$(dirname -- "${BASH_SOURCE[0]:-$0}")" && pwd -P); echo "this shell runs in: ${script_dir##*/}"
this shell runs in: hb
command -v is the portable way to ask "is this installed" (better than which). Inside a script, root checks, required tools, and required arguments belong at the top, before anything is changed.
A dry-run switch for dangerous scripts
$ DRY_RUN=1; run() { if [ "${DRY_RUN:-0}" = 1 ]; then echo "[dry-run] $*"; else "$@"; fi; }; run rm -rf /tmp/old-cache; run echo "this would run for real"
[dry-run] rm -rf /tmp/old-cache
[dry-run] echo this would run for real
$ DRY_RUN=0; run echo "now it runs for real"
now it runs for real
Source a config file safely, and pass values to subshells
$ printf 'name=web01\nport=8080\n' > settings.conf; set -a; . ./settings.conf; set +a; echo "$name listens on $port"; bash -c 'echo "child sees $name:$port"'
web01 listens on 8080
child sees web01:8080
set -a exports every variable assigned while it is on, so child processes see them. Only source files you trust: sourcing runs the file as code.
Debug a script
$ cat > dbg.sh <<'EOF'
> #!/usr/bin/env bash
> x=2
> y=$((x * 21))
> echo "answer: $y"
> EOF
$ bash -x dbg.sh
+ x=2
+ y=42
+ echo 'answer: 42'
answer: 42
$ PS4='+ line ${LINENO}: ' bash -x dbg.sh
+ line 2: x=2
+ line 3: y=42
+ line 4: echo 'answer: 42'
answer: 42
Check scripts with shellcheck (apt install shellcheck): it finds missing quotes, unused variables and portability problems before they bite.
$ shellcheck myscript.sh
System inspection: what is running, what is using what
Mostly read-only commands you can run on any server to answer "what is going on?". They depend on the machine, so they are shown without output here.
Load, memory, disk, uptime
$ uptime # load averages for 1, 5 and 15 minutes: compare with the number of CPU cores (nproc)
$ free -h # memory: look at "available", not "free"
$ df -hT -x tmpfs -x devtmpfs # space per file system; df -i for inodes
$ lsblk -f # disks, partitions, file systems and mount points
$ nproc; lscpu | head -15 # CPU count and model
$ vmstat 1 5 # five samples: watch si/so (swapping) and wa (waiting for disk)
$ iostat -xz 1 3 # per-disk utilisation (package sysstat)
Processes: who is eating the machine?
$ ps aux --sort=-%mem | head -6 # top memory users
$ ps aux --sort=-%cpu | head -6 # top CPU users
$ ps -eo pid,ppid,user,%cpu,%mem,etime,cmd --sort=-%cpu | head
$ ps -ef --forest | grep -A3 nginx # the process tree around a service
$ pgrep -a nginx # find by name, show the command line; pgrep -u www-data -c nginx counts
$ top -o %MEM # or: htop. In top press 1 for per-core, c for full command lines, q to quit
$ ps -o pid,etime,cmd -p 1234 # how long has process 1234 been running?
$ kill 1234 # polite request (SIGTERM). Wait a few seconds.
$ kill -9 1234 # last resort (SIGKILL): no cleanup, can leave lock files and half-written data
$ pkill -x myprog # by exact name; beware pkill -f, which matches the whole command line (including your own shell command!)
Ports and connections
$ ss -tulpn # listening TCP/UDP sockets and which process owns them (needs sudo to see every process)
$ ss -tn state established | head # current TCP connections
$ sudo ss -tnp | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head # who is connected the most
$ sudo lsof -i :8080 # which process has port 8080
$ sudo lsof +L1 # deleted files still held open (the classic "df says full, du says empty")
$ sudo lsof -p 1234 | head # files and sockets opened by one process
$ ip -br addr; ip route # addresses (brief) and routing table
$ ip -s link show eth0 # packet and error counters
Which package owns this file, and what is in a package?
$ dpkg -S /usr/bin/ssh # Debian, Ubuntu: which package installed it
$ dpkg -L openssh-client | head # files in a package
$ apt-file search bin/nmap # which package contains a file (apt install apt-file; apt-file update)
$ apt-cache policy nginx # installed vs available version and where it comes from
$ rpm -qf /usr/bin/ssh # RHEL family: which package owns it
$ rpm -ql openssh-clients | head # files in a package
$ dnf provides '*/nmap' # which package provides a file
$ command -v ssh; type -a ls # where a command comes from; type -a shows aliases and every match in PATH
Services (systemd)
$ systemctl status nginx --no-pager
$ systemctl is-active nginx; systemctl is-enabled nginx
$ systemctl --failed --no-pager # first command to run on a sick server
$ systemctl list-units --type=service --state=running --no-pager
$ systemctl list-timers --no-pager # scheduled jobs, and when they last/next ran
$ systemctl cat nginx # the unit file, including drop-ins
$ sudo systemctl edit nginx # add overrides safely (creates a drop-in); then sudo systemctl restart nginx
$ systemd-analyze blame | head # what slows the boot down
$ sudo systemctl daemon-reload # after editing unit files
Users, logins and permissions
$ id; id alice # uid, gid and groups of a user
$ getent passwd alice # account information even for LDAP/AD users (grep /etc/passwd misses those)
$ who -a; w; last -n 10 # who is logged in now, and recent logins
$ sudo -l # what may I run with sudo?
$ getfacl file; ls -ld file # ACLs and plain permissions
$ namei -l /var/www/html/index.html # permissions of EVERY folder on the way: finds "Permission denied" mysteries
Kernel and hardware
$ uname -r; uname -a; cat /etc/os-release | head -3
$ uptime -p; uptime -s # time since boot / boot time
$ dmesg -T | tail -20 # kernel messages with readable timestamps (sudo may be needed); dmesg -T --level=err,warn
$ lspci | head; lsusb; sudo dmidecode -t system | head
$ ls /var/run/reboot-required 2>/dev/null && echo "reboot needed" # Ubuntu: a package update is waiting for a restart
Logs: finding the needle
$ grep -c ERROR app.log
2
$ grep -n -E 'ERROR|WARN' app.log
2:2026-03-08 09:00:07 WARN slow query (1.8s)
3:2026-03-08 09:05:12 ERROR db connection lost
6:2026-03-08 09:40:00 ERROR timeout calling api
$ grep -B1 -A2 'db connection lost' app.log
2026-03-08 09:00:07 WARN slow query (1.8s)
2026-03-08 09:05:12 ERROR db connection lost
2026-03-08 09:05:13 INFO retrying
2026-03-08 09:05:15 INFO db connection restored
$ grep -v INFO app.log
2026-03-08 09:00:07 WARN slow query (1.8s)
2026-03-08 09:05:12 ERROR db connection lost
2026-03-08 09:40:00 ERROR timeout calling api
$ awk '$3 == "ERROR"' app.log | wc -l
2
$ grep '^2026-03-08 09:0[0-5]' app.log | wc -l
5
$ awk '$2 >= "09:05:00" && $2 <= "09:10:00"' app.log
2026-03-08 09:05:12 ERROR db connection lost
2026-03-08 09:05:13 INFO retrying
2026-03-08 09:05:15 INFO db connection restored
$ cut -d' ' -f3 app.log | sort | uniq -c | sort -rn
3 INFO
2 ERROR
1 WARN
Time windows in logs are easy when the timestamps are ISO: they compare correctly as plain text.
Count errors per hour
$ grep ERROR app.log | cut -c1-13 | sort | uniq -c
2 2026-03-08 09
Follow a log while it grows, and filter it live
$ tail -f /var/log/syslog
$ tail -F /var/log/nginx/error.log # -F keeps following after the file is rotated
$ tail -f /var/log/nginx/access.log | grep --line-buffered ' 500 ' # --line-buffered shows matches immediately
$ less +F /var/log/syslog # Ctrl+C to stop following and search with /, F to follow again
Rotated and compressed logs
$ zcat /var/log/nginx/access.log.2.gz | grep ' 404 ' | wc -l
$ zgrep -h ' 500 ' /var/log/nginx/access.log* # works on plain and .gz files together
$ ls -lt /var/log/nginx | head -5
The journal (systemd)
$ journalctl -u nginx --since "1 hour ago"
$ journalctl -u nginx --since "2026-03-08 09:00" --until "2026-03-08 10:00" --no-pager
$ journalctl -p err -b --no-pager # errors and worse since the last boot
$ journalctl -b -1 -p err --no-pager | tail # errors from the PREVIOUS boot: what happened before the reboot?
$ journalctl -f -u myapp # follow
$ journalctl -u myapp -o json-pretty -n 1 # structured fields: handy for filtering with jq
$ journalctl --disk-usage; sudo journalctl --vacuum-time=14d # how big is it, and trim it
$ journalctl _COMM=sshd --since today | grep 'Failed password'
$ logger -t deploy "release 42 finished" # write your own line into the journal; read it with journalctl -t deploy
Logins and attacks
$ last -n 15; lastb -n 10 2>/dev/null # recent logins, failed logins (root)
$ sudo journalctl -u ssh --since yesterday | grep -E 'Accepted|Failed' | awk '{print $1, $2, $3, $6, $9, $11}' | tail
$ sudo grep 'Failed password' /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head # top source IPs of failed logins
Web server logs: the one-liners
$ awk '{print $9}' web.log | sort | uniq -c | sort -rn
2 404
1 500
1 200
$ awk '$9 == 404 {print $7}' web.log | sort | uniq -c | sort -rn
2 /missing
$ awk '{print $1}' web.log | sort | uniq -c | sort -rn | head -3
3 203.0.113.9
1 198.51.100.4
$ awk '{bytes += $10} END {printf "%.1f KB served\n", bytes/1024}' web.log
1.0 KB served
$ awk '{print $6}' web.log | tr -d '"' | sort | uniq -c | sort -rn
3 GET
1 POST
$ awk -F'"' '{print $6}' web.log | sort | uniq -c | sort -rn
3 curl/8.5.0
1 Mozilla/5.0
These assume the common "combined" format (field 9 is the status, 7 the path, 10 the size). If your log format is different, count the fields with awk '{print NF; exit}' and print the line with fields numbered: awk '{for (i=1;i<=NF;i++) print i, $i; exit}' web.log
Network checks, SSH and copying files
Is it up, is it listening, is the name right?
$ ping -c 3 example.com # reachable? (some networks block ping: that proves nothing by itself)
$ dig +short example.com A # what does DNS say; dig +short example.com MX; dig +short -x 93.184.216.34 (reverse)
$ dig @1.1.1.1 +short example.com # ask a specific resolver: compare with your own to find DNS problems
$ getent hosts example.com # what THIS machine resolves (uses /etc/hosts, DNS, sssd...)
$ curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' https://example.com/
$ curl -sSI https://example.com/ | head -5 # headers only
$ curl -sS --connect-timeout 3 --max-time 10 -o /dev/null -w 'dns %{time_namelookup}s connect %{time_connect}s tls %{time_appconnect}s first byte %{time_starttransfer}s total %{time_total}s\n' https://example.com/
$ nc -zv example.com 443 # can I open a TCP connection to that port?
$ timeout 3 bash -c 'exec 3<>/dev/tcp/example.com/443' && echo open || echo closed # the same, with only bash
$ mtr -rwc 10 example.com # ping and traceroute combined (package mtr)
$ openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null | openssl x509 -noout -dates -subject -issuer
Check a port on this machine with plain bash (a closed port, so you can see the failure)
$ timeout 3 bash -c 'exec 3<>/dev/tcp/127.0.0.1/9' 2>&1 | head -1; echo "exit status: ${PIPESTATUS[0]}"
bash: connect: Connection refused
exit status: 1
Serve a folder over HTTP in one line (for quick file sharing on a trusted network; stop with Ctrl+C)
$ python3 -m http.server 8000 --directory ~/shared
Show my public IP address (or ask your own server: see the What's My IP tool on this site)
$ curl -s https://ifconfig.me ; echo
SSH: save typing, jump hosts and tunnels
A host entry in ~/.ssh/config turns a long command into "ssh web". Keep the file private: chmod 600 ~/.ssh/config
$ cat <<'EOF'
> Host web
> HostName 192.0.2.10
> User deploy
> IdentityFile ~/.ssh/id_ed25519
> IdentitiesOnly yes
> ServerAliveInterval 30
>
> Host internal-db
> HostName 10.0.5.20
> User admin
> ProxyJump web
> EOF
Host web
HostName 192.0.2.10
User deploy
IdentityFile ~/.ssh/id_ed25519
IdentitiesOnly yes
ServerAliveInterval 30
Host internal-db
HostName 10.0.5.20
User admin
ProxyJump web
$ ssh web # now short
$ ssh -J web admin@10.0.5.20 # one-off jump through a bastion host
$ ssh web 'df -h /; uptime' # run commands and come back
$ ssh -t web 'sudo journalctl -u nginx -n 20' # -t gives sudo/less a terminal to work with
$ ssh web 'bash -s' < local-script.sh # run a local script on the remote machine
$ ssh -L 8080:127.0.0.1:80 web # local port 8080 now reaches port 80 on web (browse http://localhost:8080)
$ ssh -D 1080 web # a SOCKS proxy on localhost:1080 through web
$ ssh -R 9000:localhost:3000 web # expose YOUR localhost:3000 as port 9000 on web (use carefully)
$ ssh-keygen -t ed25519 -C "laptop 2026" # modern key; ssh-copy-id web installs it
$ ssh-keygen -lf ~/.ssh/id_ed25519.pub # fingerprint of a key
$ ssh-keygen -R 192.0.2.10 # forget a host key after you REBUILT that server (not when you do not know why it changed!)
$ ssh -v web # debug: -v, -vv, -vvv give more detail when a login fails
Do not paste private keys anywhere. Prefer one key per machine, a passphrase, and ssh-agent (ssh-add) so you type the passphrase once.
Copy files: scp, rsync and tar over ssh
$ scp report.pdf web:/tmp/ # one file to a server; scp -r dir web:/tmp/ for a folder
$ scp web:/var/log/nginx/error.log . # one file back
$ rsync -avh --progress src/ web:/backup/src/ # the workhorse: only changed data is sent, permissions and times are kept
$ rsync -avh --delete --dry-run src/ web:/backup/src/ # --delete removes extra files at the destination: ALWAYS rehearse with --dry-run (-n) first
$ tar czf - folder | ssh web 'tar xzf - -C /tmp' # stream a folder without making an archive file
$ ssh web 'sudo tar czf - /etc' > etc-backup.tgz # pull a root-only folder through an ssh pipe
rsync's trailing slash matters: src/ means "the contents of src", src means "the folder src itself". Try the copy with -n first and read the list. A local rehearsal, with real output (-i itemizes what would change; -n changes nothing; a+ means new, "st" means size and time differ). rsync decides by size and modification time, so two different files with the same size AND time look "unchanged": add -c to compare checksums instead.
$ rsync -a -n -i src/ dst/
>f+++++++++ a.txt
>f.st...... b.txt
$ rsync -a -i src/ dst/ >/dev/null; rsync -a -n -i src/ dst/; echo "(the second run lists nothing: already in sync)"
(the second run lists nothing: already in sync)
$ sshfs web:/var/www ~/server-www # mount a remote folder (see the SSHFS guide on the Sysadmin tab)
Hashes, encoding, random data, JSON and CSV
Checksums: prove two files are the same
$ printf 'abc' | sha256sum
ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad -
$ printf 'abc' | md5sum
900150983cd24fb0d6963f7d28e17f72 -
$ echo "hello" > hello.txt; sha256sum hello.txt > hello.txt.sha256; cat hello.txt.sha256
5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03 hello.txt
$ sha256sum -c hello.txt.sha256
hello.txt: OK
$ echo "tampered" > hello.txt; sha256sum -c hello.txt.sha256 || echo "checksum FAILED: do not trust this file"
hello.txt: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match
checksum FAILED: do not trust this file
Downloads: compare against the checksum the publisher gives you (from a different page than the file if possible): echo "EXPECTED_HASH file.iso" | sha256sum -c -
Encoding: base64, hex and friends
$ printf 'hello world' | base64
aGVsbG8gd29ybGQ=
$ echo 'aGVsbG8gd29ybGQ=' | base64 -d; echo
hello world
$ printf 'user:secret' | base64
dXNlcjpzZWNyZXQ=
$ printf 'hello' | xxd -p
68656c6c6f
$ echo '68656c6c6f' | xxd -r -p; echo
hello
$ printf 'hi\n' | xxd
00000000: 6869 0a hi.
$ printf 'a b&c=d' | python3 -c 'import sys, urllib.parse; print(urllib.parse.quote(sys.stdin.read()))'
a%20b%26c%3Dd
$ printf 'A longer text\n' | od -An -tx1
41 20 6c 6f 6e 67 65 72 20 74 65 78 74 0a
base64 is NOT encryption: anybody can decode it. It only makes bytes printable (for example for HTTP basic authentication headers or certificates in PEM files).
Random data and passwords
$ openssl rand -hex 16 # 32 random hex characters: a token
$ openssl rand -base64 24 # a random secret
$ tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 20; echo # a 20-character password
$ uuidgen # a random UUID
$ shuf -i 1000-9999 -n 1 # a random 4-digit number
$ head -c 32 /dev/urandom | xxd -p -c 32 # raw random bytes as hex
Never use $RANDOM for secrets: it is predictable and only 0..32767.
JSON with jq
$ jq . servers.json | head -8
{
"region": "eu-west",
"servers": [
{
"name": "web01",
"ip": "10.0.0.11",
"cpu": 4,
"tags": [
$ jq -r '.region' servers.json
eu-west
$ jq -r '.servers[].name' servers.json
web01
web02
db01
$ jq -r '.servers[] | select(.cpu >= 4) | .name' servers.json
web01
db01
$ jq '.servers | length' servers.json
3
$ jq -r '.servers[] | "\(.name) has \(.cpu) cpus at \(.ip)"' servers.json
web01 has 4 cpus at 10.0.0.11
web02 has 2 cpus at 10.0.0.12
db01 has 8 cpus at 10.0.0.21
$ jq -r '.servers[] | [.name, .ip, .cpu] | @tsv' servers.json | column -t
web01 10.0.0.11 4
web02 10.0.0.12 2
db01 10.0.0.21 8
$ jq -r '.servers[] | [.name, .cpu] | @csv' servers.json
"web01",4
"web02",2
"db01",8
$ jq -c '.servers[] | {name, ip}' servers.json
{"name":"web01","ip":"10.0.0.11"}
{"name":"web02","ip":"10.0.0.12"}
{"name":"db01","ip":"10.0.0.21"}
$ jq '[.servers[].cpu] | add' servers.json
14
$ jq -r '.servers[] | select(.tags | index("prod")) | .name' servers.json
web01
db01
$ jq -r --arg n web02 '.servers[] | select(.name == $n) | .ip' servers.json
10.0.0.12
$ jq '.servers |= map(. + {"checked": true}) | .servers[0]' servers.json
{
"name": "web01",
"ip": "10.0.0.11",
"cpu": 4,
"tags": [
"web",
"prod"
],
"checked": true
}
$ jq -r '.servers | map(.cpu) | max' servers.json
8
-r prints raw text (no quotes), -c prints compact one-line JSON, select() filters, map() transforms a list, --arg passes in a shell value safely (never build jq programs by pasting shell variables into the filter). jq . pretty-prints and validates.
Format and check JSON without jq
$ echo '{"b":2,"a":1}' | python3 -m json.tool
{
"b": 2,
"a": 1
}
$ echo '{"broken": ' | python3 -m json.tool
Expecting value: line 2 column 1 (char 12)
# (exit status 1)
Build JSON from a shell script safely
$ name="web 03"; jq -n --arg name "$name" --argjson cpu 4 '{name: $name, cpu: $cpu}'
{
"name": "web 03",
"cpu": 4
}
Query an API and pick one field
$ curl -s https://api.github.com/repos/torvalds/linux | jq -r '.stargazers_count'
$ curl -s -H "Authorization: Bearer $TOKEN" https://api.example.com/items | jq -r '.items[] | [.id, .name] | @tsv'
YAML: yq (a separate tool, similar syntax) or: python3 -c 'import sys, yaml, json; print(json.dumps(yaml.safe_load(sys.stdin)))' < file.yaml | jq .
CSV and tables at the prompt
$ column -s, -t hosts.csv
host cpu mem
web01 4 16
db01 8 64
$ awk -F, 'NR > 1 && $3 > 32 {print $1}' hosts.csv
db01
$ python3 -c 'import csv, sys; [print(r["host"], int(r["cpu"]) * 2) for r in csv.DictReader(open("hosts.csv"))]'
web01 8
db01 16
$ sort -t, -k3 -nr hosts.csv | head -2
db01,8,64
web01,4,16
Compression
$ gzip -k nums.txt; ls -l nums.txt nums.txt.gz | awk '{print $5, $9}'
108894 nums.txt
45013 nums.txt.gz
$ zcat nums.txt.gz | wc -l
20000
$ zstd -q -k nums.txt -o nums.txt.zst && zstd -dc nums.txt.zst | head -2
1
2
$ xz -k -T0 nums.txt && ls nums.txt.xz
nums.txt.xz
$ tar czf archive.tgz nums.txt && tar tzf archive.tgz
nums.txt
gzip is everywhere, xz compresses smaller but slower, zstd is fast with good compression. -k keeps the original, -T0 uses all cores. The tar sheet on this site covers archives in depth.
Background, scheduled and long-running work
Run in the background, wait, and collect the result
$ sleep 1 & echo "started in the background"; wait; echo "finished waiting"
started in the background
finished waiting
$ (sleep 0.3; echo "task A done") & (sleep 0.1; echo "task B done") & wait; echo "all tasks done"
task B done
task A done
all tasks done
$ sleep 0.2 & pid=$!; wait "$pid"; echo "the background job ended with status $?"
the background job ended with status 0
$ bash -c 'trap "echo caught TERM, cleaning up; exit 0" TERM; while true; do sleep 0.1; done' & pid=$!; sleep 0.4; kill -TERM $pid; wait $pid; echo "stopped politely, status $?"
caught TERM, cleaning up
stopped politely, status 0
$ command > output.log 2>&1 & # start now, detach the output
$ jobs -l; fg %1; bg %1 # list, bring to the front, resume in the background (after Ctrl+Z)
$ disown -h %1 # let a job survive when you close the terminal
Keep it running after you log out
$ nohup ./long-job.sh > long-job.log 2>&1 & # ignore the hangup signal, log to a file
$ setsid ./long-job.sh > long-job.log 2>&1 < /dev/null & # a new session: fully detached
$ tmux new -s work # best for interactive work: detach with Ctrl+B then D, reattach with tmux attach -t work (see the Tmux sheet)
$ systemd-run --user --unit=myjob ./long-job.sh && journalctl --user -u myjob -f # let systemd run, supervise and log it
Run it later
$ echo 'tar czf /backup/home-$(date +%F).tgz /home' | at 02:30 # once, tonight (needs the atd service); atq lists, atrm removes
$ systemd-run --on-calendar='*-*-* 03:30:00' --unit=nightly-job /usr/local/bin/nightly.sh # a systemd timer without writing unit files
$ systemd-run --on-active=10m echo "ten minutes passed" # one-shot, relative to now
$ crontab -e # edit your recurring jobs; crontab -l lists them
A crontab line has five time fields, then the command: minute hour day-of-month month day-of-week. Examples (the Cron Generator on the Tools tab builds and explains them):
$ cat <<'EOF'
> */5 * * * * /usr/local/bin/check.sh # every 5 minutes
> 30 2 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1 # daily at 02:30, keep the output
> 0 9 * * 1-5 /usr/local/bin/report.sh # weekdays at 09:00
> @reboot /usr/local/bin/start-tunnel.sh # at every boot
> EOF
*/5 * * * * /usr/local/bin/check.sh # every 5 minutes
30 2 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1 # daily at 02:30, keep the output
0 9 * * 1-5 /usr/local/bin/report.sh # weekdays at 09:00
@reboot /usr/local/bin/start-tunnel.sh # at every boot
cron runs with a tiny environment (almost no PATH): use full paths to commands, redirect output to a log, and use flock so a slow run is not started twice: */5 * * * * flock -n /tmp/check.lock /usr/local/bin/check.sh
Limit it, repeat it
$ timeout 1 sleep 3; echo "status $? (124 = timed out)"
status 124 (124 = timed out)
$ timeout -k 1 5 bash -c 'echo "polite request first, forced kill after 1 more second"'
polite request first, forced kill after 1 more second
$ nice -n 10 ./heavy-job.sh # lower the CPU priority (higher number = nicer)
$ ionice -c3 ./heavy-job.sh # only use the disk when nobody else does
$ systemd-run --scope -p MemoryMax=500M ./hungry.sh # a memory ceiling for one command
$ watch -n 10 'systemctl is-active nginx'
$ timeout 300 bash -c 'until curl -fs http://localhost:8080/health; do sleep 2; done' # wait up to 5 minutes for a service
Get a notification when a long command finishes
$ make build; notify-send "build finished: status $?" # desktop notification
$ ./long-job.sh; printf '\a' # a terminal bell
$ ./long-job.sh && echo "done" || echo "FAILED" | mail -s "job status" me@example.com # needs a working mail command
Safety habits: how to avoid the classic "oops"
Look before you delete
$ echo safe/old/*
safe/old/-rf safe/old/a.txt safe/old/two words.txt
$ ls safe/old
-rf
a.txt
two words.txt
$ rm -- safe/old/-rf; ls safe/old
a.txt
two words.txt
Preview with echo or ls first; then run the same line without the preview. A file called -rf (or -n, or any name beginning with a dash) is read as an option unless you put -- before the names, which means "no more options". Make it a habit for rm, mv, cp, grep and touch when the names come from variables or globs.
Quote your variables
$ f="two words.txt"; (cd safe/old && ls $f 2>&1 | head -2)
ls: cannot access 'two': No such file or directory
ls: cannot access 'words.txt': No such file or directory
$ (cd safe/old && ls "$f")
two words.txt
Without quotes the shell splits the value at spaces and expands wildcards in it. "$var" is almost always what you meant; use "${array[@]}" for arrays.
Guard variables that decide WHAT gets deleted
$ bash -c 'dir=; echo "would run: rm -rf ${dir:?dir is empty, refusing}/"'
bash: line 1: dir: dir is empty, refusing
# (exit status 127)
$ dir=$PWD/safe/new; echo "would run: rm -rf ${dir:?}/*"
would run: rm -rf /home/demo/safe/new/*
${var:?message} stops the script when the variable is empty or unset. Without it, rm -rf "$dir/" with an empty $dir is rm -rf /. In scripts also use set -u.
Prefer commands that cannot go wrong in the same way
$ cp --update=none safe/old/a.txt safe/new/ && cp --update=none safe/old/a.txt safe/new/ && ls safe/new
a.txt
$ mv -n safe/old/a.txt safe/new/a.txt; ls safe/old
mv: not replacing 'safe/new/a.txt'
a.txt
two words.txt
$ ln -sfn "$PWD/safe/new" current; readlink current
/home/demo/safe/new
--update=none (cp) and -n (mv) refuse to overwrite an existing file (cp -n still works but newer GNU cp prints a warning and prefers --update=none; on older systems and macOS use cp -n); -i asks first; --backup=numbered keeps the old copy; ln -sfn replaces a symlink atomically. rm -I asks once before deleting more than three files; rmdir only removes EMPTY folders; trash-cli (trash-put) moves files to the trash instead of deleting them.
Try dangerous changes on a copy, or with a dry run
$ rsync -avn --delete src/ dst/ # -n = rehearsal
$ sed -n 's/old/new/p' file # print what WOULD change, then use sed -i.bak
$ find . -name '*.tmp' -print # look at the list, THEN add -delete
$ apt-get -s upgrade # simulate an upgrade; dnf --assumeno upgrade
$ terraform plan; kubectl diff -f app.yaml; ansible-playbook site.yml --check --diff # the same idea in other tools
$ sudo visudo; sudo sshd -t; sudo nginx -t # edit-with-a-syntax-check, or test-before-reload: never reload a broken config
Keep a second session open when you change SSH, the firewall or the network: if you lock yourself out, the open session is your way back in. At the console, "sudo shutdown -c" cancels a pending reboot.
History and repetition
$ sudo !! # repeat the last command with sudo
$ !$ # the last argument of the previous command (or press Alt+.)
$ cd - # back to the previous directory
$ history | grep ssh # find an old command; Ctrl+R searches as you type
$ fc # edit the last command in $EDITOR and run it when you save
$ ^old^new # repeat the previous command with old replaced by new
$ HISTCONTROL=ignoreboth # do not record commands that start with a space (use that for commands with passwords) or duplicates
echo is not printf, [ is not [[, which is not command -v
$ echo -n "no newline"; echo; echo "a\tb"; printf 'a\tb\n'
no newline
a\tb
a b
$ x=; [ -z $x ] && echo "worked by luck"; [ -z "$x" ] && echo "worked on purpose"
worked by luck
worked on purpose
$ [[ "ab" == a* ]] && echo "[[ understands patterns"; [ "ab" = "a*" ] || echo "[ compares text exactly"
[[ understands patterns
[ compares text exactly
$ type -a echo | head -2
echo is a shell builtin
echo is /usr/bin/echo
Prefer printf for anything with escapes or variables that might start with a dash. Use [[ ]] in bash scripts (safe with empty values, supports patterns and &&), and [ ] in scripts that must run under plain sh.
Things that look right and are not
$ for f in $(printf 'a b\nc\n'); do echo "[$f]"; done
[a]
[b]
[c]
$ printf 'a b\nc\n' | while IFS= read -r l; do echo "[$l]"; done
[a b]
[c]
$ x=$(printf 'text\n\n\n'); echo "[$x]"
[text]
$ echo $(( 010 + 1 ))
9
$ ( cd /nonexistent 2>/dev/null; echo "the script went on anyway, still in $PWD" )
the script went on anyway, still in /home/demo
Parsing "for f in $(ls)" breaks on spaces; use a glob (for f in *) or read the lines. A command substitution strips trailing newlines. A leading 0 makes a number octal (010 is 8). And a failed cd does not stop a script unless you write cd dir || exit 1 (or use set -e): the next rm then runs in the wrong folder.
One-liner hall of fame and tips
The examples that show output were run; the others (ssh, curl, history tricks) need a real server or an interactive shell.
$ echo $PATH | tr ':' '\n'
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin
$ printf '%s\n' "$PATH" | tr ':' '\n' | sort | uniq -d | head -1; echo "(no output above = no duplicate PATH entries)"
(no output above = no duplicate PATH entries)
$ find tree -type d -empty
tree/empty
$ find tree -type f | sed 's|/[^/]*$||' | sort | uniq -c
1 tree/a
1 tree/a/b
$ ls -1 tree/a | wc -l
2
$ awk '!seen[$0]++' pairs.txt
b 2
a 3
c 1
$ sort pairs.txt | uniq -c | sort -rn | head -1
2 a 3
$ seq 1 4 | paste -sd+ | bc
10
$ seq 1 5 | awk '{s += $1} END {print "sum:", s, "average:", s/NR}'
sum: 15 average: 3
$ yes | head -3 | tr '\n' ' '; echo
y y y
$ echo {a,b,c}{1,2}
a1 a2 b1 b2 c1 c2
$ mkdir -p proj/{src,docs,tests}; ls proj
docs
src
tests
$ x=report.final.tar.gz; echo "${x%%.*} | ${x%.*} | ${x#*.} | ${x##*.}"
report | report.final.tar | final.tar.gz | gz
$ s="Hello World"; echo "${s,,} ${s^^} ${#s} ${s:6:5} ${s/World/there}"
hello world HELLO WORLD 11 World Hello there
$ arr=(red green blue); echo "${arr[1]} ${#arr[@]} ${arr[@]:1}"; for c in "${arr[@]}"; do printf '<%s>' "$c"; done; echo
green 3 green blue
<red><green><blue>
$ declare -A age=([ann]=31 [bob]=27); for k in "${!age[@]}"; do echo "$k=${age[$k]}"; done | sort
ann=31
bob=27
$ printf '%s\n' a b c | xargs -I{} printf '<%s>\n' {}
<a>
<b>
<c>
$ echo "x=1;y=2" | tr ';' '\n' | cut -d= -f2 | paste -sd,
1,2
$ [ -d tree ] && echo dir || echo missing; [ -s pairs.txt ] && echo "non-empty"; [ -x /bin/ls ] && echo "executable"
dir
non-empty
executable
$ true && echo "and-then"; false || echo "or-else"; { false; echo "after false"; }
and-then
or-else
after false
$ (exit 3); echo "last status: $?"
last status: 3
$ f() { echo "args: $# first: $1 all: $*"; }; f one "two words" three
args: 3 first: one all: one two words three
$ cd tree && ls -d */ && cd ..
a/
empty/
$ command -v bash; type cd | head -1
/usr/bin/bash
cd is a shell builtin
$ ssh -t host 'sudo systemctl restart nginx'
$ curl -fsSL https://example.com/install.sh | less # READ a script before piping it into a shell (curl ... | bash is convenient and risky)
$ ls -lah | sort -k5 -h | tail -5
$ du -sh .[!.]* * 2>/dev/null | sort -h | tail
$ grep -rIl --include='*.conf' 'ServerName' /etc 2>/dev/null
$ diff <(ssh web1 cat /etc/hosts) <(ssh web2 cat /etc/hosts) # compare a file on two machines
$ history | awk '{print $2}' | sort | uniq -c | sort -rn | head # your most used commands
$ for h in web1 web2 db1; do echo "== $h"; ssh "$h" uptime; done # same command on several hosts
Tips
Learn the shortcuts of ONE editor and ONE pager (vim or nano, less) well. They are on every server you will ever log in to.
Quote variables, use set -euo pipefail in scripts, and prefer printf to echo. These three habits prevent most shell bugs.
Use ISO dates (2026-03-08, 20260308-140509) in file names, and UTC in logs. Your future self sorts with ls and nobody has to guess whether 03/04 is March or April.
Pipelines are debugged from the left: run the first command alone, then add one stage at a time, looking at the output after each |.
Test destructive commands with echo, ls, -n or --dry-run first, and keep a backup you have actually restored from at least once.
man command shows the manual, command --help the short version, apropos word finds commands by topic, tldr command (if installed) shows examples, and type command tells you what a name really is.
Write down the commands that solved a hard problem in a small notes file or in a script, with a date and a one-line reason. "How did I fix that last time?" is the most common question in operations.
When a one-liner gets longer than a line or two, make it a script. When a script gets longer than a page, ask whether it should be Python, or a proper tool.
Do not run commands from the internet as root without reading them. Do not paste commands that you do not understand into a production shell. Do not leave a password in your shell history (start the line with a space with HISTCONTROL=ignoreboth, or use read -rs).
The shell has been "dying" since 1995. It is still the first thing you open when something is on fire.
