💾 ZFS Backups to Another Machine: Send, Receive, Restore
A real backup routine between two FreeBSD machines: full and incremental sends, a restore test checked with checksums, an interrupted transfer that resumes, and a script that runs from cron and tidies up after itself
A snapshot is not a backup
The ZFS page showed snapshots: instant, nearly free, and great against mistakes like rm in the wrong folder. But a snapshot lives in the same pool, on the same disk. If the disk dies, the machine is stolen or the pool is destroyed, the snapshots go with it.
A backup is a copy on another machine. ZFS has a built-in way to make one, and it is more pleasant than most tools: zfs send turns a snapshot into a stream of bytes, and zfs receive on the other side turns the stream back into a dataset with all its snapshots. In between you put ssh. And because ZFS knows exactly which blocks changed between two snapshots, the second and later copies send only the changes.
This page builds a complete routine on two real FreeBSD 15.0 virtual machines, one called prod (the machine with the data) and one called backup. For each step you see the real output. Then we break things on purpose: we destroy the data and restore it, cut a transfer in half, and modify the backup by accident.
The setup
The two machines can reach each other (here on a private network, 192.168.99.1 and 192.168.99.2):
root@prod:~ # ping -c 2 192.168.99.2 PING 192.168.99.2 (192.168.99.2): 56 data bytes 64 bytes from 192.168.99.2: icmp_seq=0 ttl=64 time=1.389 ms 64 bytes from 192.168.99.2: icmp_seq=1 ttl=64 time=1.505 ms --- 192.168.99.2 ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 1.389/1.447/1.505/0.058 ms root@prod:~ # hostname; zpool list -H -o name,size,free zroot prod zroot 13.5G 5.88G root@backup:~ # hostname; zpool list -H -o name,size,free zroot backup zroot 13.5G 5.88G
On prod the data lives in its own dataset, zroot/data: three small reports, a file of numbers and a 20 MB random "photo". After creating it we wrote down a checksum of every file. That list is the proof we will use at the end.
root@prod:~ # zfs create zroot/data root@prod:~ # mkdir /zroot/data/docs root@prod:~ # for i in 1 2 3; do echo "Quarterly report $i: all figures are made up" > /zroot/data/docs/report-$i.txt; done root@prod:~ # seq 1 200000 > /zroot/data/numbers.txt root@prod:~ # dd if=/dev/random of=/zroot/data/photo.bin bs=1m count=20 2>&1 | tail -1 20971520 bytes transferred in 0.087078 secs (240836660 bytes/sec) root@prod:~ # ls -lR /zroot/data | grep -v "^total" drwxr-xr-x 2 root wheel 5 Oct 6 18:29 docs -rw-r--r-- 1 root wheel 1288895 Oct 6 18:29 numbers.txt -rw-r--r-- 1 root wheel 20971520 Oct 6 18:29 photo.bin /zroot/data/docs: -rw-r--r-- 1 root wheel 44 Oct 6 18:29 report-1.txt -rw-r--r-- 1 root wheel 44 Oct 6 18:29 report-2.txt -rw-r--r-- 1 root wheel 44 Oct 6 18:29 report-3.txt root@prod:~ # (cd /zroot/data && find . -type f | sort | xargs sha256 -r) > /root/data.sha256; cat /root/data.sha256 298f5078e654546ff60c40327a06482cd1100d125e6ac9d254bc0cc65be670a1 ./docs/report-1.txt 9e3a2d53a2e209df396460918a061329b2cd003c8dbec694dbb462a532d99e02 ./docs/report-2.txt d2db883e457de0b2e01aedd1e8d7cb2c903b75f71afaf7b67ebf461f52e4016c ./docs/report-3.txt 5af7b95208fdcff454bab3f5eddf567a688a3796c703d4fef91072e38645c062 ./numbers.txt ab817f163235308fdc5bab88ef69c4d8e85b47fdb051c680fd824f3b6a47ebcb ./photo.bin
A user that can only do backups
Do not send backups as root. On the backup machine we create an ordinary user, a dataset to hold the backups, and then use ZFS delegation (zfs allow) to give that user just the ZFS powers it needs, on just that dataset:
root@backup:~ # pw useradd zbackup -m -s /bin/sh -c "ZFS backup receiver" root@backup:~ # zfs create zroot/backups root@backup:~ # zfs allow -u zbackup create,mount,receive,send zroot/backups root@backup:~ # zfs allow zroot/backups ---- Permissions on zroot/backups ------------------------------------ Local+Descendent permissions: user zbackup create,mount,receive,send
create, mount and receive let the user receive a stream into zroot/backups. send lets it send things back, which we need for the restore. Nothing here lets it touch other datasets, and it has no root rights.
The key
On prod, a key made for this one job, with no passphrase so that cron can use it (so keep the file private: it is mode 600 by default). Its public half goes into the backup user's authorized_keys:
root@prod:~ # mkdir -p /root/.ssh && chmod 700 /root/.ssh root@prod:~ # ssh-keygen -t ed25519 -N '' -C 'prod zfs backup' -f /root/.ssh/zbackup_key Generating public/private ed25519 key pair. Your identification has been saved in /root/.ssh/zbackup_key Your public key has been saved in /root/.ssh/zbackup_key.pub The key fingerprint is: SHA256:Gvks7xPDTCvBapNjXi/jW1T0/KSKvd62/0qK7P86fv0 prod zfs backup The key's randomart image is: +--[ED25519 256]--+ | . | | . o | | . . o . | | o... + | | oo=S. . . | | B o**o . | | + +oo=oo .. | | . o+o. +oo. .| | .o=o+*+BB+oE| +----[SHA256]-----+ root@backup:~ # [install the public key in /home/zbackup/.ssh/authorized_keys, owned by zbackup] root@backup:~ # cut -c1-60 /home/zbackup/.ssh/authorized_keys ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIP73WvyVE8xxTDbvLCW0yLyG
The first login failed, and the failure is worth seeing, because it was our own hardening:
root@backup:~ # grep -n "^AllowUsers" /etc/ssh/sshd_config 131:AllowUsers bsd root@prod:~ # ssh -i /root/.ssh/zbackup_key -o BatchMode=yes -o StrictHostKeyChecking=accept-new zbackup@192.168.99.2 hostname 2>&1 | tail -2 Warning: Permanently added '192.168.99.2' (ED25519) to the list of known hosts. zbackup@192.168.99.2: Permission denied (publickey). root@backup:~ # sed -i '' 's/^AllowUsers bsd$/AllowUsers bsd zbackup/' /etc/ssh/sshd_config root@backup:~ # sshd -t && service sshd reload Performing sanity check on sshd configuration. root@prod:~ # ssh -i /root/.ssh/zbackup_key -o BatchMode=yes zbackup@192.168.99.2 "hostname; id -un; zfs list -H -o name,avail zroot/backups" backup zbackup zroot/backups 5.46G
The backup machine had the AllowUsers bsd line from the hardening page, so the new user was refused. Permission denied (publickey) does not say why, so when a key you installed correctly is refused, look at AllowUsers and the auth log. We added the user to the line, checked with sshd -t, reloaded, and the login worked: the user is zbackup, and zfs list works for it.
The first copy: a full send
Make a snapshot, and ask how big the stream will be before you send it (-n is a dry run, -v prints the estimate; -c estimates the compressed stream):
root@prod:~ # export K=/root/.ssh/zbackup_key B=zbackup@192.168.99.2 root@prod:~ # zfs snapshot zroot/data@mon root@prod:~ # zfs send -nv zroot/data@mon full send of zroot/data@mon estimated size is 21.3M total estimated size is 21.3M root@prod:~ # zfs send -c -nv zroot/data@mon full send of zroot/data@mon estimated size is 20.9M total estimated size is 20.9M root@prod:~ # zfs send zroot/data@mon | ssh -i $K $B zfs receive -u zroot/backups/prod-data root@backup:~ # zfs list -r -o name,used,mountpoint,mounted zroot/backups NAME USED MOUNTPOINT MOUNTED zroot/backups 21.0M /zroot/backups yes zroot/backups/prod-data 20.9M /zroot/backups/prod-data no root@backup:~ # zfs list -t snapshot -r zroot/backups NAME USED AVAIL REFER MOUNTPOINT zroot/backups/prod-data@mon 0B - 20.9M - root@backup:~ # zfs set readonly=on zroot/backups/prod-data && zfs get -H -o property,value readonly zroot/backups/prod-data readonly on
The real send is one line:
zfs send zroot/data@mon | ssh -i $K $B zfs receive -u zroot/backups/prod-data
- zfs send writes the stream to its output. ssh carries it, encrypted, and zfs receive on the far side builds the dataset zroot/backups/prod-data from it.
- -u means "do not mount it". A backup copy that mounts itself at the same place as other things is a surprise waiting to happen. The listing above shows MOUNTED no.
- The snapshot @mon arrived with it. Snapshots are the unit of backup: the copy is exactly that snapshot, frozen.
- Finally we set readonly=on on the copy. Nobody, not even root on that machine, can now change files in it by accident. (A received dataset can still receive more snapshots: ZFS itself is allowed to update it.)
The next copies: incremental sends
The next day the data changed: a report edited, one deleted, one added, and more numbers appended. We took the checksums again, then a new snapshot, and zfs diff shows what ZFS knows changed (M modified, - removed, + added):
root@prod:~ # echo "Quarterly report 2: REVISED, all figures are still made up" > /zroot/data/docs/report-2.txt root@prod:~ # echo "Quarterly report 4: new" > /zroot/data/docs/report-4.txt root@prod:~ # rm /zroot/data/docs/report-3.txt root@prod:~ # seq 200001 210000 >> /zroot/data/numbers.txt root@prod:~ # (cd /zroot/data && find . -type f | sort | xargs sha256 -r) > /root/data-tue.sha256; cat /root/data-tue.sha256 298f5078e654546ff60c40327a06482cd1100d125e6ac9d254bc0cc65be670a1 ./docs/report-1.txt 08d8770487a762b8705c02b75fb9eaccd84514edaae1914705f104d7a927b049 ./docs/report-2.txt d091b49868e0a5e279b5c25f8c0487e2fbe6998af0bdd4142363bbb5c6bc391b ./docs/report-4.txt 4210e36024f7a97b6fc33126cc996e19db9c6efc0bca0a664fd74a7ff1176649 ./numbers.txt ab817f163235308fdc5bab88ef69c4d8e85b47fdb051c680fd824f3b6a47ebcb ./photo.bin root@prod:~ # zfs snapshot zroot/data@tue root@prod:~ # zfs diff zroot/data@mon zroot/data@tue M /zroot/data/docs M /zroot/data/numbers.txt M /zroot/data/docs/report-2.txt - /zroot/data/docs/report-3.txt + /zroot/data/docs/report-4.txt
Now the send names the previous snapshot with -i. Watch the estimated size:
root@prod:~ # zfs send -nv -i @mon zroot/data@tue send from zroot/data@mon to zroot/data@tue estimated size is 298K total estimated size is 298K root@prod:~ # zfs send -i @mon zroot/data@tue | ssh -i $K $B zfs receive -u zroot/backups/prod-data root@prod:~ # zfs list -t snapshot -o name,used,refer zroot/data NAME USED REFER zroot/data@mon 144K 20.9M zroot/data@tue 0B 21.0M root@backup:~ # zfs list -t snapshot -o name,used,refer -r zroot/backups NAME USED REFER zroot/backups/prod-data@mon 144K 20.9M zroot/backups/prod-data@tue 0B 21.0M
298K instead of 21 MB. This is the heart of the whole routine: every later copy costs only what changed, so you can run it every night, or every five minutes, without caring. Both machines now show the same two snapshots.
The rule to remember: an incremental send needs the older snapshot to exist on both sides. If you delete the common snapshot from either machine, the next incremental cannot work and you must start again with a full send. The script below makes sure that never happens.
Restoring one file
Your user deleted report-3.txt and wants it back. The backup is a normal ZFS dataset with the hidden .zfs/snapshot folder, so you can look inside any snapshot and simply read the file:
root@backup:~ # zfs mount zroot/backups/prod-data root@backup:~ # ls /zroot/backups/prod-data/.zfs/snapshot mon tue root@backup:~ # cat /zroot/backups/prod-data/.zfs/snapshot/mon/docs/report-3.txt Quarterly report 3: all figures are made up root@prod:~ # ssh -i $K $B cat /zroot/backups/prod-data/.zfs/snapshot/mon/docs/report-3.txt > /zroot/data/docs/report-3.txt; cat /zroot/data/docs/report-3.txt Quarterly report 3: all figures are made up
We mounted the backup dataset (it is set to -u, so it was not mounted), listed the snapshots, and read the file out of @mon. The second command, from prod, runs cat over ssh and saves the output as the original: restored. For a few files this is all you need. No tool, no catalogue.
Restoring everything: the disaster test
This is the test that makes it a backup. We took one more snapshot (@wed) and sent it, with a checksum list taken first:
root@prod:~ # (cd /zroot/data && find . -type f | sort | xargs sha256 -r) > /root/data-wed.sha256; wc -l /root/data-wed.sha256
6 /root/data-wed.sha256
root@prod:~ # zfs snapshot zroot/data@wed
root@prod:~ # zfs send -i @tue zroot/data@wed | ssh -i $K $B zfs receive -u zroot/backups/prod-data
root@backup:~ # zfs list -t snapshot -o name,used,refer -r zroot/backups
NAME USED REFER
zroot/backups/prod-data@mon 144K 20.9M
zroot/backups/prod-data@tue 64K 21.0M
zroot/backups/prod-data@wed 0B 21.0M
Then we destroyed the dataset on prod, completely:
root@prod:~ # zfs destroy -r zroot/data root@prod:~ # ls /zroot/data ls: /zroot/data: No such file or directory root@prod:~ # zfs list zroot/data cannot open 'zroot/data': dataset does not exist
Now the restore. It is the same pipe, in the other direction: send from the backup machine, receive on prod:
root@prod:~ # ssh -i $K $B zfs send zroot/backups/prod-data@wed | zfs receive zroot/data root@prod:~ # zfs list -t snapshot zroot/data NAME USED AVAIL REFER MOUNTPOINT zroot/data@wed 0B - 21.0M - root@prod:~ # ls -l /zroot/data /zroot/data/docs | grep -v "^total" /zroot/data: drwxr-xr-x 2 root wheel 6 Oct 6 18:30 docs -rw-r--r-- 1 root wheel 1358895 Oct 6 18:30 numbers.txt -rw-r--r-- 1 root wheel 20971520 Oct 6 18:29 photo.bin /zroot/data/docs: -rw-r--r-- 1 root wheel 44 Oct 6 18:29 report-1.txt -rw-r--r-- 1 root wheel 59 Oct 6 18:30 report-2.txt -rw-r--r-- 1 root wheel 44 Oct 6 18:30 report-3.txt -rw-r--r-- 1 root wheel 24 Oct 6 18:30 report-4.txt root@prod:~ # (cd /zroot/data && find . -type f | sort | xargs sha256 -r) | diff - /root/data-wed.sha256 && echo IDENTICAL: every file matches the checksums taken before the disaster IDENTICAL: every file matches the checksums taken before the disaster
The files are back, and the checksum comparison says IDENTICAL: every one of the files matches what it was before the disaster, to the last byte. (Because we did not use -u this time, ZFS mounted the restored dataset where it belongs.)
Two things worth knowing:
- The restored dataset holds only the snapshot you sent (@wed). If you also want the older ones, use zfs send -R (a "replication stream" with all snapshots and properties): from the documentation, not run here.
- Because @wed now exists on both sides again, the incremental routine continues:
root@prod:~ # echo "Quarterly report 5: after the restore" > /zroot/data/docs/report-5.txt root@prod:~ # zfs snapshot zroot/data@thu root@prod:~ # zfs send -nv -i @wed zroot/data@thu send from zroot/data@wed to zroot/data@thu estimated size is 29.1K total estimated size is 29.1K root@prod:~ # zfs send -i @wed zroot/data@thu | ssh -i $K $B zfs receive -u zroot/backups/prod-data root@backup:~ # zfs list -t snapshot -o name -r zroot/backups NAME zroot/backups/prod-data@mon zroot/backups/prod-data@tue zroot/backups/prod-data@wed zroot/backups/prod-data@thu
The next snapshot, @thu, went as an incremental of 29.1K, and the backup machine has all four snapshots. The disaster did not even break the chain.
When a transfer is interrupted
A big first copy over a slow or flaky link can fail halfway. We made that happen by adding 100 MB of random data, then sending it and cutting the stream after 40 MB with head -c. The receive used -s, which keeps the partial data instead of throwing it away:
root@prod:~ # dd if=/dev/random of=/zroot/data/video.bin bs=1m count=100 2>&1 | tail -1
104857600 bytes transferred in 0.375049 secs (279583640 bytes/sec)
root@prod:~ # zfs snapshot zroot/data@fri
root@prod:~ # zfs send -nv -i @thu zroot/data@fri
send from zroot/data@thu to zroot/data@fri estimated size is 100M
total estimated size is 100M
root@prod:~ # zfs send -i @thu zroot/data@fri | head -c 40000000 | ssh -i $K $B zfs receive -s -u zroot/backups/prod-data 2>&1 | tail -2
warning: cannot send 'zroot/data@fri': signal received
A resuming stream can be generated on the sending system by running:
zfs send -t 1-d19ed33ee-e0-789c636064000310a501c49c50360710a715e5e7a69766a6304041e2d74bdbabfb79f915806c762475f94959a9c9250c0c4d507518f26969c5a92520993826983c1b927c5265496a319036989ac8844d7f493ec415dbdc3a37a7464ef8a98124cf0996cf4bcc4d6560a82acacf2fd14f492c4974482bca84b80500238720dd
root@backup:~ # zfs list -t snapshot -o name -r zroot/backups | tail -2
zroot/backups/prod-data@wed
zroot/backups/prod-data@thu
root@backup:~ # zfs get -H -o property,value receive_resume_token zroot/backups/prod-data | cut -c1-90
receive_resume_token 1-d19ed33ee-e0-789c636064000310a501c49c50360710a715e5e7a69766a6304041
ZFS printed the way forward by itself: "A resuming stream can be generated on the sending system", with a long token. The backup machine stored it as the property receive_resume_token. Now resume. First ask what the token holds, then send from it:
root@prod:~ # TOKEN=$(ssh -i $K $B zfs get -H -o value receive_resume_token zroot/backups/prod-data); echo ${#TOKEN} characters in the token
269 characters in the token
root@prod:~ # zfs send -nv -t $TOKEN
resume token contents:
nvlist version: 0
fromguid = 0xf0d8f7bb7d2f561
object = 0x82
offset = 0x25e0000
bytes = 0x2619530
toguid = 0xf9905965b38946b6
toname = zroot/data@fri
send from zroot/data@thu to zroot/data@fri estimated size is 62.1M
total estimated size is 62.1M
root@prod:~ # zfs send -t $TOKEN | ssh -i $K $B zfs receive -s -u zroot/backups/prod-data
root@backup:~ # zfs list -t snapshot -o name,refer -r zroot/backups | tail -2
zroot/backups/prod-data@thu 21.0M
zroot/backups/prod-data@fri 121M
root@backup:~ # zfs get -H -o property,value receive_resume_token zroot/backups/prod-data
receive_resume_token -
- zfs send -nv -t $TOKEN shows the token's contents (how far it got) and the estimated size of the rest: 62.1M of the 100M. It does not start over.
- After the real zfs send -t $TOKEN the snapshot @fri is there with 121M referenced, and receive_resume_token is - again (nothing left to resume).
Rule: use receive -s for big copies over unreliable links. Without -s, a failed receive leaves nothing and you start from zero.
The gotcha: someone modified the backup
The most common failure of this routine is that the backup copy was changed after its last snapshot. To show it, we turned off readonly on the backup and created a file in it. First the protection working, then the accident, then the next incremental:
root@backup:~ # zfs mount zroot/backups/prod-data root@backup:~ # touch /zroot/backups/prod-data/oops.txt touch: /zroot/backups/prod-data/oops.txt: Read-only file system root@backup:~ # zfs set readonly=off zroot/backups/prod-data && touch /zroot/backups/prod-data/oops.txt && ls /zroot/backups/prod-data docs numbers.txt oops.txt photo.bin video.bin root@prod:~ # echo "Quarterly report 6" > /zroot/data/docs/report-6.txt; zfs snapshot zroot/data@sat root@prod:~ # zfs send -i @fri zroot/data@sat | ssh -i $K $B zfs receive -u zroot/backups/prod-data 2>&1 | tail -3 cannot receive incremental stream: destination zroot/backups/prod-data has been modified since most recent snapshot
- With readonly=on, touch said Read-only file system. This is why we set it.
- With it off, a file oops.txt appeared, and the next incremental send failed: destination has been modified since most recent snapshot. ZFS refuses to apply changes on top of data that it cannot match with the snapshot. That is the right behaviour: the alternative is a silently wrong backup.
The fix is to roll the backup back to its last snapshot (throwing away the change) and send again:
root@backup:~ # zfs rollback zroot/backups/prod-data@fri root@backup:~ # ls /zroot/backups/prod-data docs numbers.txt photo.bin video.bin root@prod:~ # zfs send -i @fri zroot/data@sat | ssh -i $K $B zfs receive -u zroot/backups/prod-data 2>&1 | tail -3 root@backup:~ # zfs set readonly=on zroot/backups/prod-data; zfs unmount zroot/backups/prod-data; zfs list -t snapshot -o name -r zroot/backups | tail -3 zroot/backups/prod-data@thu zroot/backups/prod-data@fri zroot/backups/prod-data@sat
The rollback removed oops.txt, the same send then worked without any message, and @sat is on the backup. We turned readonly on again. In scripts you can also add -F to zfs receive, which does the rollback for you (from the documentation, not run here). We would rather not: if something is changing the backup, you want to hear about it.
A script for every night
Everything above is a routine now, so we turn it into a script, /root/zbackup.sh on prod. It does five things: asks the backup machine for its newest snapshot, makes a new snapshot, sends the difference, writes the result to the system log with logger, and deletes old snapshots so that the pools do not fill up (keeping the newest 4 here: use more in real life).
Pruning on the backup machine needs one more permission, so we added destroy:
root@backup:~ # zfs allow zroot/backups ---- Permissions on zroot/backups ------------------------------------ Local+Descendent permissions: user zbackup create,mount,receive,send root@backup:~ # zfs allow -u zbackup destroy,mount zroot/backups root@backup:~ # zfs allow zroot/backups ---- Permissions on zroot/backups ------------------------------------ Local+Descendent permissions: user zbackup create,destroy,mount,receive,send
Here is the final script, as written on prod:
root@prod:~ # cat > /root/zbackup.sh <<'EOF'
> #!/bin/sh
> # Snapshot zroot/data and send it to the backup machine.
> SRC=zroot/data
> DST=zroot/backups/prod-data
> R="ssh -o ConnectTimeout=10 -i /root/.ssh/zbackup_key zbackup@192.168.99.2"
> KEEP=4
> NEW=auto-$(date +%Y%m%d-%H%M%S)
>
> # newest snapshot the backup machine already has
> LAST=$($R zfs list -H -t snapshot -o name -s createtxg -d 1 $DST | tail -1 | cut -d@ -f2)
> if [ -z "$LAST" ]; then
> logger -t zbackup "FAILED: cannot read the snapshot list on the backup machine"
> exit 1
> fi
>
> zfs snapshot $SRC@$NEW || exit 1
>
> if zfs send -i @$LAST $SRC@$NEW | $R zfs receive -u $DST; then
> logger -t zbackup "ok: $LAST -> $NEW"
> else
> logger -t zbackup "FAILED sending $NEW (last on backup: $LAST)"
> exit 1
> fi
>
> # keep the newest $KEEP auto- snapshots on both sides
> prune() { grep "@auto-" | awk -v k=$KEEP '{a[NR]=$0} END{for(i=1;i<=NR-k;i++)print a[i]}'; }
> zfs list -H -t snapshot -o name -s createtxg -d 1 $SRC | prune | xargs -n1 zfs destroy
> $R zfs list -H -t snapshot -o name -s createtxg -d 1 $DST | prune | xargs -n1 $R zfs destroy
> EOF
root@prod:~ # sh -n /root/zbackup.sh && echo syntax ok
syntax ok
How it is built:
- The newest snapshot on the backup machine is the base of the increment, not "yesterday's" or "the last one I made here". So a failed night does not break anything: the next run just sends a bigger difference.
- Only snapshots named auto-... are pruned. Your own snapshots (@mon, @thu) are never touched.
- ssh -o ConnectTimeout=10 is not decoration. Our first version had none, and when we took the network down for the test the script hung for minutes.
The first version of the pruning line used head -n -4 (print all but the last four lines), which works on Linux. FreeBSD's head does not accept a negative number, and it said so on every run:
root@prod:~ # echo hi > /zroot/data/docs/run1.txt; /root/zbackup.sh; echo exit=$? head: illegal line count -- -4 head: illegal line count -- -4 exit=0 root@prod:~ # sleep 1; echo hi2 > /zroot/data/docs/run2.txt; /root/zbackup.sh; echo exit=$? head: illegal line count -- -4 head: illegal line count -- -4 exit=0 root@prod:~ # sleep 1; echo hi3 > /zroot/data/docs/run3.txt; /root/zbackup.sh; echo exit=$? head: illegal line count -- -4 head: illegal line count -- -4 exit=0 root@prod:~ # tail -n 3 /var/log/messages | cut -c1-120 Oct 6 18:32:18 prod zbackup[71485]: ok: sat -> auto-20261006-183217 Oct 6 18:32:20 prod zbackup[77045]: ok: auto-20261006-183217 -> auto-20261006-183219 Oct 6 18:32:23 prod zbackup[82623]: ok: auto-20261006-183219 -> auto-20261006-183222 root@prod:~ # zfs list -t snapshot -o name -r zroot/data | tail -6 zroot/data@thu zroot/data@fri zroot/data@sat zroot/data@auto-20261006-183217 zroot/data@auto-20261006-183219 zroot/data@auto-20261006-183222 root@backup:~ # zfs list -t snapshot -o name -r zroot/backups | tail -6 zroot/backups/prod-data@thu zroot/backups/prod-data@fri zroot/backups/prod-data@sat zroot/backups/prod-data@auto-20261006-183217 zroot/backups/prod-data@auto-20261006-183219 zroot/backups/prod-data@auto-20261006-183222
The sends worked (the log lines say ok:), but the old snapshots were never removed. Notice the exit=0 lines: the pipeline hid the error. That is why the final script uses awk for the pruning, and why you should look at what your script prints on the first runs, not just at its exit status. After the fix, run again several times, and the lists stay at four snapshots on each side:
root@prod:~ # sleep 1; echo hi4 > /zroot/data/docs/run4.txt; /root/zbackup.sh; echo exit=$? exit=0 root@prod:~ # sleep 1; echo hi5 > /zroot/data/docs/run5.txt; /root/zbackup.sh; echo exit=$? exit=0 root@prod:~ # sleep 1; echo hi6 > /zroot/data/docs/run6.txt; /root/zbackup.sh; echo exit=$? exit=0 root@prod:~ # zfs list -t snapshot -o name -r zroot/data | grep auto zroot/data@auto-20261006-183222 zroot/data@auto-20261006-183234 zroot/data@auto-20261006-183236 zroot/data@auto-20261006-183239 root@backup:~ # zfs list -t snapshot -o name -r zroot/backups | grep -c .; zfs list -t snapshot -o name -r zroot/backups | grep auto 11 zroot/backups/prod-data@auto-20261006-183222 zroot/backups/prod-data@auto-20261006-183234 zroot/backups/prod-data@auto-20261006-183236 zroot/backups/prod-data@auto-20261006-183239
What happens when the backup machine is not there
We took the network down on prod and ran the script:
root@prod:~ # ifconfig vtnet1 down; /root/zbackup.sh; echo exit=$?; tail -n 1 /var/log/messages | cut -c1-150 ssh: connect to host 192.168.99.2 port 22: Operation timed out exit=1 Oct 6 18:37:43 prod zbackup[81971]: FAILED: cannot read the snapshot list on the backup machine root@prod:~ # ifconfig vtnet1 up; sleep 3; zfs list -t snapshot -o name -r zroot/data | grep -c auto 4
It stopped after 10 seconds with exit status 1 and wrote FAILED to /var/log/messages, and made no new snapshot (the count is still 4). (An earlier version, without the "is the list empty?" check, ran zfs send -i @ with an empty name, and made a new local snapshot every time it failed.) Then the network comes back and the next run catches up:
root@prod:~ # /root/zbackup.sh; echo exit=$?; tail -n 1 /var/log/messages | cut -c1-150 exit=0 Oct 6 18:37:47 prod zbackup[87731]: ok: auto-20261006-183711 -> auto-20261006-183746 root@prod:~ # zfs list -t snapshot -o name -r zroot/data | grep auto zroot/data@auto-20261006-183253 zroot/data@auto-20261006-183647 zroot/data@auto-20261006-183711 zroot/data@auto-20261006-183746
The log says ok: auto-20261006-183711 -> auto-20261006-183746: from the newest snapshot the backup had, to the new one.
Put it in cron
root@prod:~ # /root/zbackup.sh; echo exit=$?; tail -n 1 /var/log/messages | cut -c1-150 exit=0 Oct 6 18:37:12 prod zbackup[68662]: ok: auto-20261006-183243 -> auto-20261006-183711 root@prod:~ # zfs list -t snapshot -o name -r zroot/data | grep auto zroot/data@auto-20261006-183243 zroot/data@auto-20261006-183253 zroot/data@auto-20261006-183647 zroot/data@auto-20261006-183711 root@backup:~ # zfs list -t snapshot -o name -r zroot/backups | grep auto zroot/backups/prod-data@auto-20261006-183236 zroot/backups/prod-data@auto-20261006-183239 zroot/backups/prod-data@auto-20261006-183243 zroot/backups/prod-data@auto-20261006-183711 root@prod:~ # crontab -l 2>&1 | tail -2 */5 * * * * /usr/local/bin/health-check.sh root@prod:~ # (crontab -l 2>/dev/null; echo "17 2 * * * /root/zbackup.sh") | crontab -; crontab -l */5 * * * * /usr/local/bin/health-check.sh 17 2 * * * /root/zbackup.sh
The last two commands show the crontab on prod: it already had the health check from the monitoring page, and we added 17 2 * * * /root/zbackup.sh: every night at 02:17. Add FAILED to the things you watch for in the log, and the monitoring page's checks tell you when the backup stopped:
grep -c 'zbackup.*FAILED' /var/log/messages
(Our log check is a one-liner idea, not run here. The point is: a backup that fails silently is the worst kind.)
Security: what the key can do
The backup key can log in to the backup machine as zbackup, and zbackup can create, receive, send and destroy under zroot/backups. It cannot touch anything else. But the key lives on prod, and anyone who takes over prod can use it: including to delete the backups, because we gave destroy for the pruning. Think about that. Two options, both from the documentation, not run here:
- In authorized_keys on the backup machine, restrict the key with command="..." and from="..." options, so it can only run one given command, from one address.
- Make the backup machine pull instead: it logs in to prod (with a key that can only zfs send) and receives. Then nobody on prod can reach the backups at all. This is a stronger design when the data is valuable.
The 3-2-1 rule
A common rule of thumb: 3 copies of the data, on 2 different kinds of storage, with 1 off-site. What we built is the second copy on another machine. A third copy somewhere else (another building, another provider) is the part that survives a fire or a theft. You can send the same snapshots again, to a second machine, with the same commands.
What we did not do
- sanoid/syncoid, zrepl and other tools that manage schedules and retention for you. They do what our script does, with more options. Install with pkg install, from the documentation, not run here.
- Encrypted datasets and raw sends (zfs send -w): the backup machine stores data it cannot read. Not run here.
- Bookmarks (zfs bookmark) for keeping the base of an increment without keeping the old snapshot. Not run here.
- Off-site or cloud copies, and restoring a whole system (the root pool, boot loader): this page only handled a data dataset.
- Testing the restore regularly. We did it once, here. Put it in your calendar.
The cheat sheet
- Snapshot: zfs snapshot POOL/DATA@NAME, list with zfs list -t snapshot, what changed with zfs diff @OLD @NEW.
- Size first: zfs send -nv DATA@NEW (full) or zfs send -nv -i @OLD DATA@NEW (incremental).
- Full copy: zfs send DATA@NAME | ssh USER@HOST zfs receive -u POOL/BACKUPS/NAME.
- Incremental: zfs send -i @OLD DATA@NEW | ssh USER@HOST zfs receive -u POOL/BACKUPS/NAME. The old snapshot must exist on both sides.
- Protect the copy: zfs set readonly=on POOL/BACKUPS/NAME.
- Backup user: pw useradd zbackup -m, zfs allow -u zbackup create,mount,receive,send,destroy POOL/BACKUPS, add the user to sshd AllowUsers.
- One file back: zfs mount COPY, then read /MOUNT/.zfs/snapshot/NAME/path.
- Everything back: ssh USER@HOST zfs send COPY@NAME | zfs receive DATA, then compare checksums (find . -type f | sort | xargs sha256 -r).
- Interrupted: receive with -s, zfs get receive_resume_token COPY, zfs send -t TOKEN | ssh ... zfs receive -s -u COPY.
- "destination has been modified": zfs rollback COPY@LASTSNAP (or receive -F), and keep readonly=on.
- Script: newest snapshot on the backup side as the base, logger the result, prune with awk (FreeBSD head has no negative count), ssh -o ConnectTimeout=10, cron line 17 2 * * * /root/zbackup.sh.
← Monitoring FreeBSD: From top to Prometheus · ↑ FreeBSD Corner · WireGuard on FreeBSD: A Private Tunnel Between Two Machines →