⬆️ Updating NetBSD with sets

tar -C / -xpf kern-GENERIC.tar.xz   # the new kernel
tar -C / -xpf base.tar.xz           # the base system
etcupdate -s etc.tar.xz             # merge /etc, never unpack it

NetBSD is updated by unpacking sets, which are tar archives, over the running system, and merging the changes to /etc. This page downloads the sets and checks them against the published checksums, replaces the kernel and keeps the old one, unpacks the base, merges /etc with etcupdate, checks the result with postinstall, and explains the rollback and the mistakes to avoid.

Every example ran for real on NetBSD 11.1 (amd64) in a virtual machine, with root commands marked #. The machine already runs 11.1, so the 11.1 sets are unpacked again over it, with exactly the commands of a real update. The output you see is the output the machine gave.

NetBSD is not updated package by package. A release is a set of tar archives called sets, and updating a machine means unpacking newer sets over the old system, in the right order, and then merging the changes to the files in /etc. The installer can do all of that from install media, and this page does it by hand, so that you see each step and can do it over ssh.

Every example here ran for real on NetBSD 11.1 (amd64) in a virtual machine. Commands that change the system start with # and run as root, because NetBSD has no sudo by default. One thing to know before you read on: this machine already runs the newest release, 11.1. So the example unpacks the 11.1 sets again over the 11.1 system. That changes nothing you can see, but it uses exactly the commands you would use to go from one release to the next, and only the address of the sets differs. First, where are we?

$ uname -srm
NetBSD 11.1 amd64
$ head -n 1 /etc/release
NetBSD 11.1/amd64
$ ls -l /netbsd
-rwxr-xr-x  1 root  wheel  30098896 Nov  2 16:34 /netbsd
$ grep -v '^#' /usr/pkg/etc/pkgin/repositories.conf | grep .
https://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/$arch/11.0/All

The sets of every release are on the NetBSD download servers, in a directory named after the release and the machine type. A file called SHA512 in the same directory lists the checksum of every set. Download the sets that an update needs: the kernel, the kernel modules, the base system and the etc files. The ftp command in the base system fetches one address at a time, so a small loop gets all five:

$ mkdir -p sets
$ U=http://cdn.netbsd.org/pub/NetBSD/NetBSD-11.1/amd64/binary/sets; cd sets && for f in SHA512 kern-GENERIC.tar.xz modules.tar.xz base.tar.xz etc.tar.xz; do ftp -4 -V -o $f $U/$f > /dev/null 2>&1; done; ls -l
total 125164
-rw-r--r--  1 bsd  users      4025 Nov  3 00:21 SHA512
-rw-r--r--  1 bsd  users  44625368 Nov  2 23:22 base.tar.xz
-rw-r--r--  1 bsd  users    510760 Nov  2 23:21 etc.tar.xz
-rw-r--r--  1 bsd  users   8592340 Nov  2 23:19 kern-GENERIC.tar.xz
-rw-r--r--  1 bsd  users       618 Nov  6 21:06 mine.sha
-rw-r--r--  1 bsd  users  10219008 Nov  2 23:22 modules.tar.xz

The -4 is there because the virtual machine has no working IPv6. Now check the sets against the list. Keep only the lines for the files you have, and let cksum compare. It prints nothing when all is well:

$ cd sets && grep -E '\((kern-GENERIC|modules|base|etc)\.tar\.xz\)' SHA512 > mine.sha
$ cd sets && cksum -a sha512 -c mine.sha; echo "exit status $?"
exit status 0

What are the other sets? kern-GENERIC is the kernel. modules holds the kernel modules, which must match the kernel. base is the whole base system, and etc the configuration files. comp is the compiler and the headers, text the old text tools, man the manual pages, misc some extras, rescue the static recovery tools, games, tests, debug the debug symbols, and the x sets are the X Window System. Look inside before you unpack, as you would with any archive:

$ cd sets && tar -tf kern-GENERIC.tar.xz
./netbsd
$ cd sets && tar -tf etc.tar.xz | head -n 5
./.cshrc
./.profile
./boot.cfg
./dev/MAKEDEV
./dev/MAKEDEV.local

The order matters. Replace the kernel first, and keep the old one under another name, so that there is something to boot if the new one does not start. The sets are unpacked from the root directory with tar. The -p option keeps the permissions and the owners:

# cp /netbsd /onetbsd
# tar -C / -xpf /home/bsd/sets/kern-GENERIC.tar.xz
# tar -C / -xpf /home/bsd/sets/modules.tar.xz
$ ls -l /netbsd /onetbsd
-rwxr-xr-x  1 root  wheel  30098896 Nov  2 16:34 /netbsd
-rwxr-xr-x  1 root  wheel  30098896 Nov  6 21:07 /onetbsd

Both files are the same size here, as they are the same release. When you go to a new release, the size and the date of /netbsd change, and /onetbsd stays the old one. The running kernel does not change until the next boot.

Now the base set, which holds the programs and libraries. This is the longest step, a few seconds here:

# tar -C / -xpf /home/bsd/sets/base.tar.xz
$ ls -ld /bin/ls /lib/libc.so.*
-r-xr-xr-x  1 root  wheel    41296 Nov  2 16:34 /bin/ls
lrwxr-xr-x  1 root  wheel       14 Nov  2 16:34 /lib/libc.so.12 -> libc.so.12.224
-r--r--r--  1 root  wheel  2589600 Nov  2 16:34 /lib/libc.so.12.224

It is safe to do while the system runs, because tar replaces each file with a new one and the programs that are running keep the old copy open. Do not unpack the etc set the same way, though. That would overwrite your configuration files, with your passwords, your users, and everything you have changed. The etc files are merged with a special tool, in the next section.

etcupdate compares the new etc set with the files that are on the system, and asks what to do with each one that differs. It only needs the set itself, there is no need to unpack it. It asks questions on a terminal, so this page answers every question with d, which means keep my file, and saves what it says in a log:

# yes d | TERM=vt100 etcupdate -s /home/bsd/sets/etc.tar.xz > /tmp/eu.log 2>&1
# grep '^File:' /tmp/eu.log
File: /etc/ssh/sshd_config (modified)
File: /etc/group (modified)
File: /etc/master.passwd (modified)
File: /etc/motd (modified)
File: /etc/rc.conf (modified)
File: /etc/sysctl.conf (modified)
File: /etc/ttys (modified)

On this machine a few files differ from the standard ones, because the machine was prepared for these pages with a user, some ssh settings and a few lines in rc.conf. On a fresh install of the previous release you would normally see only the files that the release changed. For each file the choices are i, install the new file and lose your changes, d, keep yours, m, merge the two with a text editor, and s, show the differences first. Always look at the differences with s for files such as rc.conf and master.passwd. On a real update, do this at a terminal and not with a canned answer.

The -a option makes etcupdate remember a checksum of each file it installs in /var/etcupdate. On the next update it then replaces files that you never changed without asking, and only asks about the files you edited.

The last step is a check of the whole system against the etc set. postinstall looks for missing users and groups, wrong permissions, missing device nodes, and files that a release removed. It prints a long list of what it checked, and the end is what matters:

# postinstall -s /home/bsd/sets/etc.tar.xz check 2>&1 | tail -n 3
opensslcertsrehash check:
postinstall checks passed: atf autofsconfig blocklist bluetooth catpages ddbonpanic defaults dhcpcd dhcpcdrundir envsys fontconfig gid gpio hosts iscsi makedev manconf motd mtree named opensslcertsconf pam periodic pf ptyfsoldnodes pwd_mkdb rc ssh tcpdumpchroot uid varrwho varshm wscons x11 xkb obsolete opensslcertsrehash
postinstall checks failed:

Nothing failed here. After a real update, a failed check comes with a way to repair it. Fix one named item at a time, so you can see what it changes, for example postinstall fix obsolete removes the files that the new release no longer has. The obsolete check is the one that matters most, because tar only adds and replaces files, and never deletes the old ones:

# postinstall -s /home/bsd/sets/etc.tar.xz check obsolete 2>&1 | tail -n 2
postinstall checks passed: obsolete
postinstall checks failed:

Then restart the machine:

$ shutdown -r now

This page does not run the reboot, because it would end the ssh session that runs the examples. The virtual machine was restarted after these steps, and it came up on the new kernel.

If the new kernel does not boot, the old one is still there. At the boot menu, press a key to get the prompt, and type the name of the old kernel:

$ boot onetbsd

Because the base files were replaced as well, the old kernel with the new base is only for a short time, to get in and put things right. Keep the sets, so you can unpack the kernel set again.

The packages are separate. After a release update the packages that came for the old release usually still run, but to be safe, point pkgin at the repository of the new release. The release is a part of its address (on this machine the line you saw in the first section names 11.0), so after a move to a new release the repository line needs a change. Then update:

$ pkgin update
$ pkgin upgrade

Pkgsrc has its own pages, and this one is only about the base system.

A checksum that does not match means a damaged download. Here a few bytes were added to the etc set, a copy of it is checked, and the damaged copy is removed again:

$ cd sets && cksum -a sha512 -c bad.sha; echo "exit status $?"
(SHA512) bad.tar.xz: FAILED
exit status 1

Download it again, and check again, before you unpack anything. A set that stopped in the middle gives an error from tar, which is why the checksum comes first:

$ cd sets && tar -tf cut.tar.xz > /dev/null; echo "exit status $?"
tar: Truncated input file (needed 302080 bytes, only 0 available)
tar: Error exit delayed from previous errors.
exit status 1

An address with a release that does not exist is a 404:

$ ftp -4 -V -o /dev/null http://cdn.netbsd.org/pub/NetBSD/NetBSD-99.9/amd64/binary/sets/etc.tar.xz; echo "exit status $?"
ftp: Error retrieving file `404 Not Found'
exit status 1

And the mistake behind most broken updates is unpacking the etc set over the system with tar. If you did, the users and the passwords in /etc are the ones of the set, and you can no longer log in. Make a copy of /etc before every update, for a moment like that:

$ cp -Rp /etc /etc.backup