🔏 NetBSD veriexec

veriexecgen -d /opt/tools -o /etc/signatures   # fingerprints
veriexecctl load /etc/signatures                # into the kernel
sysctl -w kern.veriexec.strict=1                # enforce

Veriexec is the file integrity system of the NetBSD kernel. It keeps a hash for every file on a list, and at strict level 1 it refuses to run or read a file that no longer matches. Even root cannot turn it off again without a reboot. This page makes a list, loads it, and shows what happens to a file that was changed.

Every example ran for real on NetBSD 11.1 (amd64) in a virtual machine, with root commands marked #. The kernel is the stock GENERIC kernel, as veriexec is built in.

Veriexec is a part of the NetBSD kernel that remembers a fingerprint, a SHA hash, for each file on a list. When a listed file is run or opened, the kernel computes the hash again and compares. If the file was changed, the kernel can refuse to let anyone use it, even root. This is how you notice, and stop, a program that was replaced by a trojan version or a configuration file that was changed behind your back.

Every example here ran for real on NetBSD 11.1 (amd64) in a virtual machine. Commands that change the system start with # and run as root, because NetBSD has no sudo by default. The default kernel already has veriexec built in, which you can see from sysctl, and the tools are in the base system:

$ sysctl kern.veriexec
kern.veriexec.verbose = 0
kern.veriexec.strict = 0
kern.veriexec.algorithms = SHA256 SHA384 SHA512
$ ls -l /sbin/veriexecctl /usr/sbin/veriexecgen /etc/rc.d/veriexec
-r-xr-xr-x  1 root  wheel    632 Nov  2 16:34 /etc/rc.d/veriexec
-r-xr-xr-x  1 root  wheel  45096 Nov  2 16:34 /sbin/veriexecctl
-r-xr-xr-x  1 root  wheel  25312 Nov  2 16:34 /usr/sbin/veriexecgen
$ sysctl kern.securelevel
kern.securelevel = -1

The strict level is 0, which is learning mode: the kernel knows the feature, but nothing is enforced yet. There are four levels, and a table of them comes later on this page. Two tools do the work. veriexecgen makes the list of fingerprints, and veriexecctl loads it into the kernel and asks questions about it.

Start small, with two scripts of your own, in a directory owned by the bsd user so that the examples do not need root for the files themselves:

# mkdir -p /opt/tools && chown bsd /opt/tools

$ cat /opt/tools/backup.sh
#!/bin/sh
echo backup ok

$ cat /opt/tools/report.sh
#!/bin/sh
echo report ok

$ chmod 755 /opt/tools/backup.sh /opt/tools/report.sh

Now make the list. -d names the directory to scan and -o the file to write. The default is /etc/signatures, but a file in /tmp is enough for a test:

$ veriexecgen -d /opt/tools -o /tmp/tools.sig
$ cat /tmp/tools.sig
/opt/tools/backup.sh SHA256 86e4bb9452fc926d8ea538b881b2b60e3c61ea11e835672e72269670deb9945d
/opt/tools/report.sh SHA256 388b01c59630bd54816bb79e039f471cf2cad237adb9e05ea242fcca68d19092

Each line is a path, the algorithm, and the hash. Load the list into the kernel, with -k so that the kernel keeps the file names for its log messages, and ask about one file:

# veriexecctl -k load /tmp/tools.sig
# veriexecctl query /opt/tools/backup.sh
Filename: /opt/tools/backup.sh
Mount: /
Entry flags: direct
Entry status: not evaluated
Fingerprint algorithm: SHA256
Fingerprint: 86e4bb9452fc926d8ea538b881b2b60e3c61ea11e835672e72269670deb9945d
$ sysctl kern.veriexec.count
kern.veriexec.count.table0.mntpt = /
kern.veriexec.count.table0.fstype = ffs
kern.veriexec.count.table0.nentries = 2

The status is "not evaluated" because the kernel computes the hash the first time the file is used, not when the list is loaded. The -e flag of load would do it at once. The count shows two entries on the root file system. Run one of the scripts, and it works, as before:

$ /opt/tools/backup.sh
backup ok

In learning mode nothing is stopped, so a changed file still runs. Change one of the scripts, as an attacker would, and see:

$ echo '# changed' >> /opt/tools/backup.sh
$ /opt/tools/backup.sh; echo "exit status $?"
backup ok
exit status 0

Now raise the strict level to 1, the IDS mode, which makes the kernel enforce the list:

# sysctl -w kern.veriexec.strict=1
kern.veriexec.strict: 0 -> 1

The same two scripts again. The one that was changed does not run, and the one that was not changed does:

$ /opt/tools/backup.sh; echo "exit status $?"
sh: /opt/tools/backup.sh: error 1
exit status 126
$ /opt/tools/report.sh; echo "exit status $?"
report ok
exit status 0
$ cat /opt/tools/backup.sh; echo "exit status $?"
cat: /opt/tools/backup.sh: Operation not permitted
exit status 1

Not even cat can read the changed file, as the kernel treats the whole file as untrustworthy. The kernel also writes down what it saw. The messages are in the kernel log:

# dmesg | tail -n 6
[    19.929389] Veriexec: Incorrect access type. [/opt/tools/backup.sh, prog=sh pid=1279, uid=1000, gid=100]
[    19.929389] Veriexec: Mismatch. [/opt/tools/backup.sh]
[    20.839173] Veriexec: Mismatch. [/opt/tools/backup.sh]
[    21.119119] Veriexec: Incorrect access type. [/opt/tools/report.sh, prog=sh pid=1356, uid=1000, gid=100]
[    21.429126] Veriexec: Incorrect access type. [/opt/tools/backup.sh, prog=cat pid=1239, uid=1000, gid=100]
[    21.429126] Veriexec: Mismatch. [/opt/tools/backup.sh]

The level also protects the list itself, as listed files cannot be removed any more, and the level cannot be lowered, not even by root:

$ rm /opt/tools/report.sh; echo "exit status $?"
rm: /opt/tools/report.sh: Operation not permitted
exit status 1
# sysctl -w kern.veriexec.strict=0; echo "exit status $?"
sysctl: kern.veriexec.strict: Operation not permitted
exit status 1

That one-way door is the point. An intruder who gets root cannot turn the checks off again without a reboot, and a reboot is something you notice. In IDS mode, writing to a listed file is still allowed, only the use of a file that does not match is refused.

Fingerprints for the scripts of one directory are a start, but the real use is the system itself. With -D veriexecgen scans the usual system directories: /bin, /sbin, /usr/bin, /usr/sbin, /lib, /usr/lib, /libexec and /usr/libexec. It reads the files, which is allowed in IDS mode:

$ veriexecgen -D -o /tmp/system.sig
$ wc -l /tmp/system.sig
    1080 /tmp/system.sig
$ head -n 3 /tmp/system.sig
/bin/[ SHA256 5a3f3997f57d6d0c3c2384efe7bf1106a3b3391b60fbfdf69464a9a4937e2982
/bin/cat SHA256 77f8344a2eacaa9ed3775405af5bb9d58c9ec29c9a87100d1c44cb25fd882386
/sbin/chown SHA256 72f41a9e8a15949148d0d25f7ed27a95c90bc2c021a72f86bdc6ba73d264f0c0

About a thousand files are listed. To have this loaded at every boot, the list goes to /etc/signatures, which is where veriexecgen writes by default, and two lines go in /etc/rc.conf:

$ # veriexecgen
$ # echo 'veriexec=YES' >> /etc/rc.conf
$ # echo 'veriexec_strict=1' >> /etc/rc.conf

The rc script /etc/rc.d/veriexec loads the file first and raises the strict level after that, because with a raised level the loading would not be allowed any more. The page does not run these lines, as they would change the next boot of the machine. In the virtual machine they were run by hand, with a reboot, and the machine came up with the strict level 1 and 984 entries loaded. A copy of a program from /tmp that was not listed ran as before, and a changed system file, such as /bin/ls, stopped working at once.

These are the four levels, from the manual page of veriexec(8):

Level 0, learning: changes to the list are allowed, nothing is enforced.
Level 1, IDS: a file with a wrong fingerprint is refused, and listed files cannot be removed.
Level 2, IPS: also no writing to listed files, and only listed programs can be executed.
Level 3, lockdown: also files that are not listed cannot be accessed, and no new files can be made.

Level 2 is the practical top for a server, but it needs a list that covers everything the machine runs, also what is in /usr/pkg. Try it in a virtual machine first. Level 3 is for machines that never change.

The protection gets in the way of your own updates, as it should. After an update, or after a change of a listed script, the fingerprint is wrong and the file is refused. The way out is the same as the way in, and it needs a reboot, because the list can only be changed in learning mode:

$ # sed -i 's/veriexec_strict=1/veriexec_strict=0/' /etc/rc.conf
$ # reboot

After the reboot the machine runs with the strict level 0. Make the change, make the list again, and set the level back:

$ # veriexecgen
$ # sed -i 's/veriexec_strict=0/veriexec_strict=1/' /etc/rc.conf
$ # reboot

That is a lot of work for a configuration file that changes every week, which is why you list programs and libraries, and not the files in /etc. For a server that is updated with sets, as in the NetBSD update guide, the order is: update, make the list again at level 0, set the level to 1, reboot. veriexecgen keeps the old list as /etc/signatures.old.

One more thing about editors. A program like sed -i, or an editor that saves by writing a new file and renaming it, creates a new file, and a new file is not on the list. The kernel does not know it, and in IDS mode it just runs. The list tells the kernel about files, not about names, so run veriexecgen again after changing a listed file in that way.

The messages of veriexec are short. These are the ones you will meet. The first is what a program says when it runs a script that does not match, and the second is what the kernel log has to say about it:

$ /opt/tools/backup.sh; echo "exit status $?"
sh: /opt/tools/backup.sh: error 1
exit status 126
# dmesg | grep -i mismatch | tail -n 2
[    21.429126] Veriexec: Mismatch. [/opt/tools/backup.sh]
[    27.028008] Veriexec: Mismatch. [/opt/tools/backup.sh]

Exit status 126 means that the file was found but could not be run. Loading a list needs level 0, so at a higher level it fails, whatever the list holds:

# veriexecctl load /tmp/tools.sig; echo "exit status $?"
veriexecctl: Cannot load '/tmp/tools.sig'
exit status 1

And to see which files are marked as wrong, ask the kernel about a file. The status line says mismatch for a file that was changed, and the fingerprint line shows the hash that the kernel found:

# veriexecctl query /opt/tools/backup.sh
Filename: /opt/tools/backup.sh
Mount: /
Entry flags: direct
Entry status: mismatch
Fingerprint algorithm: SHA256
Fingerprint: 86e4bb9452fc926d8ea538b881b2b60e3c61ea11e835672e72269670deb9945d

If you locked yourself out, for example with a list that left out an important program at level 2, the way back is to boot the machine single user, or with veriexec_strict=0 in /etc/rc.conf, as in the last section.