📈 Monitoring FreeBSD: From top to Prometheus

Four layers of monitoring, each tested: the live tools, the nightly report, your own checks run by cron, and metrics with alerts. Including what happens when something breaks

Video: Monitoring FreeBSD (FreeBSD video)
Prefer to watch? This short video runs this page's main commands on a real FreeBSD 15.0 machine, with on-screen captions and no voiceover. Click to play, or watch it on YouTube.

Three questions

Monitoring comes down to three questions, asked at different speeds:

  1. What is happening right now? The live tools. You sit and look.
  2. Is it healthy, and was it healthy while I was away? Reports and logs. The machine tells you, on a schedule.
  3. Will someone be told when it is not? Checks and alerts. This is the part people skip, and the part that matters at 3 a.m.

This page walks up those layers on a real FreeBSD 15.0 virtual machine, with something deliberately broken at each step, so that you see what the monitoring says when it has something to say. The layers: the base system's own tools, the nightly periodic report, your own checks with cron and syslog, and finally metrics with alert rules (node_exporter and Prometheus), plus a short look at SNMP. The firewall and hardening pages decide who may reach these services, so remember them as you go.

Layer 1: what is happening now

All of these are in the base system. They have interactive modes, and we show the one-shot versions, which are the ones you can paste into a report.

root@practice:~ # uptime
 5:55PM  up 1 min, 1 user, load averages: 0.36, 0.11, 0.04
root@practice:~ # top -b -d 1 | head -14
last pid: 63937;  load averages:    0.36,    0.11,    0.04  up 0+00:01:01    17:55:01
38 processes:  1 running, 37 sleeping
CPU:  3.7% user,  0.0% nice, 14.2% system,  0.6% interrupt, 81.6% idle
Mem: 220M Active, 68M Inact, 264M Wired, 1408M Free
ARC: 136M Total, 17M MFU, 115M MRU, 696K Anon, 1400K Header, 2330K Other
     100M Compressed, 202M Uncompressed, 2.02:1 Ratio
Swap: 2048M Total, 2048M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
18178 root         11   0    0   814M    59M kqread   0   0:14   2.39% bhyve
86562 root          1   0    0    25M    10M select   0   0:00   0.05% sshd
60637 root          1   9    0    25M    12M select   0   0:00   0.00% sshd-session
13349 root          1  59    0    14M  4012K wait     0   0:00   0.00% sh
78634 root          1   0    0    15M  4452K select   1   0:00   0.00% devd
root@practice:~ # vmstat 1 3
 procs    memory    page                      disks       faults       cpu
 r  b  w  avm  fre  flt  re  pi  po   fr   sr vtb0  cd0   in   sy   cs us sy id
 0  0  0 4.0T 1.4G 6.0k   0  28   0 6.5k   70    0    0  100  24k 2.9k  3 14 81
 0  0  0 4.0T 1.4G    0   0   0   0    0   90    0    0  135  503  519  0  2 97
 0  0  0 4.0T 1.4G    0   0   0   0    0   81   16    0  112 1.0k  772  0  7 91
root@practice:~ # swapinfo -h
Device              Size     Used    Avail Capacity
/dev/vtbd0p2        2.0G       0B     2.0G     0%
root@practice:~ # iostat -x -c 2 -w 1 vtbd0
                        extended device statistics  
device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b  
vtbd0         80      13   1729.8    452.3     0     2     1     0    0   2 
                        extended device statistics  
device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b  
vtbd0          2      53     11.8   1571.8     0     1     1     1    0   3 
root@practice:~ # zpool iostat zroot 1 3
              capacity     operations     bandwidth 
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
zroot       7.48G  6.02G     76     14  1.60M   479K
zroot       7.48G  6.02G      0      0      0      0
zroot       7.48G  6.02G      0      0      0      0
root@practice:~ # netstat -i -n | head -5
Name         Mtu Network        Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
vtnet0      1500 <Link#1>       52:54:00:12:34:56     2048     0     0     1351     0     0
vtnet0         - 10.0.2.0/24    10.0.2.15             2044     -     -     1346     -     -
vtnet0         - 10.0.2.50/32   10.0.2.50                0     -     -        0     -     -
lo0        16384 <Link#2>       lo0                      0     0     0        0     0     0
root@practice:~ # netstat -w 1 -q 3 -I vtnet0
            input         vtnet0           output
   packets  errs idrops      bytes    packets  errs      bytes colls
       103     0     0       7398         52     0       4800     0
        97     0     0       6966         49     0       4482     0
         1     0     0         54          1     0        162     0

Two more live tools are in the base system and are interactive, so we did not run them for the page: gstat (a live view of every disk with its busy percentage) and systat (a full-screen set of views: -vmstat, -ifstat, -iostat and others). From the documentation, not run here.

Quick health: three commands

root@practice:~ # zpool status -x
all pools are healthy
root@practice:~ # zfs list -o name,used,avail,compressratio zroot | head -3
NAME    USED  AVAIL  RATIO
zroot  7.48G  5.60G  1.44x
root@practice:~ # df -h / | tail -2
Filesystem            Size    Used   Avail Capacity  Mounted on
zroot/ROOT/default    7.0G    1.4G    5.6G    21%    /
root@practice:~ # w
 5:55PM  up 1 min, 1 user, load averages: 0.33, 0.11, 0.04
USER       TTY      FROM       LOGIN@  IDLE WHAT
bsd        pts/0    10.0.2.2   5:55PM     - w
root@practice:~ # last | head -3
bsd        pts/0    10.0.2.2               Tue Oct  6 17:55   still logged in
bsd        pts/0    10.0.2.2               Tue Oct  6 17:54 - 17:54  (00:00)
bsd        pts/0    10.0.2.2               Tue Oct  6 17:54 - 17:54  (00:00)

Layer 2: the nightly report

FreeBSD already runs a set of checks every night. The program is periodic, and it is started from cron:

root@practice:~ # grep -v "^#" /etc/crontab | grep .
SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin
*/11	*	*	*	*	operator /usr/libexec/save-entropy
0	*	*	*	*	root	newsyslog
1	3	*	*	*	root	periodic daily
15	4	*	*	6	root	periodic weekly
30	5	1	*	*	root	periodic monthly
1,31	0-5	*	*	*	root	adjkerntz -a
root@practice:~ # grep -v "^#" /etc/newsyslog.conf | grep . | head -5
/var/log/all.log			600  7	   *	@T00  J
/var/log/auth.log			600  7	   1000 @0101T JC
/var/log/console.log			600  5	   1000	*     J
/var/log/cron				600  3	   1000	*     JC
/var/log/daemon.log			644  5	   1000	@0101T JC

The system crontab runs periodic daily at 03:01, periodic weekly on Saturdays at 04:15 and periodic monthly on the 1st at 05:30. newsyslog runs every hour to rotate logs: its configuration lists each log file, how many old copies to keep and when to rotate (for example /var/log/auth.log keeps 7 copies and rotates when it reaches 1000 KB, or on New Year's Day).

The checks are small scripts in /etc/periodic/daily (and weekly, monthly, security). What runs is switched by variables in /etc/defaults/periodic.conf:

root@practice:~ # grep -E "^[a-z_]*status_zfs_enable|^daily_status_(disks|network|security|uptime|ntpd)_enable|^daily_output|^security_status_pkgaudit|^daily_scrub_zfs|^daily_status_security_pkgaudit" /etc/defaults/periodic.conf
daily_output="root"					# user or /file
daily_status_disks_enable="YES"				# Check disk status
daily_status_zfs_enable="NO"				# Check ZFS
daily_status_network_enable="YES"			# Check network status
daily_status_uptime_enable="YES"			# Check system uptime
daily_status_security_enable="YES"			# Security check
daily_status_ntpd_enable="NO"				# Check NTP status
daily_scrub_zfs_enable="NO"
daily_scrub_zfs_pools=""			# empty string selects all pools
daily_scrub_zfs_default_threshold="35"		# days between scrubs
root@practice:~ # ls /usr/local/etc/periodic/daily /usr/local/etc/periodic/security
/usr/local/etc/periodic/daily:
411.pkg-backup		490.status-pkg-changes

/usr/local/etc/periodic/security:
405.pkg-base-audit	410.pkg-audit		460.pkg-checksum

Look at the defaults:

We ran periodic daily to see what it does. It printed nothing. Where did the report go?

root@practice:~ # ls -l /var/mail/
total 9
-rw-------  1 bsd  bsd      0 Oct  5 20:58 bsd
-rw--w----  1 root mail 18807 Oct  6 17:55 root
root@practice:~ # mail -H 2>&1 | head -5
Mail version 8.1 6/6/93.  Type ? for help.
"/var/mail/root": 2 messages 2 new
>N  1 root@practice         Tue Oct  6 17:55 386/13564 "practice daily securi"
 N  2 root@practice         Tue Oct  6 17:55 106/5243  "practice daily run ou"

Into root's mailbox (/var/mail/root, 18 KB): two messages, "daily security run output" and "daily run output". Nobody reads root's mail unless you set that up. So there are two good options: forward root's mail to a real address (needs a working mail setup: from the documentation, not run here), or write the report to a file, which is what we did next:

root@practice:~ # cat > /etc/periodic.conf <<'EOF'
> daily_output="/var/log/daily.log"
> daily_status_zfs_enable="YES"
> EOF
root@practice:~ # cat /etc/periodic.conf
daily_output="/var/log/daily.log"
daily_status_zfs_enable="YES"
root@practice:~ # periodic daily
root@practice:~ # ls -l /var/log/daily.log
you have mail
-rw-r--r--  1 root wheel 5022 Oct  6 17:56 /var/log/daily.log
root@practice:~ # grep -c "" /var/log/daily.log
97
root@practice:~ # sed -n "1,70p" /var/log/daily.log (excerpt)

Removing stale files from /var/preserve:

Cleaning out old system announcements:

Removing stale files from /var/rwho:

Backup passwd and group files:

Verifying group file syntax:
/etc/group is fine

Backing up mail aliases:

Dump of kern.geom.conftxt:

Backup of partitions information for:
vtbd0

Backup of boot partition content:
vtbd0p1

Disk status:
Filesystem                       Size    Used   Avail Capacity  Mounted on
zroot/ROOT/default               7.0G    1.4G    5.6G    21%    /
devfs                            1.0K      0B    1.0K     0%    /dev
fdescfs                          1.0K      0B    1.0K     0%    /dev/fd
zroot/var/audit                  5.6G    120K    5.6G     0%    /var/audit
zroot/home                       5.6G    132K    5.6G     0%    /home
[... the rest of the list of file systems is left out ...]

Checking status of zfs pools:
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zroot  13.5G  7.48G  6.02G        -         -    13%    55%  1.00x    ONLINE  -

all pools are healthy

Network interface status:
Name         Mtu Network        Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll  Drop
vtnet0      1500 <Link#1>       52:54:00:12:34:56     3017     0     0     1934     0     0     0

The security half of the nightly run is a separate message. Here is the one that went to root's mailbox:

root@practice:~ # awk '/daily securi/,0' /var/mail/root | sed -n '1,45p'
Subject: practice daily security run output
Date: Tue, 06 Oct 2026 17:55:44 +0000
From: <root@practice>

Checking setuid files and devices:

No /var/log/setuid.today

Checking negative group permissions:

No /var/log/mount.today

Checking for uids of 0:
root 0
toor 0

Checking for passwordless accounts:

Checking login.conf permissions:

No /var/log/ipfw.today

No /var/log/dmesg.today

practice login failures:

practice refused connections:

Checking for security vulnerabilities in base (userland & kernel):
Database fetched: 2026-10-06T09:15+00:00
FreeBSD-kernel-15.0 is vulnerable:
  FreeBSD -- IPv6 UDP sendto(2) bypasses jail loopback restriction
  CVE: CVE-2026-101303
  WWW: https://vuxml.FreeBSD.org/freebsd/6d92b2a3-bcce-11f1-906f-bc241121aa0a.html

  FreeBSD -- Remote DoS via receive-side kernel TLS
  CVE: CVE-2026-101302
[... more findings follow ...]

It checks for set-user-ID files and devices that changed, uids of 0 (root and toor, as expected: anything else here is an alarm), accounts without a password, login failures and refused connections, and, from the package system, a vulnerability check of the base system itself. This lab machine runs a plain, unpatched 15.0-RELEASE, so the report listed known vulnerabilities in FreeBSD-kernel-15.0, with their CVE numbers and links. That is the report doing its job: it is the nudge to run freebsd-update (see the upgrade page) and pkg audit (see the hardening page).

Logs

Services write their messages through syslog, and a rule file, /etc/syslog.conf, decides which messages go where. One line matters for monitoring:

root@practice:~ # grep -n "notice" /etc/syslog.conf | head -3
7:*.err;kern.warning;auth.notice;mail.crit		/dev/console
8:*.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err	/var/log/messages
28:# news.notice					/var/log/news/news.notice

Line 8 sends everything of level notice and above to /var/log/messages. Messages at a lower level (info, debug) are dropped there. We learned this by testing the script in the next section: our first version logged its "all is well" line at the info level, and it never appeared in /var/log/messages, while the warnings did. Log the "ok" lines at notice, or they vanish.

To send a line to the log yourself, use logger: logger -p daemon.warning -t healthcheck "text" writes a line at that level, tagged with a name. Anything you can run in a shell can report this way. (To collect logs from other machines on one log server, syslogd can listen on the network. We switched that off on the hardening page, with syslogd_flags="-ss", so a log server would turn it back on and restrict who may send to it: from the documentation, not run here.)

Layer 3: your own checks

The nightly report is for humans. For things that must be noticed fast, write a small script, and let cron run it every few minutes. This is ours: it checks the root file system, the ZFS pools and two services, and logs one line per problem (or one "ok" line). Its exit status is the number of problems, so other tools can use it.

root@practice:~ # cat > /usr/local/bin/health-check.sh <<'EOF'
> #!/bin/sh
> # Log one line per problem, or a single "ok" line. Exit status = number of problems.
> MAX_DISK=${MAX_DISK:-80}
> problems=0
> use=$(df -P / | awk 'NR == 2 { sub("%", "", $5); print $5 }')
> if [ "$use" -ge "$MAX_DISK" ]; then
>     logger -p daemon.warning -t healthcheck "root filesystem ${use}% full (limit ${MAX_DISK}%)"
>     problems=$((problems + 1))
> fi
> pools=$(zpool status -x)
> if [ "$pools" != "all pools are healthy" ]; then
>     logger -p daemon.warning -t healthcheck "ZFS problem: $(echo "$pools" | head -1)"
>     problems=$((problems + 1))
> fi
> for svc in sshd cron; do
>     if ! service $svc status > /dev/null 2>&1; then
>         logger -p daemon.warning -t healthcheck "service $svc is not running"
>         problems=$((problems + 1))
>     fi
> done
> [ $problems -eq 0 ] && logger -p daemon.notice -t healthcheck "ok: disk ${use}%, pools healthy, services running"
> exit $problems
> EOF
root@practice:~ # chmod 755 /usr/local/bin/health-check.sh
root@practice:~ # sh -n /usr/local/bin/health-check.sh && echo syntax-ok
syntax-ok

First, healthy, and then with a deliberately silly limit (MAX_DISK=10, so that 21 percent counts as "too full"):

root@practice:~ # /usr/local/bin/health-check.sh; echo "exit status: $?"
exit status: 0
root@practice:~ # tail -1 /var/log/messages
Oct  6 17:57:08 practice healthcheck[10552]: ok: disk 21%, pools healthy, services running
root@practice:~ # MAX_DISK=10 /usr/local/bin/health-check.sh; echo "exit status: $?"
exit status: 1
root@practice:~ # tail -2 /var/log/messages
Oct  6 17:57:08 practice healthcheck[10552]: ok: disk 21%, pools healthy, services running
Oct  6 17:57:08 practice healthcheck[13849]: root filesystem 21% full (limit 10%)

Healthy: exit status 0 and one line, ok: disk 21%, pools healthy, services running. With the limit lowered: exit status 1 and the warning, root filesystem 21% full (limit 10%). The script works. But a test that cannot fail proves nothing, so now a real fault. We built a small ZFS mirror out of two files, took one of the "disks" offline, and ran the script:

root@practice:~ # mkdir -p /var/tmp/probe && truncate -s 64m /var/tmp/probe/d1 /var/tmp/probe/d2
root@practice:~ # zpool create probe mirror /var/tmp/probe/d1 /var/tmp/probe/d2
root@practice:~ # zpool status -x
all pools are healthy
root@practice:~ # zpool offline probe /var/tmp/probe/d2
root@practice:~ # zpool status -x
  pool: probe
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
	Sufficient replicas exist for the pool to continue functioning in a
	degraded state.
action: Online the device using 'zpool online' or replace the device with
	'zpool replace'.
config:

	NAME                   STATE     READ WRITE CKSUM
	probe                  DEGRADED     0     0     0
	  mirror-0             DEGRADED     0     0     0
	    /var/tmp/probe/d1  ONLINE       0     0     0
	    /var/tmp/probe/d2  OFFLINE      0     0     0

errors: No known data errors
root@practice:~ # /usr/local/bin/health-check.sh; echo "exit status: $?"
exit status: 1
root@practice:~ # tail -2 /var/log/messages
Oct  6 17:57:09 practice ZFS[44510]: vdev state changed, pool_guid=15432617247174042435 vdev_guid=11204619220799171741
Oct  6 17:57:10 practice healthcheck[49093]: ZFS problem:   pool: probe
root@practice:~ # zpool events | tail -4
Oct  6 2026 17:57:09.308968548 sysevent.fs.zfs.history_event
Oct  6 2026 17:57:09.344983913 sysevent.fs.zfs.config_sync
Oct  6 2026 17:57:09.694970656 resource.fs.zfs.statechange
Oct  6 2026 17:57:09.712978393 sysevent.fs.zfs.config_sync
root@practice:~ # zpool destroy probe; rm -r /var/tmp/probe; zpool status -x
all pools are healthy

Now cron, which makes the check happen without you:

root@practice:~ # crontab -l 2>&1 | head -2
crontab: no crontab for root
root@practice:~ # (crontab -l 2>/dev/null; echo '* * * * * /usr/local/bin/health-check.sh') | crontab -
root@practice:~ # crontab -l
* * * * * /usr/local/bin/health-check.sh
root@practice:~ # grep healthcheck /var/log/messages | tail -4
Oct  6 17:57:08 practice healthcheck[13849]: root filesystem 21% full (limit 10%)
Oct  6 17:57:10 practice healthcheck[49093]: ZFS problem:   pool: probe
Oct  6 17:58:00 practice healthcheck[18266]: ok: disk 21%, pools healthy, services running
Oct  6 17:59:00 practice healthcheck[50720]: ok: disk 21%, pools healthy, services running
root@practice:~ # (crontab -l | grep -v health-check; echo '*/5 * * * * /usr/local/bin/health-check.sh') | crontab -
root@practice:~ # crontab -l
*/5 * * * * /usr/local/bin/health-check.sh
root@practice:~ # grep -c "" /var/cron/log 2>/dev/null; grep health-check /var/log/cron | tail -2
Oct  6 17:58:00 practice /usr/sbin/cron[89236]: (root) CMD (/usr/local/bin/health-check.sh)
Oct  6 17:59:00 practice /usr/sbin/cron[18638]: (root) CMD (/usr/local/bin/health-check.sh)

Where do the warnings go from here? Into /var/log/messages, which nobody reads at 3 a.m. The next step is to turn a warning into a message to a person: a mail to you, a message to a chat service, a call to an alerting service from a line in the script (mail, or a web request): from the documentation, not run here. Or use a monitoring system that does this for you, which is the next layer.

Layer 4: metrics and alerts with Prometheus

The checks above answer "is it OK right now?" They cannot answer "was the disk filling up slowly over the last month?" or "how does this week compare with last week?" For that you store numbers over time. The most common free tool for it is Prometheus, and FreeBSD has packages for the whole kit. It has two parts:

⚠️ Newer package repository (checked October 2026): the package repository now targets FreeBSD 15.1, while 15.0-RELEASE is end of life. On a 15.0 machine pkg may stop with Newer FreeBSD version for package ... [y/N], and the version numbers in the output below may differ from yours. To carry on, run export IGNORE_OSVERSION=yes first, or better, move to a supported release (see the upgrading guide).

root@practice:~ # pkg install -y node_exporter prometheus 2>&1 | grep -E "Number|download|space|Installing node|Installing prom"
Number of packages to be installed: 2
The process will require 212 MiB more space.
45 MiB to be downloaded.
[1/2] Installing node_exporter-1.11.0_9...
[2/2] Installing prometheus-2.55.1_20...
root@practice:~ # grep -nE '_enable|_args|listen|_config|data|rcvar' /usr/local/etc/rc.d/node_exporter | head -12
10:# node_exporter_enable (bool):          Set to NO by default.
16:# node_exporter_args (string):          Set extra arguments to pass to node_exporter
18:# node_exporter_listen_address (string):Set ip:port that node_exporter will listen on
28:rcvar=node_exporter_enable
30:load_rc_config $name
32:: ${node_exporter_enable:="NO"}
35:: ${node_exporter_args:=""}
36:: ${node_exporter_listen_address:=":9100"}
43:command_args="-f -p ${node_exporter_pidfile} -T ${name} \
45:    --web.listen-address=${node_exporter_listen_address} \
47:    ${node_exporter_args}"
69:load_rc_config $name
root@practice:~ # grep -nE '_enable|_args|listen|_config|data|storage|rcvar' /usr/local/etc/rc.d/prometheus | head -14
10:# prometheus_enable (bool)
19:# prometheus_config (string)
25:# prometheus_syslog_output_enable (bool)
46:# prometheus_data_dir (string)
58:# prometheus_args (string)
65:rcvar=prometheus_enable
67:load_rc_config $name
69:: ${prometheus_enable:="NO"}
72:: ${prometheus_config:="/usr/local/etc/prometheus.yml"}
74:: ${prometheus_syslog_output_enable:="YES"}
78:: ${prometheus_data_dir:="/var/db/prometheus"}
82:if checkyesno prometheus_syslog_output_enable; then
97:    prometheus_storage_flags="--enable-feature=agent --storage.agent.path=${prometheus_data_dir}"
99:    prometheus_storage_flags="--storage.tsdb.path=${prometheus_data_dir}"
root@practice:~ # ls /usr/local/etc/prometheus* 2>&1 | head
/usr/local/etc/prometheus.yml
/usr/local/etc/prometheus.yml.sample
root@practice:~ # which promtool
/usr/local/bin/promtool

Two packages, 45 MiB to download (212 MiB installed). The rc scripts show their settings: node_exporter_enable, node_exporter_listen_address (default :9100, every address), and for Prometheus prometheus_enable, prometheus_config (default /usr/local/etc/prometheus.yml), prometheus_data_dir (/var/db/prometheus) and prometheus_args.

root@practice:~ # sysrc node_exporter_enable=YES
node_exporter_enable:  -> YES
root@practice:~ # service node_exporter start
Starting node_exporter.
root@practice:~ # sockstat -4l -6l | grep node_exp
nobody node_expor 39040  3 tcp46 *:9100                *:*                  
root@practice:~ # fetch -qo - http://127.0.0.1:9100/metrics | grep -c '^node_'
435
root@practice:~ # fetch -qo - http://127.0.0.1:9100/metrics | grep -E '^node_(load1|load5|boot_time_seconds|memory_size_bytes|memory_free_bytes|zfs_arc_size|uname_info)' | cut -c1-150
node_boot_time_seconds 1.791309240211743e+09
node_load1 0.2353515625
node_load15 0.1005859375
node_load5 0.21142578125
node_memory_free_bytes 6.23734784e+08
node_memory_size_bytes 2.056421376e+09
node_uname_info{domainname="(none)",machine="amd64",nodename="practice",release="15.0-RELEASE",sysname="FreeBSD",version="FreeBSD 15.0-RELEASE releng/
root@practice:~ # fetch -qo - http://127.0.0.1:9100/metrics | grep -E '^node_filesystem_(avail|size)_bytes.mountpoint="/"' | cut -c1-150
root@practice:~ # fetch -qo - http://127.0.0.1:9100/metrics | grep -E '^node_network_receive_bytes_total.device="vtnet0"'
node_network_receive_bytes_total{device="vtnet0"} 4.9856989e+07
root@practice:~ # sysrc node_exporter_listen_address="127.0.0.1:9100"
node_exporter_listen_address:  -> 127.0.0.1:9100
root@practice:~ # service node_exporter restart
Stopping node_exporter.
Waiting for PIDS: 39040.
Starting node_exporter.
root@practice:~ # sockstat -4l -6l | grep node_exp
nobody node_expor 78725  3 tcp4  127.0.0.1:9100        *:*

Now Prometheus, with a scrape job for the exporter and an alert rule file:

root@practice:~ # cp /usr/local/etc/prometheus.yml /usr/local/etc/prometheus.yml.orig
root@practice:~ # cat > /usr/local/etc/prometheus.yml <<'EOF'
> global:
>   scrape_interval: 5s
>   evaluation_interval: 5s
> rule_files:
>   - /usr/local/etc/prometheus-rules.yml
> scrape_configs:
>   - job_name: node
>     static_configs:
>       - targets: ['127.0.0.1:9100']
> EOF
root@practice:~ # cat > /usr/local/etc/prometheus-rules.yml <<'EOF'
> groups:
>   - name: lab
>     rules:
>       - alert: HighLoad
>         expr: node_load1 > 4
>         for: 1m
>       - alert: RootDiskLow
>         expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.9
>         for: 10s
>         annotations:
>           summary: "root file system has less than 90 percent free (a demo threshold, deliberately strict)"
> EOF
root@practice:~ # promtool check config /usr/local/etc/prometheus.yml
Checking /usr/local/etc/prometheus.yml
  SUCCESS: 1 rule files found
 SUCCESS: /usr/local/etc/prometheus.yml is valid prometheus config file syntax

Checking /usr/local/etc/prometheus-rules.yml
  SUCCESS: 2 rules found
root@practice:~ # sysrc prometheus_enable=YES
prometheus_enable:  -> YES
root@practice:~ # sysrc prometheus_args="--web.listen-address=127.0.0.1:9090"
prometheus_args:  -> --web.listen-address=127.0.0.1:9090
root@practice:~ # service prometheus start
Starting prometheus.
root@practice:~ # sockstat -4l -6l | grep -E "prometheus|node_exp"
prometheus prometheus 37431  5 tcp4  127.0.0.1:9090        *:*                  
nobody     node_expor 78725  3 tcp4  127.0.0.1:9100        *:*                  
root@practice:~ # fetch -qo - http://127.0.0.1:9090/-/healthy
Prometheus Server is Healthy.

We started it with prometheus_args="--web.listen-address=127.0.0.1:9090": its own web port is also open to everyone by default, so we bound it to the loopback address too. sockstat shows both programs on 127.0.0.1, and the health check answers Prometheus Server is Healthy.

root@practice:~ # fetch -qo - 'http://127.0.0.1:9090/api/v1/query?query=up' | cut -c1-300
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"up","instance":"127.0.0.1:9100","job":"node"},"value":[1791309670.490,"1"]}]}}
root@practice:~ # fetch -qo - 'http://127.0.0.1:9090/api/v1/query?query=node_load1' | cut -c1-300
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"node_load1","instance":"127.0.0.1:9100","job":"node"},"value":[1791309670.695,"0.26416015625"]}]}}
root@practice:~ # fetch -qo - 'http://127.0.0.1:9090/api/v1/alerts' | cut -c1-420
{"status":"success","data":{"alerts":[{"labels":{"alertname":"RootDiskLow","device":"zroot/ROOT/default","fstype":"zfs","instance":"127.0.0.1:9100","job":"node","mountpoint":"/"},"annotations":{"summary":"root file system has less than 90 percent free (a demo threshold, deliberately strict)"},"state":"firing","activeAt":"2026-10-06T18:00:56.424967785Z","value":"7.945111440277886e-01"}]}}

Prometheus has an HTTP API. We asked it questions with fetch:

root@practice:~ # fetch -qo - 'http://127.0.0.1:9090/api/v1/alerts' | grep -o '"alertname":"[A-Za-z]*"\|"state":"[a-z]*"'
"alertname":"RootDiskLow"
"state":"firing"
root@practice:~ # fetch -qo - 'http://127.0.0.1:9090/api/v1/rules' | grep -o '"name":"[A-Za-z]*","query"\|"health":"[a-z]*"' | head -6
"name":"HighLoad","query"
"health":"ok"
"name":"RootDiskLow","query"
"health":"ok"
root@practice:~ # fetch -qo - 'http://127.0.0.1:9090/api/v1/query?query=node_memory_free_bytes/node_memory_size_bytes' | grep -o '"value":\[[^]]*\]'
"value":[1791309682.188,"0.24175390793058943"]

The alert's state is firing, as intended, and /api/v1/rules shows both rules with health: ok (they are valid and evaluating). And a calculation: node_memory_free_bytes/node_memory_size_bytes is 0.24: about a quarter of the memory is free. The query language can compute with metrics.

And the most important test of any monitoring system: what happens when the monitored thing dies?

root@practice:~ # service node_exporter stop
Stopping node_exporter.
Waiting for PIDS: 78725.
root@practice:~ # fetch -qo - 'http://127.0.0.1:9090/api/v1/query?query=up' | grep -o '"value":\[[^]]*\]'
"value":[1791309697.587,"0"]
root@practice:~ # service node_exporter start
Starting node_exporter.
root@practice:~ # fetch -qo - 'http://127.0.0.1:9090/api/v1/query?query=up' | grep -o '"value":\[[^]]*\]'
"value":[1791309709.989,"1"]

We stopped the exporter. Fifteen seconds later up was 0. We started it again and up was 1 again twelve seconds after that. An alert rule such as up == 0 for 1m is the standard "the machine or the exporter is down" alarm, and the best single alert there is, because it also catches the case where the whole monitoring path is broken.

Prometheus also has a web page of its own (a graph browser), at the same port: from the documentation, not run here. Two more packages complete the picture, and both are in the package list: Alertmanager, which takes the firing alerts and sends them to mail or chat (with grouping, silencing and so on), and Grafana, for dashboards. We did not run either. To watch other machines, add more targets to the scrape job, and install node_exporter on each (and open its port to the Prometheus server only): from the documentation, not run here.

A short look at SNMP

Network equipment (routers, switches, printers) mostly speaks SNMP, and FreeBSD's base system has an SNMP agent, bsnmpd:

root@practice:~ # grep -nE 'read :=|write :=|community|^begemotSnmpdPortStatus|:= *"' /etc/snmpd.config | head -8
8:location := "Room 200"
9:contact := "sysmeister@example.com"
24:read := "public"
25:# Uncomment begemotSnmpdCommunityString.0.2 below that sets the community
27:write := "geheim"
28:trap := "mytrap"
70:#user1 := "bsnmp"
83:# The default value of the community strings is NULL (note, that this is
root@practice:~ # sysrc bsnmpd_enable=YES
bsnmpd_enable: NO -> YES
root@practice:~ # service bsnmpd start
Starting bsnmpd.
root@practice:~ # sockstat -4l -6l | grep bsnmpd
root       bsnmpd     13193  5 udp4  *:*                   *:*                  
root       bsnmpd     13193  6 udp4  *:161                 *:*                  
root       bsnmpd     13193  7 udp6  *:161                 *:*                  
root@practice:~ # bsnmpget -s public@127.0.0.1 sysDescr.0 sysUpTime.0 2>&1 | cut -c1-150
sysDescr.0 = practice 3019700627 FreeBSD 15.0-RELEASE
sysUpTime.0 = 34
root@practice:~ # bsnmpwalk -s public@127.0.0.1 ifInOctets 2>&1 | head -4
No Such Instance
ifInOctets[0] = 
root@practice:~ # bsnmpwalk -s public@127.0.0.1 begemotSnmpdModuleSection 2>&1 | head -4
No Such Object
begemotSnmpdModuleSection[] = 
root@practice:~ # grep -n "^begemotSnmpdModulePath" /etc/snmpd.config | head -6
143:begemotSnmpdModulePath."mibII"	= "/usr/lib/snmp_mibII.so"
root@practice:~ # service bsnmpd stop
Stopping bsnmpd.
Waiting for PIDS: 13193.
root@practice:~ # sysrc bsnmpd_enable=NO
bsnmpd_enable: YES -> NO
root@practice:~ # sockstat -4l -6l | grep -c bsnmpd
0

What to alert on

Do not alert on every number. Alert on conditions that need a person to act, and on the absence of the monitoring itself. A good first list, every item of which you have seen above:

Tune the thresholds so that every alert means something. If an alert goes off every day and is ignored, it is worse than no alert.

Monitoring the rest of the corner

What we did not do

All from the documentation, not run here: Grafana dashboards, Alertmanager, other monitoring systems (Nagios-style checks with plug-ins for ZFS and pf, Zabbix, netdata: all in the package list), smartmontools for disk health (a virtual disk has no SMART data, so there was nothing to test), zfsd, FreeBSD's daemon that reacts to disk faults (it can replace a failed disk with a spare), and DTrace for deep tracing. Also remote log collection.

⚠️ Do not leave monitoring ports open to the world. node_exporter (9100) and Prometheus (9090) listen on every address by default, and bsnmpd (161) arrives with well-known passwords. Bind them to a private address or 127.0.0.1, and use the firewall to allow only your monitoring server. A monitoring system should not be the way in.

Why did the sysadmin set the disk alert to 99 percent?<br>
So that it would give him a full 30 seconds<br>
of warning before the weekend.

🧪 Try it yourself

On a FreeBSD virtual machine:

1. Run uptime, top -b, vmstat 1 3, iostat -x -c 2 -w 1 DISK and zpool iostat 1 3. How many processors do you have, and is the load below that? 2. Run zpool status -x. Then make a mirror of two files, take one offline, and run it again. What changed? 3. Create /etc/periodic.conf with daily_output set to a file and daily_status_zfs_enable="YES". Run periodic daily and read the report. 4. Write /usr/local/bin/health-check.sh (copy ours), check it with sh -n, run it, and force a failure with MAX_DISK=10. 5. Log a line with logger -p daemon.info and another with daemon.notice. Which one reaches /var/log/messages? 6. Put the script in cron, and watch the log for two minutes. 7. Install node_exporter and prometheus. Bind both to 127.0.0.1. Check the config with promtool check config. 8. Ask the API for up. Stop the exporter. What does up say 15 seconds later? 9. Write an alert rule for free disk space with a threshold that fires. Look at /api/v1/alerts.

The cheat sheet

← FreeBSD in the Cloud: First-Boot Provisioning with nuageinit · ↑ FreeBSD Corner · ZFS Backups to Another Machine: Send, Receive, Restore →

© 2026 TechCommune. All rights reserved.
"If nobody is told when it breaks, it is not monitored. It is just running."