🚨 Nagios Core Guide

Nagios Core: the monitoring server that checks things and tells you
sudo apt install nagios4 apache2 monitoring-plugins nagios-nrpe-plugin
sudo nagios4 -v /etc/nagios4/nagios.cfg

Nagios Core runs small programs called plugins on a schedule, reads the exit code of each one, and turns the result into a state (OK, WARNING, CRITICAL, UNKNOWN). When a state changes and stays changed, it tells the right people. That is the whole idea; the rest is configuration. It does not draw pretty graphs on its own and it does not discover your network. You tell it what exists (hosts), what to check on each (services), who to tell (contacts) and when (time periods), in plain text files.

Every example below ran for real: Nagios Core 4.4.6 (the nagios4 package of Ubuntu 24.04), monitoring-plugins 2.3.5, NRPE 4.1.0, Apache 2.4.58, in two Docker containers on a private network: "nag" (the Nagios server) and "web1" (a web server with nginx, sshd and the NRPE agent). A third host, web2, was pointed at an address that does not exist, to see what a dead host looks like. A container has no systemd, so the lab started the daemons by hand; commands that need systemd are marked "not run in the lab". Times were shortened (one-minute check intervals, three attempts) so that outages played out in seconds; the shipped defaults are slower, and each section says so.

Four things to learn first.

  host       a machine or device with an address: web1, a switch, a printer
  service    one thing checked on a host: PING, HTTP, SSH, "Root disk". A service always belongs to a host
  plugin     a small program that does one check, prints one line, and exits with a code
  contact    a person (or a group) who is told when something changes

The exit code of the plugin is the state:

  exit 0   OK
  exit 1   WARNING
  exit 2   CRITICAL
  exit 3   UNKNOWN      (the plugin could not tell: bad arguments, timeout, cannot connect to the thing that holds the answer)

Hosts have their own states: UP, DOWN and UNREACHABLE (down because something on the path to it is down).

A state is also SOFT or HARD. A problem is SOFT when it was seen but is not yet confirmed (attempt 1 of 3, 2 of 3). It becomes HARD when it has been seen the full number of attempts in a row (max_check_attempts). Only HARD changes send notifications. This is what keeps a single lost packet from waking you. The lab saw exactly that on web1 (details in "Soft and hard states").

Four packages cover a working server with a web interface and remote checks:

$ sudo apt install nagios4 apache2 monitoring-plugins nagios-nrpe-plugin
$ nagios4 --version | head -2
Nagios Core 4.4.6
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors

  nagios4              4.4.6-4ubuntu0.24.04.1     the daemon, the web pages (CGI programs) and the default configuration
  apache2              2.4.58                     the web server for the pages
  monitoring-plugins   2.3.5-1ubuntu3             the plugins (the project was renamed from "nagios-plugins")
  nagios-nrpe-plugin   4.1.0-1ubuntu3             check_nrpe, to run checks on other machines

Ubuntu 24.04 ships Nagios Core 4.4.6. The upstream project has a newer release; installing it means building from source, which was not run here. The packaged version is the one this guide describes.

Two things the lab needed that you should check on a real machine:

1. The web pages are CGI programs, and Apache must have mod_cgi enabled. In the lab it was not, and the pages did not run: Apache sent the programs as file downloads, a screen of binary garbage. The fix is one command:

$ sudo a2enmod cgi
$ sudo systemctl restart apache2        (not run in the lab: no systemd in a container)

2. The daemon must be running. On a normal server the package installs a systemd unit: systemctl status nagios4 shows it (not run in the lab, where the daemon was started by hand with nagios4 -d /etc/nagios4/nagios.cfg).

Before you start, check the configuration. The packaged one is valid out of the box:

$ nagios4 -v /etc/nagios4/nagios.cfg
Checking objects...
        Checked 8 services.
        Checked 1 hosts.
        Checked 1 host groups.
        Checked 0 service groups.
        Checked 1 contacts.
        Checked 1 contact groups.
        Checked 180 commands.
        Checked 5 time periods.
...
Total Warnings: 0
Total Errors:   0

Things look okay - No serious problems were detected during the pre-flight check

The web interface is at http://SERVER/nagios4/. Out of the box it is open to private networks only and has no login (see "The web interface and its security" before you change that).

The Debian and Ubuntu package is laid out like this (the upstream tarball uses /usr/local/nagios instead):

  /etc/nagios4/nagios.cfg           the main file: which other files to read, where the logs and status files go, global switches
  /etc/nagios4/conf.d/              your own object files: put every new host and service here (it starts empty)
  /etc/nagios4/objects/             the shipped definitions: commands.cfg, contacts.cfg, templates.cfg, timeperiods.cfg, localhost.cfg and some examples (printer, switch, windows)
  /etc/nagios4/cgi.cfg              settings for the web pages, including the login switch
  /etc/nagios4/htdigest.users       the web users (digest authentication)
  /etc/nagios4/resource.cfg         $USER1$ and friends, the macros for paths and secrets: $USER1$=/usr/lib/nagios/plugins
  /etc/nagios-plugins/config/       one command file per plugin family (ping, http, nrpe...), read by the main file
  /usr/lib/nagios/plugins/          the plugins themselves
  /etc/apache2/conf-available/nagios4-cgi.conf   the Apache side
  /var/log/nagios4/nagios.log       the log: every state change, notification and command
  /var/lib/nagios4/status.dat       the current state of everything (rewritten every 10 seconds)
  /var/lib/nagios4/retention.dat    state kept across restarts
  /var/lib/nagios4/rw/nagios.cmd    the command file (a named pipe): how you give Nagios orders from outside

The main file is a list of includes. These are the lines that matter in the shipped nagios.cfg:

cfg_dir=/etc/nagios-plugins/config
cfg_dir=/etc/nagios4/conf.d
cfg_file=/etc/nagios4/objects/commands.cfg
cfg_file=/etc/nagios4/objects/contacts.cfg
cfg_file=/etc/nagios4/objects/timeperiods.cfg
cfg_file=/etc/nagios4/objects/templates.cfg
cfg_file=/etc/nagios4/objects/localhost.cfg

cfg_dir reads every .cfg file in the folder, so adding a monitored host is "drop a file into conf.d". Keep your files there and leave the shipped ones alone.

The packaged configuration already monitors the server itself (localhost) with eight services: PING, Root Partition, Current Users, Total Processes, Current Load, Swap Usage, SSH and HTTP.

Nagios reads its configuration at start and when told to reload. A mistake does not stop at a warning. Always run the pre-flight check first:

$ sudo nagios4 -v /etc/nagios4/nagios.cfg
...
Total Warnings: 0
Total Errors:   0

Then reload (the daemon re-reads everything without losing state):

$ sudo systemctl reload nagios4        (not run in the lab; the lab sent the signal directly: kill -HUP PID)

Why the order matters, as it happened in the lab: a reload was sent with a broken file without checking first. The daemon did not keep running on the old configuration. It stopped:

Error: Service check command 'check_dummy!3!No backup report received yet' specified in service 'Nightly backup' for host 'web1' not defined anywhere!
Bailing out due to errors encountered while running the pre-flight check.  Run Nagios from the command line with the -v option to verify your config before restarting.

and from that moment nothing was monitored, nothing alerted, and the web pages went silent until it was started again. So: validate, then reload, in one line you can paste or script:

nagios4 -v /etc/nagios4/nagios.cfg && sudo systemctl reload nagios4

The lab used this small script for the same idea (it only signals when the check is clean):

#!/bin/bash
nagios4 -v /etc/nagios4/nagios.cfg >/tmp/preflight.txt 2>&1 || { grep -E "Error|Total" /tmp/preflight.txt; echo "NOT reloading"; exit 1; }
kill -HUP "$(cat /var/run/nagios4/nagios4.pid)" && echo "reload signal sent"

What the errors look like. Real ones from the lab, each from a small deliberate mistake:

a service on a host that is not defined (typo in host_name):
Error: Could not find any host matching 'web9' (config file '/etc/nagios4/conf.d/bad.cfg', starting on line 1)
Error: Failed to expand host list 'web9' for service 'Typo host' (/etc/nagios4/conf.d/bad.cfg:1)

a template name that does not exist:
Error: Template 'no-such-template' specified in service definition could not be not found (config file '/etc/nagios4/conf.d/bad.cfg', starting on line 8)

a misspelt directive (addres instead of address):
Error: Invalid host object directive 'addres'.
Error: Could not add object property in file '/etc/nagios4/conf.d/bad.cfg' on line 19.
Error: Failed to locate notification_period '24x7' for host 'web1'!
Error: Could not register host (config file '/etc/nagios4/conf.d/web1.cfg', starting on line 1)

Read the first error and fix that one. The last two lines of the third case are not about web1: once a file fails to parse, Nagios cannot finish reading the objects, and the checks that follow blame innocent files. It also stops at parse errors before it looks at references, so the typo in host_name in the same file was not reported until the directive was fixed. Fix, validate, repeat.

A name rule from the lab: service descriptions cannot contain some characters. "Load (NRPE)" was refused:

Error: The description string for service 'Load (NRPE)' on host 'web1' contains one or more illegal characters.

"Load" was accepted. Keep names to letters, digits, spaces, dashes and underscores.

A plugin is an ordinary program. You can run it in a shell, look at the output and the exit code, and tune the thresholds before you give it to Nagios. This is the best habit in monitoring: if it does not work by hand, it will not work from Nagios either.

$ P=/usr/lib/nagios/plugins
$ $P/check_http -H 172.21.0.3 ; echo "exit=$?"
HTTP OK: HTTP/1.1 200 OK - 857 bytes in 0.001 second response time |time=0.000933s;;;0.000000;10.000000 size=857B;;;0;
exit=0
$ $P/check_http -H 172.21.0.3 -u /nothing.html ; echo "exit=$?"
HTTP WARNING: HTTP/1.1 404 Not Found - 321 bytes in 0.001 second response time |time=0.000823s;;;0.000000;10.000000 size=321B;;;0;
exit=1
$ $P/check_tcp -H 172.21.0.3 -p 5666 ; echo "exit=$?"
TCP OK - 0.000 second response time on 172.21.0.3 port 5666|time=0.000177s;;;0.000000;10.000000
exit=0
$ $P/check_ping -H 172.21.0.3 -w 100,20% -c 500,60% -p 3 ; echo "exit=$?"
PING OK - Packet loss = 0%, RTA = 0.12 ms|rta=0.123000ms;100.000000;500.000000;0.000000 pl=0%;20;60;0;
exit=0
$ $P/check_ssh 172.21.0.3
SSH OK - OpenSSH_9.6p1 Ubuntu-3ubuntu13.20 (protocol 2.0) | time=0.016762s;;;0.000000;10.000000

The output has two parts, split at the | sign. Before it, the text people read. After it, the performance data: name=value;warn;crit;min;max, which graphing tools read. Nagios shows the first part and keeps the second.

Thresholds: -w is the warning level and -c the critical level. The simple form is "alert when the value is above this":

$ $P/check_procs -C apache2 -w 10 -c 20 ; echo "exit=$?"
PROCS OK: 6 processes with command name 'apache2' | procs=6;10;20;0;
exit=0

Ranges say more. The syntax is the same in all plugins that follow the standard (the lab ran them on check_procs, which saw 6 apache2 processes):

  -w 10            alert if the value is outside 0 to 10 (above 10)
  -w 1:2           alert if the value is outside 1 to 2 (too few or too many)
  -w 1:            alert if the value is below 1
  -w '~:3'         alert if the value is above 3 (the ~ means minus infinity)
  -w @1:10         alert if the value is INSIDE 1 to 10 (the @ turns the range around)

$ $P/check_procs -C apache2 -w 1:2 -c 1:3 ; echo "exit=$?"
PROCS CRITICAL: 6 processes with command name 'apache2' | procs=6;1:2;1:3;0;
exit=2
$ $P/check_procs -C apache2 -w @1:10 -c 20 ; echo "exit=$?"
PROCS WARNING: 6 processes with command name 'apache2' | procs=6;@1:10;20;0;
exit=1

Gotcha: write '~:3' in quotes. The shell turns an unquoted ~ into your home directory, and the lab saw exactly that: -w ~:3 reached the plugin as /root:3 and was reported as procs=6;/root:3;/root:4;0.

Gotcha for disks: check_disk thresholds are about FREE space, not used space. On a disk that is 77 percent free:

$ $P/check_disk -w 20% -c 10% -p / ; echo "exit=$?"
DISK OK - free space: / 685143MiB (77% inode=97%);| /=213844492288B;785791949209;884015942860;0;982239936512
exit=0
$ $P/check_disk -w 99% -c 98% -p / ; echo "exit=$?"
DISK CRITICAL - free space: / 685143MiB (77% inode=97%);| /=213844492288B;9822399365;19644798730;0;982239936512
exit=2

A plugin of your own can be any program: print one line, exit 0, 1, 2 or 3. check_dummy returns whatever you tell it, which is handy for testing:

$ $P/check_dummy 3 "plugin exit codes" ; echo "exit=$?"
UNKNOWN: plugin exit codes
exit=3

Each plugin has --help with the whole list of options.

Objects are written as define blocks. A minimal monitored web server is one host and a few services. This is the file the lab used (/etc/nagios4/conf.d/web1.cfg, shortened intervals as noted):

define host {
    use                 linux-server
    host_name           web1
    alias               Web server 1
    address             172.21.0.3
    hostgroups          web-servers
    check_interval      1               ; the template says 5
    max_check_attempts  3               ; the template says 10
    notification_period 24x7            ; the template says workhours
}

define hostgroup {
    hostgroup_name  web-servers
    alias           Web servers
}

define service {
    use                   local-service
    host_name             web1
    service_description   HTTP
    check_command         check_http
    check_interval        1
    max_check_attempts    3
}

define service {
    use                   local-service
    host_name             web1
    service_description   SSH
    check_command         check_ssh
    check_interval        1
    max_check_attempts    3
}

Reading it:
  use          inherit everything from a template (next paragraph), then override what you list
  check_command  the name of a command object. check_http and check_ssh come from the plugin command files in /etc/nagios-plugins/config
  ; text       a comment

Templates carry the defaults, so each host and service stays short. The shipped ones, in objects/templates.cfg, with the values that matter:

  linux-server     check every 5 min, retry every 1 min, max_check_attempts 10, notifications only in "workhours", contact group admins
  local-service    check every 5 min, retry every 1 min, max_check_attempts 4
  generic-contact  notifies by email, all states, 24x7

Read those numbers: a host that stops answering is only declared DOWN (HARD) after 10 failed checks one minute apart, so about ten minutes, and a linux-server host sends no notification outside Monday to Friday 9 to 5 (workhours: monday to friday 09:00-17:00, no weekends). Plenty of sites are surprised by one or the other. Change them for what you want, per host or in your own template.

Groups save repeating yourself. A service attached to a host group applies to every member, present and future:

define service {
    use                   local-service
    hostgroup_name        web-servers
    service_description   Web response
    check_command         check_http!-w 1 -c 3
}

check_http!-w 1 -c 3 passes the text after the ! as $ARG1$ to the command definition. In the lab, adding the host web2 to the group web-servers gave it the Web response service with no extra lines. A service group collects services (for the web page and for reports):

define servicegroup {
    servicegroup_name   web-checks
    alias               Web checks
    members             web1,HTTP,web1,Web response     ; host,service pairs
}

Commands connect a name to a plugin line. Your own are one definition:

define command {
    command_name    check_dummy_msg
    command_line    $USER1$/check_dummy $ARG1$ "$ARG2$"
}

and a service uses it as check_command check_dummy_msg!3!No backup report received yet. $USER1$ is /usr/lib/nagios/plugins from resource.cfg. Other macros, such as $HOSTADDRESS$, are filled in by Nagios for each host.

This is what the log records when something breaks, with check_interval 1 and max_check_attempts 3 (the times are converted from the log's seconds-since-1970 to hh:mm:ss, UTC). nginx was stopped on web1 and the HTTP check was run three times (the lab used forced checks to make it quicker; on its own, Nagios does the same with the retry interval):

$ sudo nginx -s stop            (on web1)
15:26:44 SERVICE ALERT: web1;HTTP;CRITICAL;SOFT;1;connect to address 172.21.0.3 and port 80: Connection refused
15:26:48 SERVICE ALERT: web1;HTTP;CRITICAL;SOFT;2;connect to address 172.21.0.3 and port 80: Connection refused
15:26:52 SERVICE NOTIFICATION: nagiosadmin;web1;HTTP;CRITICAL;notify-service-by-email;connect to address 172.21.0.3 and port 80: Connection refused
15:26:52 SERVICE ALERT: web1;HTTP;CRITICAL;HARD;3;connect to address 172.21.0.3 and port 80: Connection refused

SOFT 1, SOFT 2: seen, not confirmed, nobody told. HARD 3: confirmed, and the notification is sent in the same second. When the service comes back:

$ sudo nginx                    (on web1)
15:27:34 SERVICE NOTIFICATION: nagiosadmin;web1;HTTP;OK;notify-service-by-email;HTTP OK: HTTP/1.1 200 OK - 857 bytes in 0.001 second response time
15:27:34 SERVICE ALERT: web1;HTTP;OK;HARD;3;HTTP OK: HTTP/1.1 200 OK - 857 bytes in 0.001 second response time

A host works the same way. web2 pointed at an address nobody uses:

15:30:12 HOST ALERT: web2;DOWN;SOFT;1;CRITICAL - Host Unreachable (172.21.0.99)
15:30:15 HOST ALERT: web2;DOWN;SOFT;2;CRITICAL - Host Unreachable (172.21.0.99)
15:30:21 HOST NOTIFICATION: nagiosadmin;web2;DOWN;notify-host-by-email;CRITICAL - Host Unreachable (172.21.0.99)
15:30:21 HOST ALERT: web2;DOWN;HARD;3;CRITICAL - Host Unreachable (172.21.0.99)

The time to a HARD state is roughly (max_check_attempts - 1) x retry_interval after the first failure. With the shipped defaults (10 and 1 minute for a host) a dead server is only announced about nine minutes after the first failed check. Choose the numbers on purpose: fewer attempts for critical servers, more for flaky links.

Flapping. When a state keeps switching, Nagios stops treating each change as news. The lab started and stopped nginx quickly and the log said:

15:28:10 SERVICE FLAPPING ALERT: web1;HTTP;STARTED; Service appears to have started flapping (24.1% change >= 20.0% threshold)

Notifications are held while it flaps. If you see this on a real service, the thing is unstable or the check is too sensitive.

The plugins on the Nagios server can reach network services (HTTP, SSH, ping, TCP). To see inside another machine (load, disk, processes), run the plugin there and ask for the result. NRPE (Nagios Remote Plugin Executor) is that bridge: an agent (nrpe) on the target, and check_nrpe on the server.

Target machine (web1):

$ sudo apt install nagios-nrpe-server monitoring-plugins-basic monitoring-plugins-standard

Edit /etc/nagios/nrpe.cfg and allow the Nagios server (comma separated, localhost first):

allowed_hosts=127.0.0.1,::1,172.21.0.2

The agent ships with a few commands defined in the same file, each one a plugin line with fixed arguments:

command[check_users]=/usr/lib/nagios/plugins/check_users -w 5 -c 10
command[check_load]=/usr/lib/nagios/plugins/check_load -r -w .15,.10,.05 -c .30,.25,.20
command[check_hda1]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/hda1
command[check_zombie_procs]=/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -w 150 -c 200

Look at check_load: those thresholds (.15 scaled) are so low that a perfectly quiet machine reports WARNING. The lab did, in its first minute:

$ /usr/lib/nagios/plugins/check_nrpe -H 172.21.0.3 -c check_load ; echo "exit=$?"
LOAD WARNING - scaled load average: 0.06, 0.08, 0.09 - total load average: 0.50, 0.65, 0.73|load1=0.500;;;0; scaled_load1=0.062;0.150;0.300;0; ...
exit=1

Do not edit the shipped lines. Put your own in /etc/nagios/nrpe_local.cfg, which the main file includes at the end. A second definition of the same name wins:

command[check_root]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /
command[check_load]=/usr/lib/nagios/plugins/check_load -w 4,3,2 -c 8,6,4

Restart the agent (sudo systemctl restart nagios-nrpe-server; not run in the lab, which started and restarted nrpe -c /etc/nagios/nrpe.cfg -d by hand). Then, from the Nagios server:

$ /usr/lib/nagios/plugins/check_nrpe -H 172.21.0.3
NRPE v4.1.0
$ /usr/lib/nagios/plugins/check_nrpe -H 172.21.0.3 -c check_load
LOAD OK - total load average: 0.59, 0.55, 0.67|load1=0.590;4.000;8.000;0; load5=0.550;3.000;6.000;0; load15=0.670;2.000;4.000;0;

The first command with no -c is the way to test the connection: it prints the agent version. Then the Nagios side. The lab defined one command and used it for services:

define command {
    command_name    check_nrpe_cmd
    command_line    $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

define service {
    use                   local-service
    host_name             web1
    service_description   Load
    check_command         check_nrpe_cmd!check_load
}

define service {
    use                   local-service
    host_name             web1
    service_description   Root disk
    check_command         check_nrpe_cmd!check_root
}

(The package also installs check_nrpe and check_nrpe_nossl with the same idea.) Errors you will see, each reproduced in the lab:

CHECK_NRPE: Error - Could not connect to 172.21.0.3: Connection reset by peer       exit 2: the server is not in allowed_hosts of the agent (or a firewall is in the way)
NRPE: Command 'no_such_cmd' not defined                                                exit 3: the name after -c is not a command[...] line on the agent
CHECK_NRPE: Receive header underflow - only 0 bytes received (4 expected).             exit 3: arguments were sent (-a 1 2 3) and the agent refused them. Agents refuse arguments by default, which is the safe setting (dont_blame_nrpe=0)

Keep dont_blame_nrpe at 0 and define fixed commands. Allowing arguments lets whoever can reach the agent choose what it runs. An alternative that needs no agent is check_by_ssh (the plugin is there; it was not run in the lab). Also not run: encryption settings of NRPE, which matter when the agent is reachable from untrusted networks, so restrict port 5666 to the Nagios server with a firewall.

The shipped contact is one person, and the shipped command sends mail through /usr/bin/mail:

define contact {
    contact_name   nagiosadmin
    use            generic-contact
    alias          Nagios Admin
    email          nagios@localhost
}

define contactgroup {
    contactgroup_name  admins
    alias              Nagios Administrators
    members            nagiosadmin
}

Change the email line to a real address. Then make sure the server can send mail, because Nagios does not: it runs the command and moves on. The shipped service command is

/usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$ ..." | /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$

Test that command's last part by hand first. The lab installed a local-only mail setup (sudo apt install postfix mailutils, choosing "Local only") and sent a test:

$ echo "lab test body" | mail -s "lab test" nagios@localhost
$ ls -l /var/mail/nagios
-rw------- 1 nagios mail 2057 Oct 14 15:25 /var/mail/nagios

For real mail you need an MTA that delivers outside (postfix as a satellite or relay, or msmtp); that setup was not run here. If the test does not arrive, no alert will.

What an alert looks like when it arrives (from the lab):

Subject: ** PROBLEM Service Alert: Web server 1/HTTP is CRITICAL **

***** Nagios *****

Notification Type: PROBLEM

Service: HTTP
Host: Web server 1
Address: 172.21.0.3
State: CRITICAL

Date/Time: Wed Oct 14 15:26:52 UTC 2026

Additional Info:

connect to address 172.21.0.3 and port 80: Connection refused

The subjects seen in the lab, for the four message types: PROBLEM, RECOVERY, ACKNOWLEDGEMENT and, for a host, ** PROBLEM Host Alert: web2 is DOWN **.

Which states notify is set in the contact (service_notification_options w,u,c,r,f,s means warning, unknown, critical, recovery, flapping, scheduled downtime) and in the host or service (notification_options, notification_interval, notification_period). The time periods are plain definitions:

define timeperiod { name 24x7       timeperiod_name 24x7       sunday 00:00-24:00 ... saturday 00:00-24:00 }
define timeperiod { name workhours  timeperiod_name workhours  monday 09:00-17:00 ... friday 09:00-17:00 }
define timeperiod { name none       timeperiod_name none }     ; no time is a good time

Some services should not page anyone. The shipped localhost SSH and HTTP services have notifications_enabled 0 for exactly that reason. Escalations (tell someone else if nobody reacts) and dependencies (do not alert about the web site when its database is down) exist as host and service escalation and dependency objects; they were not run in the lab.

Everything in the web interface that changes something is an external command: one line written to the command file, /var/lib/nagios4/rw/nagios.cmd. It is a named pipe owned by nagios, readable by the web server group (check_external_commands=1 in nagios.cfg). You can write to it from a script. The lab used this one-line helper (run it as the nagios user or root):

#!/bin/bash
printf '[%lu] %s\n' "$(date +%s)" "$1" > /var/lib/nagios4/rw/nagios.cmd

Check now. Do not wait for the next interval:

ncmd "SCHEDULE_FORCED_SVC_CHECK;web1;HTTP;$(date +%s)"

Acknowledge. Tell Nagios you know. It stops repeating the notification for that problem until the state changes. Arguments: host; service; sticky; notify; persistent; author; comment.

$ ncmd "ACKNOWLEDGE_SVC_PROBLEM;web1;HTTP;1;1;1;admin;nginx stopped on purpose, config change"
15:27:15 EXTERNAL COMMAND: ACKNOWLEDGE_SVC_PROBLEM;web1;HTTP;1;1;1;admin;nginx stopped on purpose, config change
15:27:15 SERVICE NOTIFICATION: nagiosadmin;web1;HTTP;ACKNOWLEDGEMENT (CRITICAL);notify-service-by-email;connect to address 172.21.0.3 and port 80: Connection refused;admin;ngi...

Colleagues get a message with your comment (Subject: ** ACKNOWLEDGEMENT Service Alert: Web server 1/HTTP is CRITICAL **), and the next forced check sent no second alert. When the service recovered, the acknowledgement cleared by itself and the RECOVERY mail followed.

Schedule downtime for planned work. Arguments: host; service; start; end; fixed; trigger; duration; author; comment (times in seconds since 1970).

$ N=$(date +%s); E=$((N+1800))
$ ncmd "SCHEDULE_SVC_DOWNTIME;web1;HTTP;$N;$E;1;0;1800;admin;nginx upgrade window"
15:27:45 SERVICE DOWNTIME ALERT: web1;HTTP;STARTED; Service has entered a period of scheduled downtime
15:27:48 SERVICE ALERT: web1;HTTP;CRITICAL;SOFT;1;connect to address 172.21.0.3 and port 80: Connection refused
15:27:52 SERVICE ALERT: web1;HTTP;CRITICAL;SOFT;2;...
15:27:56 SERVICE ALERT: web1;HTTP;CRITICAL;HARD;3;...

Nginx was stopped during the window. The alerts were logged, but no email was sent: the last subject in the mailbox stayed the earlier RECOVERY. The state data confirmed it (scheduled_downtime_depth was 1 while the window was open). End a downtime early with DEL_DOWNTIME_BY_HOST_NAME;web1;HTTP, which the lab did:

[1791991690] EXTERNAL COMMAND: DEL_DOWNTIME_BY_HOST_NAME;web1;HTTP
[1791991690] SERVICE DOWNTIME ALERT: web1;HTTP;CANCELLED; Scheduled downtime for service has been cancelled.

Passive checks: the thing being watched reports in, instead of Nagios going to look. Right for jobs that run on their own schedule (a nightly backup), for events, and for hosts Nagios cannot reach. Define the service so it expects results and complains when they stop coming:

define service {
    use                     local-service
    host_name               web1
    service_description     Nightly backup
    check_command           check_dummy_msg!3!No backup report received yet
    active_checks_enabled   0
    passive_checks_enabled  1
    check_freshness         1
    freshness_threshold     86400          ; seconds: with no report for a day, Nagios runs the check command (here it returns UNKNOWN; the stale case was not run in the lab)
    max_check_attempts      1
}

and the backup job ends with one line:

$ ncmd "PROCESS_SERVICE_CHECK_RESULT;web1;Nightly backup;0;BACKUP OK - 14.2 GB written in 312 s"
15:29:06 EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;web1;Nightly backup;0;BACKUP OK - 14.2 GB written in 312 s
SERVICE web1       Nightly backup OK        HARD 1/1  BACKUP OK - 14.2 GB written in 312 s

The number after the service name is the exit code, 0 to 3. From another machine, the same message can arrive through NSCA or NRDP, add-ons that were not run in the lab.

Other commands you will use: DISABLE_SVC_NOTIFICATIONS and ENABLE_SVC_NOTIFICATIONS (host;service), DISABLE_HOST_SVC_CHECKS, SCHEDULE_HOST_DOWNTIME, SCHEDULE_FORCED_HOST_CHECK (used for web2 above). The list is in the project documentation under "External Commands".

The pages are at http://SERVER/nagios4/. The useful ones: Tactical Overview (tac.cgi, one screen of counts), Hosts and Services (status.cgi), Problems, Service Detail, Reports, and the Process Info page with the global switches. Clicking a service gives the same commands as the previous section, as forms.

The packaged defaults are a compromise and the file says so: authentication is off (use_authentication=0 in cgi.cfg), and Apache only allows private address ranges (Require ip ::1/128 fc00::/7 ... 10.0.0.0/8 127.0.0.0/8 172.16.0.0/12 192.168.0.0/16). Anyone on your LAN can see everything, and by default only the page that submits commands (cmd.cgi) is protected by a login stanza, which in the packaged file ends in "Require all granted", i.e. open. Turn on real login before the server is reachable by anyone you do not trust.

1. Create a user (Apache digest authentication; the realm name must match: Nagios4):

$ sudo apt install apache2-utils
$ sudo htdigest -c /etc/nagios4/htdigest.users Nagios4 nagiosadmin
New password:
Re-type new password:
$ ls -l /etc/nagios4/htdigest.users
-rw-r----- 1 nagios www-data 53 Oct 14 15:29 /etc/nagios4/htdigest.users

Without -c the command adds a user to the existing file. (-c creates a new file, replacing the old one, so use it for the first user only.) What a user may see and do is set by user name in cgi.cfg: authorized_for_all_services=nagiosadmin, authorized_for_system_commands=nagiosadmin and so on (all of them name nagiosadmin by default), so the first user should keep that name or you edit those lines.

2. In /etc/nagios4/cgi.cfg:

use_authentication=1

3. In /etc/apache2/conf-available/nagios4-cgi.conf, replace the "Require ip ..." line in the DirectoryMatch block by the login stanza, and in the cmd.cgi block change "Require all granted" to "Require valid-user":

    AuthDigestDomain "Nagios4"
    AuthDigestProvider file
    AuthUserFile "/etc/nagios4/htdigest.users"
    AuthName "Nagios4"
    AuthType Digest
    Require valid-user

$ sudo apache2ctl configtest
Syntax OK
$ sudo systemctl reload apache2          (the lab used apache2ctl graceful)

Test it like a client does (HTTP status codes from the lab):

$ curl -s -o /dev/null -w "HTTP %{http_code}\n" http://localhost/nagios4/
HTTP 401
$ curl -s -o /dev/null -w "HTTP %{http_code}\n" --digest -u nagiosadmin:wrong http://localhost/nagios4/
HTTP 401
$ curl -s -o /dev/null -w "HTTP %{http_code}\n" --digest -u nagiosadmin:labpass http://localhost/nagios4/
HTTP 200
cmd.cgi without a login:  HTTP 401        with a login:  HTTP 200
tac.cgi without a login:  HTTP 401        with a login:  HTTP 200

The lab password was a throwaway. Digest authentication does not hide the traffic: put the pages behind HTTPS (a certificate on Apache, or a reverse proxy) if they cross a network you do not control. That setup was not run in the lab.

The same pages have a JSON interface, which is the easy way to read state from scripts (statusjson.cgi, with the same login):

$ curl -s --digest -u nagiosadmin:PASSWORD "http://localhost/cgi-bin/nagios4/statusjson.cgi?query=servicelist&details=true"
$ curl -s --digest -u nagiosadmin:PASSWORD "http://localhost/cgi-bin/nagios4/statusjson.cgi?query=service&hostname=web1&servicedescription=HTTP"

Each service has a numeric status: 1 pending, 2 ok, 4 warning, 8 unknown, 16 critical. For hosts: 1 pending, 2 up, 4 down, 8 unreachable. A small script turns that into a readable table. As used in the lab (host states and service states, SOFT or HARD with the attempt count):

HOST    localhost  UP          PING OK - Packet loss = 0%, RTA = 0.08 ms
HOST    web1       UP          PING OK - Packet loss = 0%, RTA = 0.13 ms
HOST    web2       DOWN        CRITICAL - Host Unreachable (172.21.0.99)
SERVICE web1       HTTP           OK        HARD 1/3  HTTP OK: HTTP/1.1 200 OK - 857 bytes in 0.001 second res
SERVICE web1       Nightly backup OK        HARD 1/1  BACKUP OK - 14.2 GB written in 312 s
SERVICE web1       Web response   PENDING   HARD 1/3  Service check scheduled for Wed Oct 14 15:30:46 UTC 2026

The log is plain text, one line per event, with the time as seconds since 1970 in square brackets:

[1791991604] SERVICE ALERT: web1;HTTP;CRITICAL;SOFT;1;connect to address 172.21.0.3 and port 80: Connection refused
[1791991665] SERVICE DOWNTIME ALERT: web1;HTTP;STARTED; Service has entered a period of scheduled downtime

date -d @1791991604 turns one into a readable time. The lab used a helper to print the last lines of events (alerts, notifications, external commands) with the times converted; a plain grep does the same:

$ grep -E "ALERT|NOTIFICATION|EXTERNAL COMMAND" /var/log/nagios4/nagios.log | tail

The log rotates daily by default (log_rotation_method=d) into /var/log/nagios4/archives. Every notification is logged (log_notifications=1), including who was told and with which command, which makes "did it really send anything?" a one-line answer. If a mail is missing but the NOTIFICATION line is there, the problem is mail delivery, not Nagios.

nagios4stats prints counters from the status file:

$ nagios4stats
Nagios Stats 4.4.6
CURRENT STATUS DATA
------------------------------------------------------
Status File:                            /var/lib/nagios4/status.dat
Status File Age:                        0d 0h 0m 2s
Status File Version:                    4.4.6

Program Running Time:                   0d 0h 0m 41s
Nagios PID:                             15205

Total Services:                         14
Services Checked:                       14
Services Scheduled:                     13
Services Actively Checked:              13
Services Passively Checked:             1

"Services Scheduled" counts checks Nagios runs by itself; the passive one is not among them. Further down it reports latency (how late checks start) and execution time, each as minimum / maximum / average:

Active Service Latency:                 0.000 / 0.001 / 0.000 sec
Active Service Execution Time:          0.003 / 4.124 / 0.770 sec
Passive Service Latency:                0.965 / 0.965 / 0.965 sec
Active Host Execution Time:             0.007 / 3.095 / 1.037 sec

If the latency keeps growing, the server cannot keep up: use fewer or slower checks, longer intervals, or look at max_concurrent_checks (0, the default, means unlimited).

State is kept across restarts: retain_state_information=1 writes it to retention.dat and the daemon reads it back at start.

Install             sudo apt install nagios4 apache2 monitoring-plugins nagios-nrpe-plugin   then sudo a2enmod cgi if pages download
Validate            nagios4 -v /etc/nagios4/nagios.cfg           ALWAYS before a reload; a bad reload stops the daemon
Reload / status     sudo systemctl reload nagios4   systemctl status nagios4
Your config         /etc/nagios4/conf.d/*.cfg     (host, service, hostgroup, servicegroup, command, contact, timeperiod)
Plugins             /usr/lib/nagios/plugins/check_*    run by hand, look at the exit code (echo $?)
Exit codes          0 OK   1 WARNING   2 CRITICAL   3 UNKNOWN
Thresholds          -w 10   -w 1:2   -w 1:   -w '~:3'   -w @1:10      (check_disk: FREE space)
Remote checks       agent: nagios-nrpe-server, allowed_hosts, nrpe_local.cfg     server: check_nrpe -H HOST -c NAME   (-H only to test)
States              OK WARNING CRITICAL UNKNOWN  /  host UP DOWN UNREACHABLE  /  SOFT then HARD after max_check_attempts
Command file        printf '[%lu] CMD;args\n' $(date +%s) > /var/lib/nagios4/rw/nagios.cmd
Force a check       SCHEDULE_FORCED_SVC_CHECK;host;service;time      SCHEDULE_FORCED_HOST_CHECK;host;time
Acknowledge         ACKNOWLEDGE_SVC_PROBLEM;host;service;sticky;notify;persistent;author;comment
Downtime            SCHEDULE_SVC_DOWNTIME;host;service;start;end;fixed;trigger;duration;author;comment
Passive result      PROCESS_SERVICE_CHECK_RESULT;host;service;code;output
Web login           sudo htdigest -c /etc/nagios4/htdigest.users Nagios4 USER   +  use_authentication=1 in cgi.cfg   +  Require valid-user in Apache
Log                 /var/log/nagios4/nagios.log      state: /var/lib/nagios4/status.dat      nagios4stats
Defaults to change  host max_check_attempts 10 and "workhours" in linux-server; nrpe check_load thresholds; the web login

Web pages download as files, or show binary text    mod_cgi is not enabled: sudo a2enmod cgi and restart Apache
Nothing is monitored after a reload              the reload hit a config error and Nagios bailed out. Read the end of /var/log/nagios4/nagios.log, run nagios4 -v, fix, start again
Error: Could not find any host matching 'X'      the host_name in a service is misspelt or the host is in a file that is not included (it must be under conf.d)
Template 'X' ... could not be found            the name after use does not exist: check the spelling and that the file defining it is included
Invalid ... directive 'addres'                   a typo. The errors after it are usually noise: fix the first
Service description contains illegal characters  parentheses and other symbols: use letters, digits, spaces, - and _
A check shows PENDING for a long time          checks are spread over the interval after a start; wait one interval, or force a check
UNKNOWN with "(No output returned from plugin)"  the plugin could not run: run the command line by hand as the nagios user (sudo -u nagios /usr/lib/nagios/plugins/check_X ...)
HTTP CRITICAL - Unable to open TCP socket      the service is down, or a firewall or wrong address; test with curl from the Nagios server
CHECK_NRPE: Could not connect ... reset by peer  the Nagios server is not in allowed_hosts of the agent, or port 5666 is blocked
NRPE: Command 'X' not defined                  no command[X]= line on the agent, or the agent was not restarted after editing
Receive header underflow ... 0 bytes received   you sent arguments (-a ...); agents refuse them by default. Define fixed commands instead
Quiet servers report WARNING for load          the NRPE default check_load thresholds are tiny: override in nrpe_local.cfg
A dead host takes ten minutes to show DOWN     max_check_attempts 10 and retry_interval 1 in the linux-server template
No mail but a NOTIFICATION line in the log     the notification command ran: mail delivery is the problem; test mail by hand as the nagios user
No alerts on weekends or at night              the template says workhours: set notification_period 24x7 on the host and service
A check never goes HARD                        max_check_attempts is too high for the interval, or the state is flapping between OK and a problem
Passive result ignored                         the service must have passive_checks_enabled 1 and the name must match exactly (host;service)
Pages say you are not authorized               the user is not listed in the authorized_for_* lines of cgi.cfg (all of them are nagiosadmin by default)

Run the plugin by hand first, as the nagios user, every time. If it fails there, Nagios cannot fix it.
Put your own files in conf.d and leave the shipped files alone. One file per host (or group of hosts) is easy to diff and to generate.
Validate before every reload. Make it a habit that is one command: nagios4 -v ... && reload.
Use templates and host groups. A new web server should be one host definition, with its services coming from the group.
Choose max_check_attempts and the intervals on purpose, per host class. The defaults are generic and slow.
Keep check_command arguments in the service, not in the command, so one command serves many services.
Prefer a few checks that matter (is the site up, is the disk filling, is the backup fresh) over hundreds that nobody reads. Alert fatigue is the real failure mode.
Use acknowledgements and downtime for real. A dashboard full of known problems hides the new one.
Passive checks are good for batch jobs: the "no news is bad news" freshness check catches the job that did not run.
Monitor the monitor: check Nagios from outside (a simple HTTP check from another place, or a dead man's switch that stops arriving), because a stopped Nagios sends no alerts about itself.
Protect the web interface (login, then HTTPS) and the NRPE port (firewall) before you give them an address anybody can reach.
Test notifications on purpose: stop a test service, watch SOFT, HARD and the mail, then bring it back and watch the RECOVERY.