🐌 Runbook: High Load / Slow Server

Runbook: the server is slow or the load is high
uptime; vmstat 1 5; ps -eo pid,stat,pcpu,pmem,comm --sort=-pcpu | head

A runbook is a checklist you can follow at 3 a.m. when the pager goes off and thinking is expensive. This one starts at the symptom ("the server is slow", "load average is 25", "the site times out") and walks one fixed path: is it CPU, memory, disk or something stuck, then WHO is doing it, then what to do about it. The aim is a verdict in two minutes, before you restart anything.
The numbers below are real. They were taken on an 8-core Ubuntu 24.04 machine (31 GB RAM, 8 GB swap file) while a lab workload ran: three endless busy loops at the lowest priority (nice 19) plus one process writing 300 MB files to disk with direct I/O, for about 30 seconds. Other processes on the machine are trimmed from the listings. Commands that change the system are shown without output.

Run these in order and write down what you see. Do not restart anything until you know which resource is short.
  uptime                          load averages (1, 5, 15 minutes) and how long it has been up
  nproc                           number of CPUs, to put the load in proportion
  vmstat 1 5                      CPU split, memory, swap, run queue, blocked processes
  top -b -n1 | head -15           the busiest processes right now
  ps -eo pid,ppid,stat,ni,pcpu,pmem,etimes,comm --sort=-pcpu | head     busiest by CPU (use --sort=-pmem for memory)
  free -h; swapon --show          memory and swap
  iostat -x 1 3                   disk utilisation and latency
  df -h; df -i                    full disks and inodes also make a server "slow" (see the Disk Full runbook)
  dmesg -T | tail -30             kernel messages: OOM kills, disk errors, hardware trouble (sudo on many systems)
  journalctl -p err -b --since "1 hour ago" --no-pager | tail -20      recent errors
That is the whole method. The rest of this page explains what each output means.

$ nproc
8
$ uptime
 06:58:54 up  1:06,  1 user,  load average: 0.36, 0.39, 0.45
The three numbers are the average number of tasks that were running or waiting to run (and, on Linux, also those waiting for disk) over the last 1, 5 and 15 minutes. Compare them with the number of CPUs: a load of 8 on an 8-core machine means every core is busy, a load of 24 means three tasks fight for each core. A load of 1 on 8 cores is idle.
Read the three numbers as a trend: 20, 8, 2 means the problem is recent and getting worse; 2, 8, 20 means the worst is over; all three high means a long problem.
It is slow to react. Twelve seconds after four busy tasks had started on this 8-core machine:
$ uptime
 06:59:06 up  1:06,  1 user,  load average: 1.00, 0.53, 0.49
The 1-minute value was still 1.00, a quarter of the real number of runnable tasks, because it is a smoothed average. Ten seconds after the workload ended the figure had only reached about double that, and it was still catching up with what had happened:
$ uptime
 06:59:24 up  1:07,  1 user,  load average: 2.15, 0.82, 0.59
So load average confirms a problem; it does not tell you what is happening right now. For "right now" use vmstat, top and PSI. And a high load does not always mean a busy CPU: processes blocked on slow disk or NFS count too, so a machine with 5% CPU and a load of 40 is waiting for I/O.

$ vmstat 1 4
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
 3  1      0 25350180 161972 3984384    0    0   782  5178 2685    3  3  2 96  0  0  0
 3  1      0 25348912 162032 3984652    0    0     0 1649604 20080 5485 39  6 48  6  0  0
 3  1      0 25350956 162104 3984644    0    0     0 1626240 19775 5255 40  6 49  6  0  0
 4  0      0 25351096 162164 3984684    0    0     0 1637312 19914 5370 39  5 49  6  0  0
The first line is the average since boot and is not useful: read the lines after it. The columns:
  r      tasks running or waiting for a CPU. If r stays above the number of CPUs, the CPU is saturated
  b      tasks blocked in uninterruptible sleep, usually waiting for disk. A steady b above 0 points to I/O
  swpd, si, so   swap in use, and pages swapped in (si) and out (so) per second. Nonzero si and so over time means the machine is out of memory
  bi, bo   blocks read and written per second (here 1.6 million written per second by the lab writer)
  us, sy   CPU in user programs and in the kernel. High sy with low us often means heavy I/O or system calls
  id       idle. wa is time waiting for I/O, st is time stolen by the hypervisor (a noisy neighbour on a VM)
Here r is 3 or 4 (the busy loops and the writer) on 8 CPUs, us is 40% (about 3 of 8 cores) and wa is 6%: busy but not saturated.
top shows the same split on the %Cpu line:
$ top -b -n1 -w 120 | sed -n '1,5p'
top - 06:59:06 up  1:06,  1 user,  load average: 1.00, 0.53, 0.49
Tasks: 326 total,   5 running, 321 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.0 us,  6.1 sy, 38.8 ni, 46.9 id,  7.1 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  31778.5 total,  24757.1 free,   3284.7 used,   4048.9 buff/cache
MiB Swap:   8192.0 total,   8192.0 free,      0.0 used.  28493.8 avail Mem
Note "ni": the busy loops ran at nice 19, so their CPU time shows in the nice column, not in us. The number to read for memory is "avail Mem", not "free" (buff/cache is reclaimable). In interactive top, press 1 for one line per CPU, M to sort by memory, P by CPU, and c for the full command line.

$ ps -eo pid,ppid,stat,ni,pcpu,pmem,etimes,comm --sort=-pcpu | head -5
    PID    PPID STAT  NI %CPU %MEM ELAPSED COMMAND
  10135   10130 RN    19  100  0.0      12 sh
  10136   10131 RN    19  100  0.0      12 sh
  10137   10129 RN    19  100  0.0      12 sh
  10209   10134 R      0 50.0  0.0       0 dd
Three loops at 100% each (one core each) and a dd at 50%. STAT tells you the state: R running, S sleeping, D uninterruptible wait (disk), Z zombie, T stopped; the extra letter N means low priority. ELAPSED is the age in seconds: a process that is 12 seconds old and eats a core points to something that just started (a cron job, a deploy, a runaway script); one that has run for weeks is a different story.
Find where a process came from and what it is doing:
  ps -fp PID                       its full command line and parent
  pstree -ps PID                   the chain of parents (cron? a shell? a service?)
  systemctl status PID             the service it belongs to
  ls -l /proc/PID/cwd /proc/PID/exe        its working directory and program
  sudo lsof -p PID | head -30      the files and sockets it has open
  sudo strace -p PID -f -c         which system calls it makes (press Ctrl+C for the summary); see the strace page
For a per-process view over time, pidstat samples CPU per process (and -d for disk, -r for memory, -w for context switches):
$ pidstat 1 2
Average:      UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
Average:        0       282    0.00    0.49    0.00    0.00    0.49     -  jbd2/nvme0n1p2-8
(other processes trimmed; the column %wait is time spent waiting for a CPU, which is the sign of too many runnable tasks)

$ iostat -x -d nvme0n1 1 2
Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz ...
nvme0n1          0.00      0.00     0.00   0.00    0.00     0.00 13008.00 1664064.00   225.00   1.70    0.24   127.93 ...
The important columns: r/s and w/s (operations per second), rkB/s and wkB/s (throughput), r_await and w_await (average milliseconds per request, the latency), and %util at the far right of the line. Rules of thumb: on a spinning disk, await above 20 ms and %util near 100 means the disk is the bottleneck; a fast NVMe like this one wrote 1.6 GB per second with 0.24 ms latency and was not struggling. Always judge by latency and queue, not by throughput alone.
Who is writing? Without iotop (often not installed), use pidstat -d 1 or see which processes sit in D state (Step 6). On a database server, look at the slow-query log before blaming the disk.
Two different disk problems feel the same: a disk that is too slow (high await) and a disk that is full (see the Disk Full runbook). On a VM or cloud instance, also check for a throttled volume: burst credits exhausted look exactly like a slow disk.

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            31Gi       3.2Gi        24Gi        88Mi       4.0Gi        27Gi
Swap:          8.0Gi          0B       8.0Gi
$ swapon --show
NAME      TYPE SIZE USED PRIO
/swap.img file   8G   0B   -1
Read "available", not "free": Linux uses spare RAM for the file cache (buff/cache) and gives it back when a program needs it. Trouble looks like: available close to zero, swap in use and growing, vmstat showing steady si and so, and the load average climbing while CPU use is low (everything waits for swap).
When memory runs out completely, the kernel's OOM killer picks a process and kills it, which is often the database or the application you care about. The evidence is in the kernel log:
$ sudo dmesg -T | grep -i -E 'out of memory|oom-kill|killed process'
$ sudo journalctl -k -g 'Out of memory' --since "1 day ago"
Typical lines (illustrative, from a server that ran out of memory): "Out of memory: Killed process 2412 (mysqld) total-vm:6841240kB, anon-rss:5120224kB". The name and the size tell you who was killed and how big it was; the real culprit may be another process that grew until the killer had to act. Look at the process list in the OOM report, and at ps --sort=-pmem.
A service that dies "for no reason" and restarts is often an OOM victim: check the log before blaming the application.

D means uninterruptible sleep: the process is inside the kernel waiting for something (almost always I/O) and cannot even be killed until that completes. A few for a moment are normal; many that stay are the problem. List them with what they wait on:
$ ps -eo pid,stat,wchan:20,comm | awk '$2 ~ /^D/'
  10255 D    blk_io_schedule      dd
This lab process was in D state because it was waiting for the block layer (blk_io_schedule) to finish a write. On a real incident, the wchan column is the clue: nfs or rpc wording means a hung network filesystem (check the NFS server or the network, and umount -f -l as a last resort), jbd2 or ext4 means journal or disk pressure, a device name means a failing disk (check dmesg and smartctl -a /dev/DEVICE).
A D-state process cannot be fixed by kill -9: the signal waits until the I/O returns. Solve the thing it waits for.

Pressure Stall Information (kernel 4.20 and newer, enabled on Ubuntu) tells you directly what share of the last 10, 60 and 300 seconds tasks were stalled waiting for a resource, which is more honest than load average:
$ cat /proc/pressure/cpu /proc/pressure/io
some avg10=0.09 avg60=0.02 avg300=0.00 total=16742512
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
some avg10=4.67 avg60=1.61 avg300=0.54 total=10527867
full avg10=4.67 avg60=1.60 avg300=0.53 total=9815531
The first two lines are CPU, the last two I/O (taken during the lab). "some" is the percent of time at least one task was stalled; "full" is the percent of time every task was stalled. Under the I/O load here, some and full were both 4.67%: the machine spent that share of the last 10 seconds doing nothing but waiting for disk. Single digits are fine; tens of percent are a real problem. /proc/pressure/memory exists too and is the best early warning before an OOM kill.

Start with the actions that are easy to undo and work outward:
  1. Find the process before you kill anything, and decide whether it is supposed to be running (a backup, an index rebuild, a legitimate traffic spike).
  2. Lower its priority instead of killing it: sudo renice +10 -p PID (CPU), sudo ionice -c3 -p PID (disk, idle class only).
  3. Pause it and resume later: kill -STOP PID ... kill -CONT PID. Memory stays in use, but the CPU is free at once.
  4. Ask it to stop cleanly: systemctl stop SERVICE, or kill PID (SIGTERM). Wait 10 to 30 seconds.
  5. Only then kill -9 PID. It cannot clean up: lock files, temporary files and half-written data stay behind.
  6. For a runaway cron job, check the crontab for an overlapping schedule (see the cron page, flock).
  7. For a web server flooded by requests, look at the access log, find the top clients (awk '{print $1}' access.log | sort | uniq -c | sort -rn | head) and block or rate-limit them (see the nginx and firewall pages).
  8. For memory: restart the leaking service on a schedule while the bug is fixed, add swap as a stopgap (an 8 GB swap file: fallocate -l 8G /swap2; chmod 600 /swap2; mkswap /swap2; swapon /swap2), and set a memory limit in the unit (MemoryMax=) so one service cannot take the machine down.
  9. For disk: stop the heavy writer, free space, and only then consider faster storage.
  10. If the machine does not answer at all: wait; try a second SSH session (sshd may be slow, not dead); use the console. A hard reboot is the last resort, and a journal that survives (persistent storage) will tell you what happened afterwards.
What NOT to do: do not run a heavy diagnostic (find /, du /, a full backup) on a machine that is already struggling; do not clear caches (echo 3 > /proc/sys/vm/drop_caches) as a fix, it only makes the next minute slower; do not reboot before you have saved ps, vmstat and dmesg output, because a reboot erases the evidence.

High load, high us, one process at the top      a CPU-bound process: runaway script, busy loop, a cryptominer, a heavy query. ps, then renice or stop it
High load, low CPU use, high wa, D-state tasks   I/O wait: slow or failing disk, full disk, a hung NFS mount, a backup running. iostat -x, ps for D state, dmesg
High load, low CPU, but free memory is near zero and swap is active    memory pressure: swapping. free -h, vmstat si/so, find the biggest process, look for OOM kills
Load is huge, but everything is idle and fast   many short-lived processes or a fork bomb: ps -e | wc -l, a runaway cron or a loop spawning children
r high, us low, sy high                         a lot of kernel work: many system calls, lock contention, network storms. perf top or strace -c
st above a few percent in vmstat or top         steal: the hypervisor gives your CPU to others. Ask the provider or move the VM
Load spikes every N minutes exactly             a scheduled job: crontab -l, /etc/cron.d, systemctl list-timers
Slow only for one user or one URL               an application problem, not the server: check that app's logs and its database queries
Slow only on first access, fast afterwards      cold caches or DNS: time the steps with curl -w and dig
Cannot log in, existing sessions work           the machine is out of memory, PIDs or inodes, or the disk is full: free what you can from the open session
Load average is high but nothing looks busy     processes blocked on a network filesystem or a dead mount: ps D-state wchan, mount | grep nfs
After a reboot the problem returns at the same time    something starts at boot or on a timer: systemd-analyze blame, systemctl list-timers

Save the evidence while it is fresh: the vmstat, ps and dmesg output, the time it began and ended, what you changed. Check the journal for the same window (journalctl --since "..." --until "..." -p warning). If it will happen again, install sysstat (sar keeps history, so you can ask "what was the load at 03:10 last night?" with sar -q and sar -u) and add an alert on load per CPU, PSI and the available memory. Write the cause down in one sentence; that sentence is the next runbook.

Put the load in proportion: divide by the number of CPUs, and read the 1, 5 and 15 minute figures as a trend.
Name the resource first (CPU, memory, disk, waiting), then the process. vmstat 1 5 does the first step in five seconds.
The first vmstat line is an average since boot; ignore it.
Read "available" memory, never "free", and "await" and PSI for disks, not throughput.
Slow down before you speed up: renice, ionice and kill -STOP are reversible; kill -9 and reboot are not.
Do not kill the evidence: capture the output first, then fix, and write down the cause.
Keep history (sysstat/sar) so the next incident starts with data instead of guesses.