Runbook: the disk is full
df -h; df -i; du -xh --max-depth=1 /var | sort -h | tail
"No space left on device" breaks things in odd ways: a database refuses to write, logins fail, package installs stop halfway, the web site shows errors, mail is lost. The fix is usually fast once you answer four questions in order: which filesystem is full, is it full of data or of inodes, what is using the space, and is some of that space held by a deleted file that a program still has open.
This runbook follows that path and then lists the safe ways to free space. The lab output is real: a tree of fake logs, caches and a database folder was built under a scratch directory, and the tools were pointed at it. I could not fill a real filesystem without root, so the "full" df displays are marked as illustrative. Commands that need root or would change the system are shown without output.
The five-minute path
df -h which mount is at or near 100%
df -i are the inodes used up instead?
sudo du -xh --max-depth=1 /var | sort -h | tail -8 the biggest folders on that filesystem (-x stays on one filesystem)
sudo find /var -xdev -type f -size +100M -printf '%s\t%p\n' | sort -rn | head the biggest files
sudo lsof +L1 deleted files still held open (space not returned)
then free space (Step 5), restart whatever failed, and put a prevention in place
Work on the filesystem that is full, not the whole machine: /, /var, /home, /tmp and a data disk can each fill by itself.
Step 1: which filesystem? df -h
$ df -h --output=source,size,used,avail,pcent,target /
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p2 915G 156G 713G 18% /
This is the healthy machine the lab ran on. A full one looks like this (illustrative, not from the lab):
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 40G 40G 0 100% /
/dev/sdb1 200G 81G 108G 43% /data
tmpfs 3.9G 3.9G 0 100% /dev/shm
Look at the Mounted on column to see WHICH filesystem is full: a full /dev/shm or /tmp (tmpfs, in memory) is cleaned differently from a full / (see below). On a full /, even root's shell can misbehave, so start by finding what you can delete safely (Step 5).
Two traps in the numbers: Use% is rounded (99% of a 10 TB disk is still 100 GB), and Avail is smaller than Size minus Used on ext4, because 5% is reserved for root (see "Reserved blocks").
Step 2: data or inodes? df -i
Every file uses one inode, and a filesystem has a fixed number of them, chosen when it was created. A disk with free space but no inodes gives exactly the same error:
$ echo x > /dev/full
bash: echo: write error: No space left on device
(/dev/full is a special device that always answers "full": it is a safe way to see the message.) The error text is the same for both cases, so check:
$ df -i --output=itotal,iused,iavail,ipcent /
Inodes IUsed IFree IUse%
60981248 1133920 59847328 2%
IUse% near 100% with plenty of space free means too many small files: session files, mail queue, cache entries, a runaway script that makes one file per run. Find which directory holds them. In the lab, 6000 tiny session files were created in a spool directory:
$ find var -xdev -type f | cut -d/ -f1-3 | sort | uniq -c | sort -rn | head -3
6000 var/spool/sessions
2 var/log/app
1 var/tmp/f9
$ for d in var/*/*; do echo "$(find "$d" -xdev | wc -l) $d"; done | sort -rn | head -3
6001 var/spool/sessions
3 var/log/app
2 var/lib/db
The first command counts files per second-level folder; the second counts per folder one level down. On a real server do it from the top of the full filesystem with sudo, one level at a time: for d in /var/*; do echo "$(sudo find "$d" -xdev 2>/dev/null | wc -l) $d"; done | sort -rn | head
The fix is to delete the files: find /var/spool/sessions -type f -mtime +7 -delete (rm * fails with "Argument list too long" on huge folders). Then stop whatever creates them. The inode count of an ext4 filesystem cannot grow later; XFS and Btrfs allocate inodes dynamically, so they hit the limit far less often. Here is how the number is fixed at creation, from a 32 MB ext4 image made in the lab with mkfs.ext4 -N 2048:
$ tune2fs -l t.img | grep -E 'Inode count|Block count|Reserved block count|Free blocks|Free inodes|Block size'
Inode count: 2048
Block count: 8192
Reserved block count: 409
Free blocks: 7027
Free inodes: 2037
Block size: 4096
Eleven inodes were already used by the empty filesystem itself. Note the "Reserved block count": 409 of 8192 blocks, the 5% for root.
Step 3: find what is big: du and find
du adds up sizes folder by folder. The flags to remember: -x (do not cross into other filesystems), -h (human sizes), --max-depth=1 (one level), and sort -h at the end. Start at the top of the full filesystem and go down into the biggest folder, one level at a time:
$ du -xh --max-depth=1 var | sort -h
592K var/tmp
2.9M var/lib
12M var/cache
47M var/log
62M var
var/log is the biggest. Go into it:
$ du -xh --max-depth=2 var | sort -rh | head -5
62M var
47M var/log/app
47M var/log
12M var/cache/pkg
12M var/cache
It is var/log/app. List its files by size:
$ ls -lSh var/log/app
total 47M
-rw-rw-r-- 1 demo demo 39M Oct 2 06:59 app.log
-rw-rw-r-- 1 demo demo 8.6M Oct 2 06:59 app.log.1
To go straight to the biggest files on a whole filesystem instead:
$ find var -xdev -type f -size +1M -printf '%s\t%p\n' | sort -rn | head -4 | awk '{printf "%.1fM\t%s\n",$1/1048576,$2}'
38.1M var/log/app/app.log
11.4M var/cache/pkg/big.deb
8.6M var/log/app/app.log.1
2.9M var/lib/db/data.ibd
On a real server run these with sudo (otherwise folders you cannot read are skipped silently, which gives a too-small answer), and use -xdev or -x every time: without it, du / also walks /proc, other disks and network mounts and takes forever. If ncdu is installed it is the fastest interactive way (ncdu -x /var).
Useful variations: find / -xdev -type f -mtime -1 -size +10M (big files created in the last day, which finds the thing that just broke), find /var/log -type f -name '*.gz' -mtime +30 (old rotated logs), and du -ah /var | sort -rh | head -20 (biggest files and folders together).
Step 4: df says full, du says it is not
You add up the folders and they come to far less than df reports. The common causes, in order of likelihood:
1. A deleted file that a program still has open. The name is gone, so du cannot see it, but the space is returned only when the last program closes it. Typical: a log file deleted while the service keeps writing to it. Reproduced in the lab with a 50 MB file kept open by a process, then deleted:
$ du -sh var/log/app
95M var/log/app
$ rm var/log/app/held.log
$ du -sh var/log/app
47M var/log/app
du dropped by 50 MB, but on a real filesystem df would not change, because the open process still holds the space. Find these files:
$ lsof +L1
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
sleep 10360 demo 0r REG 259,2 50000000 0 19011675 /home/demo/lab/var/log/app/held.log (deleted)
+L1 lists open files with a link count below 1, that is, deleted ones; SIZE/OFF is their size. The same information is in /proc (the lab experiment was run twice, which is why the process numbers differ between these listings):
$ ls -l /proc/10824/fd/0
lr-x------ 1 demo demo 64 ... /proc/10824/fd/0 -> /home/demo/lab/held.log (deleted)
Two ways to get the space back. The clean one is to restart the program (systemctl restart SERVICE), which closes the file. When it cannot be restarted, truncate the file through its /proc entry: the program keeps writing, the space comes back. The check below uses stat -L, which follows the link and reports the size of the real file:
$ stat -L -c %s /proc/10824/fd/0
50000000
$ : > /proc/10824/fd/0
$ stat -L -c %s /proc/10824/fd/0
0
Same effect, no restart. (Take care to use the right PID and descriptor number: it empties whatever that descriptor points to.) To avoid the problem in the first place, empty a live log with truncate -s 0 FILE or : > FILE instead of rm: the file stays, the writer keeps its handle, and the space is freed at once.
2. Files hidden under a mount point. If a disk was mounted over a folder that already had files in it, those files still occupy the lower filesystem but are invisible. Look at the root filesystem without the mounts on top using a bind mount (sudo mount --bind / /mnt/rootview; sudo du -xh --max-depth=1 /mnt/rootview | sort -h | tail; sudo umount /mnt/rootview).
3. Reserved blocks and filesystem overhead: ext4 sets aside 5% for root, plus the journal and metadata. It explains a few percent, not half the disk.
4. Snapshots and thin provisioning: LVM snapshots, Btrfs snapshots, ZFS snapshots and Docker layers hold space for data that is no longer visible in the normal tree. See btrfs filesystem du, zfs list -t snapshot and lvs.
5. Sparse files and mounted images look larger or smaller than they are: ls -ls shows the real blocks used next to the apparent size.
6. A filesystem that needs a check after a crash: df and du can disagree until e2fsck runs on the unmounted filesystem. In the lab, on the 32 MB image:
$ e2fsck -fn t.img
Pass 5: Checking group summary information
t.img: 11/2048 files (9.1% non-contiguous), 1165/8192 blocks
("-n" only reads and answers "no" to every repair question, which makes it safe to try. A clean run ends with this summary line; never run a repairing e2fsck on a mounted filesystem.)
Step 5: free space safely
Order matters. Start with things that are certainly safe, check before each deletion, and keep a note of what you removed.
Safe and fast:
sudo journalctl --vacuum-size=200M shrink the systemd journal (see the journalctl page)
sudo apt clean the downloaded package cache (dnf clean all, yum clean all)
sudo apt autoremove --purge old kernels and unused packages (check the list first; keep the running kernel and one older one: uname -r)
docker system prune stopped containers, unused networks, dangling images (add -a for all unused images, and --volumes only on purpose)
sudo find /var/log -name '*.gz' -mtime +30 -delete old compressed logs (or move them to another disk first)
rm -rf ~/.cache/* caches rebuild themselves
sudo find /var/crash -type f -delete; sudo coredumpctl list old crash dumps and core files
sudo snap list --all | awk '/disabled/{print $1, $3}' old snap revisions, removed with: sudo snap remove NAME --revision=N
Needs a decision:
large log files that are still in use truncate, do not delete (truncate -s 0 FILE), and fix logrotate so it does not happen again
old backups, ISO images and downloads move to another disk or delete once you are sure
database dumps, binary logs, WAL, replicated data use the database's own command (PURGE BINARY LOGS in MySQL), never rm in the data folder
Docker volumes and container logs docker system df shows where the space is; limit container logs (max-size) so it does not return
Real output of docker system df on the lab machine, to show what to look for in the RECLAIMABLE column:
$ docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 20 9 24.28GB 3.202GB (13%)
Containers 12 2 111.4MB 110.3MB (99%)
Local Volumes 9 8 3.333GB 48.47MB (1%)
Build Cache 33 0 374.7MB 102.2MB
Never delete files you have not looked at. Before an rm -rf on a path built from a variable, print it first (echo rm -rf "$DIR"); an empty variable is how a / gets wiped.
When even that is not enough, give the filesystem more room: grow the LVM volume (sudo lvextend -r -L +10G /dev/VG/LV, the -r also resizes the filesystem), enlarge the cloud disk and run growpart and resize2fs (ext4) or xfs_growfs (XFS), or move the big folder to another disk with rsync and a bind mount or a symlink.
When the disk is 100% and nothing will run: the reserved 5% is for root, so root can often still delete. If even deleting fails, free a few MB first with truncate -s 0 on an old log, then continue.
After freeing space, restart what failed (a database that stopped on write errors may need a restart or a repair) and check logs from the full period for corrupted or half-written files.
Common space hogs and their cleanup commands
/var/log logs without rotation; check logrotate (cat /etc/logrotate.d/*) and compress with gzip; see the journalctl page for the journal
/var/lib/docker images, containers, build cache, volumes; docker system df and docker builder prune
/var/lib/mysql, /var/lib/postgresql data, binary logs, WAL archives; clean through the database's own tools, not rm
/var/cache/apt, /var/cache/dnf package downloads; apt clean
/var/crash, /var/lib/systemd/coredump crash reports and core dumps; coredumpctl; limit with Storage= in coredump.conf
/var/spool (mail, cups, cron) a stuck mail queue (postqueue -p | tail -1); thousands of files here also eat inodes
/tmp, /var/tmp, /dev/shm temporary files, often tmpfs (in memory); files here are removed by the owning programs, not at random
/home downloads, trash (~/.local/share/Trash), VM images, browser caches
/snap and /var/lib/snapd old revisions of every snap package
/boot old kernels fill a small boot partition: sudo apt autoremove --purge, then ls /boot (several config-*, initrd.img-* and vmlinuz-* files per kernel)
swap files (/swap.img) and hibernation size of RAM or more: normal, and not something to delete while swapon is active
A tmpfs full of files is memory, not disk: free it by removing the files, and look for a program writing there on purpose.
Reserved blocks, and why root can write at "100%"
ext4 reserves a share of blocks for root (5% by default) so a full filesystem does not lock the system out and so fragmentation stays low. On a data disk that only holds files of ordinary users, 5% of a 4 TB disk is 200 GB doing nothing. See the current value and lower it:
$ sudo tune2fs -l /dev/sdb1 | grep -i 'reserved block'
$ sudo tune2fs -m 1 /dev/sdb1 # reserve 1%
Do not lower it on / or /var of a production system: that reserve is what lets root log in and fix things when everything else is full. XFS and Btrfs handle this differently (XFS keeps a small amount free automatically; Btrfs needs a balance when it runs out of chunks, not just space: btrfs filesystem usage /).
When the full disk is your own mistake: the full / or full /var
If / is full, the machine may not even accept SSH logins (it cannot write a session file), systemd cannot start services, and package tools fail. Use the console or an existing session, then:
1. Truncate the biggest known log (find it with the du commands above): sudo truncate -s 0 /var/log/BIGFILE
2. sudo journalctl --vacuum-size=100M
3. sudo apt clean
4. Check that you can write again (touch /testfile && rm /testfile), then continue with the normal path.
Make this impossible next time: put /var (or at least /var/log) on its own filesystem, so logs cannot take / down.
Prevention
logrotate on every log you create: size or time based, compressed, with a limit on the number of old files (see /etc/logrotate.d)
set SystemMaxUse= for the journal and max-size for Docker logs
alert at 80% used and again at 90% (inodes too), with a script or your monitoring: df -P | awk 'NR>1 && $5+0 > 80 {print $6, $5}'
put data that grows (logs, databases, uploads, backups) on its own volume that can be extended
keep a "ballast" file, for example fallocate -l 2G /ballast on the root filesystem: delete it in an emergency to get breathing room, then fix the real problem
clean temporary files automatically (systemd-tmpfiles, tmpwatch) and set cron jobs with output redirected, so a failing job does not mail itself to death in /var/mail
look at trends, not just thresholds: a disk growing by 2 GB per day with 20 GB left is a problem in ten days, not today
Decision table: symptom to cause
"No space left on device" but df shows free space inodes: df -i, then find the folder with millions of small files
"No space left on device" on a tmpfs or /dev/shm the in-memory filesystem is full: remove files, or look for a program filling it
df 100%, du much smaller a deleted file still open (lsof +L1), files under a mount point, or a snapshot
Space disappears after deleting a log the service still holds the file: restart it, or truncate through /proc/PID/fd/N
Disk fills a little more every hour a log or a queue without rotation: du twice, an hour apart, and compare
Root can write, normal users cannot the reserved 5% is all that is left: free space now
Package install fails halfway /var or /boot full: apt clean, apt autoremove --purge, check /boot
Docker "no space left on device" while df is fine docker's own storage: docker system df, and the /var/lib/docker filesystem
Database stopped with write errors data or binary logs filled the disk: free space first, then start it and check integrity
After deleting, space did not come back (Btrfs/ZFS) snapshots keep the old blocks: list and delete snapshots
Disk looks fine for a user but not for a service a per-user quota or a container limit: quota -u USER, repquota -a
Tips
Use -x or -xdev on every du and find, so you stay on the filesystem you are looking at.
Check inodes with df -i whenever the error and df -h disagree.
lsof +L1 solves most "df and du do not match" mysteries in one command.
Truncate a live log; never rm it. truncate -s 0 FILE gives the space back at once and keeps the writer happy.
Look before you delete: print the path, check the size and the age, and keep a list of what you removed.
After you fix it, fix the cause (rotation, limits, a bigger volume), or you will be back here in a week.
