Disks, Partitions, LVM and Filesystems
lsblk -f; df -hT; findmnt
From a blank disk to a mounted folder there are layers, and every "disk problem" lives in exactly one of them: the device, the partition table, an optional volume layer (LVM, RAID, encryption), the filesystem, or the mount. This guide walks the layers, shows how to look without touching anything, how to practice safely on an image file, how to grow storage with LVM, and how to find out where "disk full" really comes from.
Rule one: destructive commands (mkfs, wipefs, dd, sfdisk, parted, pvcreate) erase whatever is on the target. Identify the device TWICE, by size, model and serial, before you press Enter. Everything that is marked as run below happened on a plain file, never on a real disk.
The layers
disk /dev/sda, /dev/nvme0n1, /dev/vda (a virtual disk)
partition table GPT (modern, any size) or MBR (old, 2 TiB limit, 4 primary partitions)
partition /dev/sda1, /dev/nvme0n1p1, /dev/vda1
[LVM / RAID / LUKS] optional: /dev/mapper/vg0-root, /dev/md0, /dev/mapper/cryptdata
filesystem ext4, xfs, btrfs, vfat ... (mkfs makes one)
mount a folder where it shows up (/, /home, /var, /data)
Names like /dev/sdb can CHANGE between boots when disks are added or discovered in a different order. Never put them in /etc/fstab or scripts. Use stable names:
/dev/disk/by-uuid/... the filesystem's UUID (best for fstab) /dev/disk/by-label/... a label you chose
/dev/disk/by-id/... model plus serial number of the disk /dev/disk/by-path/... the controller port
$ ls -l /dev/disk/by-id/ | head
$ lsblk -o NAME,SIZE,TYPE,FSTYPE,LABEL,UUID,MOUNTPOINT,MODEL,SERIAL
Look without touching (all read-only)
$ lsblk -f # tree of disks, partitions, filesystem type, label, UUID, where it is mounted
$ lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,MODEL
$ sudo blkid # UUIDs and filesystem types of everything (needs root for some devices)
$ findmnt # all mounts as a tree; findmnt /var what is mounted there? findmnt -T /var/log/syslog which mount holds this file?
$ df -hT -x tmpfs -x devtmpfs # space used and free per filesystem; df -i shows INODES (see "disk full" below)
$ sudo fdisk -l /dev/sdb # partition table of one disk; sudo parted -l does all disks
$ cat /proc/partitions; cat /proc/mounts
A sample lsblk -f output (typed in as an illustration) for a typical server with LVM, and how to read it:
$ cat <<'EOF'
> NAME FSTYPE LABEL UUID MOUNTPOINT
> sda
> ├─sda1 vfat 1A2B-3C4D /boot/efi
> ├─sda2 ext4 0f4b... /boot
> └─sda3 LVM2_member Xk2P...
> ├─vg0-root ext4 7a31... /
> └─vg0-var xfs c9d8... /var
> sdb
> └─sdb1 ext4 data 55ee...
> EOF
NAME FSTYPE LABEL UUID MOUNTPOINT
sda
├─sda1 vfat 1A2B-3C4D /boot/efi
├─sda2 ext4 0f4b... /boot
└─sda3 LVM2_member Xk2P...
├─vg0-root ext4 7a31... /
└─vg0-var xfs c9d8... /var
sdb
└─sdb1 ext4 data 55ee...
sda3 is an LVM physical volume ("LVM2_member"); vg0-root and vg0-var are logical volumes carved out of it. sdb1 holds a filesystem labelled "data" that is NOT mounted (empty MOUNTPOINT): the disk is attached but nobody mounted it.
Practice on an image file (safe: no real disk is touched)
A file can be partitioned, formatted, checked and resized like a disk. Everything in this section was run for real on files.
$ truncate -s 64M disk.img; ls -lh disk.img | awk '{print $5, $9}'
64M disk.img
Partition it with sfdisk (scriptable: the layout is plain text), GPT, 1 MiB aligned (start sector 2048):
$ printf 'label: gpt\nunit: sectors\n\nstart=2048, size=65536, type=L, name="data"\nstart=67584, size=32768, type=S, name="swap"\n' | sfdisk --quiet disk.img
$ fdisk -l disk.img | grep -vE 'Disk identifier|^$|Sector size|I/O size|Units'
Disk disk.img: 64 MiB, 67108864 bytes, 131072 sectors
Disklabel type: gpt
Device Start End Sectors Size Type
disk.img1 2048 67583 65536 32M Linux filesystem
disk.img2 67584 100351 32768 16M Linux swap
$ parted -s disk.img unit MiB print | grep -vE '^$|Disk Flags|Model|Sector size'
Disk /home/demo/disk.img: 64.0MiB
Partition Table: gpt
Number Start End Size File system Name Flags
1 1.00MiB 33.0MiB 32.0MiB data
2 33.0MiB 49.0MiB 16.0MiB swap swap
$ sfdisk -d disk.img | sed -E 's/^label-id: .*/label-id: <random>/; s/uuid=[0-9A-F-]+/uuid=<random>/'
label: gpt
label-id: <random>
device: disk.img
unit: sectors
first-lba: 2048
last-lba: 131038
sector-size: 512
disk.img1 : start= 2048, size= 65536, type=0FC63DAF-8483-4772-8E79-3D69D8477DE4, uuid=<random>, name="data"
disk.img2 : start= 67584, size= 32768, type=0657FD6D-A4AB-43C4-84E5-0933C84B4F4F, uuid=<random>, name="swap"
sfdisk -d dumps a layout you can save and replay onto another disk (sfdisk /dev/sdc < layout.txt): handy for identical disks in a mirror. Interactive tools: fdisk (MBR and GPT; keys: p print, n new, d delete, t type, w WRITE and exit, q quit without saving), gdisk (GPT), parted (scriptable, can resize).
Make a filesystem and inspect it
$ truncate -s 32M fs.img; mkfs.ext4 -q -F -L data fs.img; echo "mkfs exit status $?"
mkfs exit status 0
$ blkid -s TYPE -o value fs.img; blkid -s LABEL -o value fs.img
ext4
data
$ tune2fs -l fs.img | grep -E 'Filesystem volume name|Filesystem features|Filesystem state|Block size|Block count|Reserved block count|Inode count'
Filesystem volume name: data
Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem state: clean
Inode count: 8192
Block count: 8192
Reserved block count: 409
Block size: 4096
$ e2fsck -fn fs.img | tail -1
e2fsck 1.47.0 (5-Feb-2023)
data: 11/8192 files (9.1% non-contiguous), 1549/8192 blocks
e2fsck -fn is a read-only check ("-n" answers no to every repair question). Only run a repairing fsck (-y or -p) on an UNMOUNTED filesystem.
Dry-run a wipe before erasing signatures: wipefs without options only LISTS what it would remove
$ wipefs -n fs.img | awk '{print $1, $2, $3, $5}'
DEVICE OFFSET TYPE LABEL
fs.img 0x438 ext4 data
Grow the filesystem: first the container (here the file), then the filesystem
$ truncate -s 96M fs.img; e2fsck -fy fs.img >/dev/null; resize2fs fs.img 2>&1 | tail -1
e2fsck 1.47.0 (5-Feb-2023)
$ tune2fs -l fs.img | grep 'Block count'
Block count: 24576
$ sudo mount -o loop fs.img /mnt && ls /mnt && sudo umount /mnt # mounting a file needs root
$ sudo losetup -fP --show disk.img # attach a whole disk image as /dev/loopN with partitions /dev/loopNp1 ...; sudo losetup -d /dev/loop0 detaches it
Copy raw data exactly (and prove it): dd on files
$ dd if=/dev/zero of=blank.img bs=1M count=4 status=none; ls -l blank.img | awk '{print $5, "bytes"}'
4194304 bytes
$ head -c 1000000 /dev/urandom > source.bin; dd if=source.bin of=copy.bin bs=64K status=none; cmp source.bin copy.bin && echo "copy is identical"; [ "$(sha256sum < source.bin)" = "$(sha256sum < copy.bin)" ] && echo "checksums match"
copy is identical
checksums match
Choosing and maintaining a filesystem
Filesystem Good for Grow online Shrink Notes
ext4 the safe default yes yes, UNMOUNTED mature, simple, fast fsck, journalled
xfs big files, big servers, RHEL yes NO (never) xfs_growfs to grow, xfs_repair to repair; cannot be made smaller
btrfs snapshots, checksums, subvolumes yes yes compression and snapshots built in; keep it balanced
vfat/exfat USB sticks, EFI system partition no no works everywhere, no permissions
$ sudo mkfs.ext4 -L data /dev/sdb1 # make it (ERASES the partition)
$ sudo mkfs.xfs -L data /dev/sdb1 # xfs: add -f to overwrite an old signature
$ sudo tune2fs -L newlabel /dev/sdb1 # change the label of ext4; xfs_admin -L newlabel /dev/sdb1 for xfs
$ sudo tune2fs -m 1 /dev/sdb1 # reserve 1% for root instead of the default 5%: a big win on a large data disk
$ sudo e2fsck -f /dev/sdb1 # check ext4 (UNMOUNTED); sudo xfs_repair /dev/sdb1 for xfs (UNMOUNTED)
$ sudo resize2fs /dev/vg0/data # grow ext4 to fill its container (works mounted); sudo xfs_growfs /data for xfs (give it the MOUNT POINT)
$ sudo fstrim -av # tell an SSD or thin-provisioned disk which blocks are free; sudo systemctl enable --now fstrim.timer runs it weekly
Mount options worth knowing: noatime (do not update the access time on every read: faster), nodev,nosuid,noexec (hardening on /tmp or data disks), ro, discard (online trim: usually the fstrim timer is better), nofail (do not stop the boot if the disk is missing).
fstab and mounting
/etc/fstab lists what to mount at boot. Six fields: what, where, type, options, dump (0), fsck order (1 for /, 2 for others, 0 for none).
$ cat <<'EOF'
> # <file system> <mount point> <type> <options> <dump> <pass>
> UUID=1A2B-3C4D /boot/efi vfat umask=0077 0 1
> /dev/mapper/vg0-root / ext4 defaults,noatime 0 1
> UUID=55ee0c1a-0000-4000-8000-000000000001 /data ext4 defaults,noatime,nofail 0 2
> nas.lan:/export/media /mnt/media nfs _netdev,noauto,x-systemd.automount,x-systemd.idle-timeout=60 0 0
> tmpfs /tmp tmpfs defaults,nosuid,nodev,size=2G 0 0
> /swapfile none swap sw 0 0
> EOF
# <file system> <mount point> <type> <options> <dump> <pass>
UUID=1A2B-3C4D /boot/efi vfat umask=0077 0 1
/dev/mapper/vg0-root / ext4 defaults,noatime 0 1
UUID=55ee0c1a-0000-4000-8000-000000000001 /data ext4 defaults,noatime,nofail 0 2
nas.lan:/export/media /mnt/media nfs _netdev,noauto,x-systemd.automount,x-systemd.idle-timeout=60 0 0
tmpfs /tmp tmpfs defaults,nosuid,nodev,size=2G 0 0
/swapfile none swap sw 0 0
Use the UUID, not /dev/sdb1. Add nofail to anything that is not needed to boot, so a missing data disk does not drop the server into emergency mode. x-systemd.automount mounts a network share on first use and, with the idle timeout, unmounts it again.
$ sudo mkdir -p /data && sudo mount /dev/disk/by-uuid/55ee... /data # try it by hand first
$ sudo findmnt --verify # lint /etc/fstab: unreachable sources, missing mount points, wrong types
$ sudo mount -a # mount everything in fstab that is not mounted: a broken line shows up NOW, not at the next reboot
$ sudo systemctl daemon-reload # systemd turns fstab into mount units: reload after editing it
$ sudo umount /data # "target is busy"? find who holds it: sudo lsof +f -- /data or sudo fuser -vm /data
$ sudo mount -o remount,ro /data # change options of a live mount
$ sudo mount --bind /srv/photos /home/alice/photos # the same folder in two places
Never reboot after editing fstab without running mount -a (or findmnt --verify) first: a typo can leave the machine waiting at the emergency prompt. If it happens, at the console log in as root, fix /etc/fstab (mount -o remount,rw / first if it is read-only), and run mount -a.
LVM: volumes you can resize
Physical Volumes (PV: disks or partitions) are pooled into a Volume Group (VG); you carve Logical Volumes (LV) out of it, format them, and can grow them later while mounted.
$ sudo pvs; sudo vgs; sudo lvs -o +devices # the three one-line-per-object summaries: how much is free in each VG is the key number
$ sudo lvdisplay /dev/vg0/var # details of one LV
Add a new disk and give the space to /var (ext4 or xfs, online, no reboot):
$ lsblk -o NAME,SIZE,TYPE,MOUNTPOINT # find the new disk, say /dev/sdc: CHECK it is empty and the one you mean
$ sudo pvcreate /dev/sdc # make it a physical volume
$ sudo vgextend vg0 /dev/sdc # add it to the volume group
$ sudo lvextend -r -L +50G /dev/vg0/var # grow the LV by 50 GiB AND the filesystem on it (-r)
$ sudo lvextend -r -l +100%FREE /dev/vg0/var # or: use ALL the free space
$ df -h /var # verify
Grow after enlarging a virtual disk in the hypervisor or cloud console (the same disk got bigger):
$ sudo growpart /dev/vda 3 # grow partition 3 to fill the disk (package cloud-guest-utils)
$ sudo pvresize /dev/vda3 # tell LVM the PV got bigger
$ sudo lvextend -r -l +100%FREE /dev/vg0/root # then the LV and filesystem
Create a new volume
$ sudo lvcreate -n data -L 200G vg0 # a 200 GiB logical volume
$ sudo mkfs.ext4 -L data /dev/vg0/data && sudo mkdir -p /data && echo '/dev/vg0/data /data ext4 defaults,nofail 0 2' | sudo tee -a /etc/fstab && sudo mount /data
Snapshot, back up from it, throw it away (the snapshot only needs room for the CHANGES made while it exists)
$ sudo lvcreate -s -n var-snap -L 5G /dev/vg0/var
$ sudo mount -o ro /dev/vg0/var-snap /mnt/snap # xfs also needs: -o ro,nouuid
$ sudo umount /mnt/snap && sudo lvremove /dev/vg0/var-snap # a full snapshot becomes invalid: do not leave them around
Shrinking and moving
$ sudo lvreduce -r -L -20G /dev/vg0/data # ext4 ONLY, it unmounts, checks and shrinks the filesystem first. xfs cannot shrink. BACK UP FIRST.
$ sudo pvmove /dev/sdb /dev/sdc && sudo vgreduce vg0 /dev/sdb && sudo pvremove /dev/sdb # migrate everything off a disk, live, then retire it
Order matters: when GROWING, enlarge the container (LV) first, then the filesystem (lvextend -r does both). When SHRINKING, shrink the filesystem first, then the container: the opposite order destroys data. lvextend/lvreduce with -r do the filesystem step for you.
$ sudo vgcfgbackup; ls /etc/lvm/backup /etc/lvm/archive # LVM keeps its own metadata backups: the way back from a mistake
RAID, encryption and swap
Software RAID with mdadm (mirrors protect against a disk failure, they are NOT a backup)
$ cat /proc/mdstat # state of every array: [UU] is healthy, [U_] is degraded, a progress bar means rebuilding
$ sudo mdadm --detail /dev/md0
$ sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1 # ERASES both; wait for the first sync before relying on it
$ sudo mdadm /dev/md0 --fail /dev/sdb1 --remove /dev/sdb1 # a disk died: remove it, replace it, then:
$ sudo mdadm /dev/md0 --add /dev/sdd1 # rebuild onto the new disk
$ sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf && sudo update-initramfs -u # remember the array (Debian, Ubuntu)
Encryption with LUKS: encrypt a data disk
$ sudo cryptsetup luksFormat /dev/sdb1 # ERASES it and sets a passphrase: back up the LUKS header with sudo cryptsetup luksHeaderBackup ...
$ sudo cryptsetup open /dev/sdb1 secure && sudo mkfs.ext4 /dev/mapper/secure
$ sudo cryptsetup close secure
For unattended mounting at boot, a key file plus an /etc/crypttab entry is needed: keep the key file on an encrypted root, mode 0400.
Swap
$ swapon --show; free -h; cat /proc/sys/vm/swappiness
$ sudo fallocate -l 4G /swapfile && sudo chmod 600 /swapfile && sudo mkswap /swapfile && sudo swapon /swapfile
$ echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
$ sudo swapoff /swapfile && sudo rm /swapfile # remove it
A swap file works on ext4 and xfs; on btrfs it needs special handling (chattr +C, no compression, created with btrfs filesystem mkswapfile). zram (compressed swap in memory: apt install zram-config or systemd-zram-generator) is often a better first choice on small machines. vm.swappiness lowers how eagerly the kernel swaps (sudo sysctl vm.swappiness=10).
Wiping and imaging disks
$ sudo wipefs -a /dev/sdb # remove filesystem and RAID signatures (the quick "make it look empty")
$ sudo dd if=/dev/zero of=/dev/sdb bs=1M status=progress conv=fsync # overwrite the whole disk with zeros
$ sudo blkdiscard /dev/nvme1n1 # SSD/NVMe: discard every block (fast); real secure erase uses the drive's own command (nvme format, hdparm --security-erase)
$ sudo dd if=/dev/sda of=/mnt/backup/sda.img bs=4M status=progress conv=sync,noerror # image a disk to a file (the disk should be unmounted or read-only)
$ sudo ddrescue -d -r3 /dev/sdb sdb.img sdb.map # a FAILING disk: ddrescue retries and keeps a map; plain dd gives up or repeats damage
$ gzip -c < sda.img | ssh backup 'cat > sda.img.gz' # stream an image over ssh
dd has no safety net and its nickname is "disk destroyer": if= is the source, of= is the destination, swapping them erases the source. Say the sentence "if is input, of is output" out loud, check the device with lsblk, then press Enter.
"Disk full" and the mysteries
df says full, but du finds little. Work down this list in order.
$ df -h; df -i # 1. is it SPACE or INODES? 100% in the IUse% column means too many tiny files (sessions, mail queues, caches)
$ sudo du -xh /var --max-depth=2 2>/dev/null | sort -rh | head -15 # 2. where is the space? -x stays on one filesystem
$ sudo find / -xdev -type f -size +500M -printf '%s %p\n' 2>/dev/null | sort -rn | head # 3. the big files
$ sudo lsof +L1 # 4. DELETED files still held open by a running program: the space is not freed until it closes them
$ sudo truncate -s 0 /proc/1234/fd/5 # empty a deleted file through its descriptor without restarting (or just restart the service)
$ sudo tune2fs -l /dev/sdb1 | grep -i 'reserved block' # 5. ext4 reserves 5% for root: a 4 TB disk wastes 200 GB (tune2fs -m 1)
$ sudo mkdir /mnt/rootcheck && sudo mount --bind / /mnt/rootcheck && sudo du -xh /mnt/rootcheck --max-depth=1 | sort -rh | head; sudo umount /mnt/rootcheck
The bind mount trick in step 6 shows files that are HIDDEN underneath another mount: if /data was not mounted when something wrote into /data, those files sit in the folder under the mount point and du /data does not see them after /data is mounted.
Safe clean-up commands
$ sudo journalctl --disk-usage; sudo journalctl --vacuum-size=500M # systemd journal
$ sudo apt clean && sudo apt autoremove --purge # Debian, Ubuntu package cache and unused packages; sudo dnf clean all on RHEL
$ docker system df; docker system prune # Docker: images, containers, volumes (read what prune will delete first)
$ sudo find /var/log -name '*.gz' -mtime +30 -print # old rotated logs: look, then -delete
$ : > /var/log/app/big.log # empty a log that a program still has open (do NOT rm it: the program keeps the space)
$ ls -ld /tmp /var/tmp; sudo du -sh /var/tmp /tmp 2>/dev/null
Watch it before it happens: alert at 80% (a check like df --output=pcent / | tail -1 | tr -dc 0-9 in a timer), and use separate filesystems (or LVM volumes with a quota) for /var/log, /var/lib/docker and data so that one runaway log cannot fill the root and take the whole machine with it.
Health
$ sudo smartctl -H /dev/sda; sudo smartctl -a /dev/sda | grep -Ei 'reallocated|pending|uncorrect|power_on|temperature' # SATA/SAS health (package smartmontools)
$ sudo nvme smart-log /dev/nvme0 # NVMe: percentage_used, media_errors, critical_warning (package nvme-cli)
$ sudo dmesg -T | grep -iE 'i/o error|ata[0-9]|nvme|blk_update_request|ext4-fs error|xfs' # kernel-reported disk trouble
$ iostat -xz 1 3 # await and queue length: see the Performance Triage guide
$ sudo badblocks -sv /dev/sdb # read-only surface test of a disk that is NOT in use (slow); never the -w (write) mode on data you need
Growing "reallocated" or "pending" sector counts, repeated I/O errors in dmesg, or a RAID array that keeps degrading means: back up NOW, then replace the disk. Do not wait for it to die.
Tips
Identify the target device twice (lsblk, then lsblk -o NAME,SIZE,MODEL,SERIAL) before any command that erases.
Use UUIDs or labels in fstab and nofail for non-essential disks. Run findmnt --verify and mount -a before rebooting.
Growing is easy and online (lvextend -r, resize2fs, xfs_growfs). Shrinking is risky: only ext4/btrfs, take a backup, shrink the filesystem BEFORE the container.
Practice on image files (truncate, sfdisk, mkfs, resize2fs): they behave like real disks and cannot hurt anything.
RAID is not a backup, a snapshot is not a backup, and a backup you have never restored is a hope. Test the restore.
Keep free space: 10-20% free on busy filesystems, and separate volumes for logs and containers.
"The disk is full" is the root cause of an amazing share of outages. It deserves a monitoring alert more than most things do.
