🖥️ QEMU and KVM Guide

qemu-img create -f qcow2 -b base.img -F qcow2 web1.qcow2 10G
qemu-system-x86_64 -enable-kvm -m 1024 -smp 2 -drive file=web1.qcow2,if=virtio -nic user,hostfwd=tcp::2222-:22 -daemonize
ssh web1        # a full Ubuntu machine, up in seconds

Proxmox, libvirt, virt-manager and OpenStack all run QEMU underneath. Using it directly shows you what they do for you, and gets you a throwaway machine in seconds when you only need to try something. This page builds one from a downloaded cloud image, with no installer.

You will make a thin disk on top of a base image, give the first boot a user and a key with cloud-init, start the machine in the background, talk to it with ssh and through the monitor socket, take and apply a snapshot, grow the disk, convert it to other formats, and see the errors you are most likely to meet.

Every example ran for real, as an ordinary user, on Ubuntu 24.04 with QEMU 8.2.2 and a real /dev/kvm. The machine inside is Ubuntu 24.04 too. The output you see is the output the machine gave.

QEMU is a program that pretends to be a whole computer: a processor, memory, a disk, a network card. On its own it is slow, because it has to translate every instruction. With KVM, a part of the Linux kernel, the guest's instructions run directly on the real processor, and the machine is nearly as fast as the host. Proxmox, libvirt, virt-manager and OpenStack all start QEMU underneath. This page uses QEMU directly, from the command line, so you can see what those tools do for you.

Every example ran for real, as an ordinary user, on Ubuntu 24.04 with QEMU 8.2.2 and a real /dev/kvm. First check the three things you need: the KVM device, your membership in the group that owns it, and QEMU itself:

$ ls -l /dev/kvm
crw-rw---- 1 root kvm 10, 232 Nov  6 03:16 /dev/kvm
$ id -nG | tr ' ' '\n' | grep kvm
kvm
$ qemu-system-x86_64 --version | head -n 1
QEMU emulator version 8.2.2 (Debian 1:8.2.2+ds-0ubuntu1.18)
$ qemu-system-x86_64 -accel help
Accelerators supported in QEMU binary:
tcg
kvm

If /dev/kvm is missing, the processor has no virtualization turned on in the firmware, or you are inside a virtual machine that does not pass it through. If you are not in the kvm group, add yourself with sudo usermod -aG kvm $USER and log in again. Without KVM the same commands still work with -accel tcg, only much slower.

The packages are qemu-system-x86 and qemu-utils on Ubuntu and Debian, and qemu-kvm and qemu-img on Rocky and RHEL. We also use cloud-image-utils for one small tool, and socat to talk to a socket:

$ sudo apt install qemu-system-x86 qemu-utils cloud-image-utils socat

A virtual machine needs a disk. Instead of installing an operating system, we download a cloud image: a small, ready-made disk that Ubuntu publishes for exactly this. It is a file in the qcow2 format, which only stores the blocks that are in use, and can stack on top of another file.

$ mkdir -p ~/vms
$ wget -P ~/vms https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img

We will never write to that file. Each machine gets its own thin disk that points at it, called a backing file. Everything the machine writes goes to the thin disk, and everything it reads and has not changed comes from the base. First look at the base:

$ qemu-img info vms/noble-server-cloudimg-amd64.img | head -n 5
image: vms/noble-server-cloudimg-amd64.img
file format: qcow2
virtual size: 3.5 GiB (3758096384 bytes)
disk size: 593 MiB
cluster_size: 65536

It is 593 MiB on disk and says it is 3.5 GiB. The second number is the size the guest sees. Now create the disk for a machine called web1, ten gigabytes, backed by the base:

$ qemu-img create -f qcow2 -b noble-server-cloudimg-amd64.img -F qcow2 vms/web1.qcow2 10G
Formatting 'vms/web1.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=10737418240 backing_file=noble-server-cloudimg-amd64.img backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
$ qemu-img info vms/web1.qcow2 | head -n 6
image: vms/web1.qcow2
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 196 KiB
cluster_size: 65536
backing file: noble-server-cloudimg-amd64.img (actual path: vms/noble-server-cloudimg-amd64.img)
$ ls -lh vms
total 593M
-rw-r--r-- 1 tc tc 593M Nov  6 03:17 noble-server-cloudimg-amd64.img
-rw-r--r-- 1 tc tc 193K Nov  6 03:44 web1.qcow2

The new file is 193 kilobytes. The guest sees ten gigabytes, but nothing has been written yet. The backing file name is relative to the new file, so keep both in the same directory. This is why you can have twenty machines from one image: each takes the space of what it changed.

A cloud image has no password and no user you know. It reads its settings on the first boot from cloud-init, which looks for a small disk labelled cidata. We build that disk ourselves. It holds two text files. First make a key pair for logging in, and set up a name for the machine in the ssh configuration, so that later commands are short:

$ ssh-keygen -q -t ed25519 -N '' -f ~/.ssh/id_qemu
$ cat .ssh/config
Host web1
	HostName 127.0.0.1
	Port 2222
	User ubuntu
	IdentityFile ~/.ssh/id_qemu
	StrictHostKeyChecking no
	UserKnownHostsFile /dev/null
	LogLevel ERROR
$ chmod 600 .ssh/config

The StrictHostKeyChecking lines are for a throwaway lab machine whose host key changes each time it is rebuilt. Do not copy them for a server. Now the user-data file, which sets the host name, puts your public key on the ubuntu user, and also gives that user a password for the console:

$ printf '#cloud-config\nhostname: web1\npassword: ubuntu\nchpasswd: { expire: false }\nssh_authorized_keys:\n  - %s\n' "$(cat ~/.ssh/id_qemu.pub)" > vms/user-data
$ cat vms/user-data | cut -c1-60
#cloud-config
hostname: web1
password: ubuntu
chpasswd: { expire: false }
ssh_authorized_keys:
  - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILIWpb56DA2hrsWV/rkn
$ printf 'instance-id: web1\n' > vms/meta-data
$ cloud-localds vms/seed.img vms/user-data vms/meta-data
$ ls -lh vms/seed.img
-rw-r--r-- 1 tc tc 366K Nov  6 03:44 vms/seed.img

The instance-id matters: cloud-init runs its setup once for each id, so a copy of this disk with a new id would configure itself again.

Here is the whole command. Every option is something a graphical tool would otherwise choose for you. Put it in a script, so you can start any machine by name:

$ cat start.sh
#!/bin/sh
# usage: ./start.sh NAME PORT
qemu-system-x86_64 -enable-kvm -cpu host -m 1024 -smp 2 -name $1 \
  -drive file=vms/$1.qcow2,if=virtio -drive file=vms/seed.img,if=virtio,format=raw \
  -nic user,model=virtio-net-pci,hostfwd=tcp::$2-:22 -display none -serial file:vms/$1.log \
  -monitor unix:vms/$1.mon,server,nowait -daemonize -pidfile vms/$1.pid
$ chmod +x start.sh

-enable-kvm uses the kernel's virtualization, and -cpu host gives the guest the real processor's features. -m 1024 is one gigabyte of memory and -smp 2 is two processors. The two -drive options attach the machine's disk and the cloud-init disk, both as virtio, the fast paravirtual kind. -nic user is the simplest network: QEMU gives the guest a private address behind its own built-in NAT, and hostfwd makes port 2222 on your machine reach port 22 in the guest. It needs no root and no setup.

-display none -serial file:... means no window, and the console output goes to a log file. -monitor opens a control socket, which we use in a moment. -daemonize puts QEMU in the background once it has started, and -pidfile records its process number. Start it:

$ ./start.sh web1 2222; echo "exit status $?"
exit status 0
$ cat vms/web1.pid
7583

It returned at once. The machine is booting. The console log shows how far it has got, and the first boot, with cloud-init, takes only a few seconds with KVM:

$ until grep -aq 'login:' vms/web1.log; do sleep 1; done; sed 's/\x1b\[[0-9;]*m//g' vms/web1.log | grep -a 'login:' | tail -n 1
web1 login:

To see the console live instead, replace the display and serial options with -nographic. Your terminal then is the console, and you leave QEMU with Ctrl-a, then x. Everything else is the same.

When ssh answers, the machine is up. This loop keeps trying until it does, and then asks a few questions:

$ until ssh -o ConnectTimeout=2 web1 true 2>/dev/null; do sleep 1; done; echo "ssh is up"
ssh is up
$ ssh web1 'hostname; uname -r; nproc; free -m | head -n 2'
web1
6.8.0-147-generic
2
               total        used        free      shared  buff/cache   available
Mem:             961         311         506           0         286         650
$ ssh web1 'systemd-detect-virt; df -h / | tail -n 1; lsblk -d -o NAME,SIZE'
kvm
/dev/vda1       8.7G  1.7G  7.1G  19% /
NAME  SIZE
fd0     4K
sr0  1024M
vda    10G
vdb   366K

The guest has the host name from user-data, two processors and about a gigabyte of memory, and systemd-detect-virt reports kvm. The root file system is the full ten gigabytes of the disk we made, because the image grows its partition on the first boot. The small vdb disk is the cloud-init seed.

The network is what -nic user gave it. The guest has a private address, and reaches the outside through the QEMU gateway:

$ ssh web1 'ip -4 -brief addr show | tail -n 1; ip route | head -n 1'
ens3             UP             10.0.2.15/24 metric 100
default via 10.0.2.2 dev ens3 proto dhcp src 10.0.2.15 metric 100

This is why port forwarding was needed: from your machine you cannot reach 10.0.2.15, only the ports you forwarded. For machines that need to be seen on the network, QEMU can attach to a bridge instead, which needs root and some set-up on the host, and which libvirt and Proxmox do for you.

The -monitor option made a socket. Anything you send to it is a QEMU command, with no need to log in to the guest. socat connects a pipe to a socket, so a small script is enough:

$ cat hmp.sh
#!/bin/sh
# usage: ./hmp.sh NAME 'monitor command'
echo "$2" | socat - UNIX-CONNECT:vms/$1.mon
$ chmod +x hmp.sh
$ ./hmp.sh web1 'info status' | grep -a 'VM status'
VM status: running
$ ./hmp.sh web1 'info block' | grep -a -A3 virtio0
virtio0 (#block198): vms/web1.qcow2 (qcow2)
    Attached to:      /machine/peripheral-anon/device[0]/virtio-backend
    Cache mode:       writeback
    Backing file:     vms/noble-server-cloudimg-amd64.img (chain depth: 1)

The status is running, and info block shows the disk and, below it, the backing file. Two more commands are worth knowing. stop freezes the machine, as if the processors were switched off, and cont resumes it. While it is stopped, nothing answers:

$ ./hmp.sh web1 'stop' > /dev/null; ./hmp.sh web1 'info status' | grep -a 'VM status'
VM status: paused
$ timeout 3 ssh web1 hostname; echo "exit status $?"
exit status 124
$ ./hmp.sh web1 'cont' > /dev/null; ./hmp.sh web1 'info status' | grep -a 'VM status'
VM status: running
$ ssh web1 hostname
web1

Exit status 124 is timeout giving up. After cont, the machine carries on as if nothing happened. The right way to turn a machine off is system_powerdown, which presses the virtual power button, so the guest shuts down cleanly. QEMU exits when the guest is off, and removes the pid file:

$ ./hmp.sh web1 'system_powerdown' > /dev/null
$ until [ ! -e vms/web1.pid ]; do sleep 1; done; echo "stopped"
stopped
$ ls vms
meta-data
noble-server-cloudimg-amd64.img
seed.img
user-data
web1.log
web1.mon
web1.qcow2

The quit command in the monitor stops QEMU at once, like pulling the plug. Use it only on a machine you do not care about.

Because the disk is a qcow2 file, it can keep snapshots of itself. A snapshot costs almost nothing when you take it, and grows only as the disk changes afterwards. The machine must be off for this, which it is now. Take one called clean, and list it:

$ qemu-img snapshot -c clean vms/web1.qcow2
$ qemu-img snapshot -l vms/web1.qcow2
Snapshot list:
ID        TAG               VM SIZE                DATE     VM CLOCK     ICOUNT
1         clean                 0 B 2026-11-06 03:44:48 00:00:00.000          0

Now start the machine, make a change, and shut down:

$ ./start.sh web1 2222; until ssh -o ConnectTimeout=2 web1 true 2>/dev/null; do sleep 1; done
$ ssh web1 'touch ~/important.txt; ls ~'
important.txt
$ ./hmp.sh web1 'system_powerdown' > /dev/null; until [ ! -e vms/web1.pid ]; do sleep 1; done; echo "stopped"
stopped

The file exists on the disk. Apply the snapshot to go back, start again, and look:

$ qemu-img snapshot -a clean vms/web1.qcow2
$ ./start.sh web1 2222; until ssh -o ConnectTimeout=2 web1 true 2>/dev/null; do sleep 1; done
$ ssh web1 'ls -A ~ | grep -c important; echo done'
0
done

The count is 0: the file is gone, and the machine is as it was when the snapshot was taken. Use -d to delete a snapshot you no longer need. If you try to touch the image while the machine is running, QEMU refuses, and that is a good thing, because two writers would damage a disk:

$ qemu-img snapshot -l vms/web1.qcow2; echo "exit status $?"
qemu-img: Could not open 'vms/web1.qcow2': Failed to get shared "write" lock
Is another process using the image [vms/web1.qcow2]?
exit status 1

For a disk you can throw away every time, there is also the -snapshot option of qemu-system, which sends all writes to a temporary file and leaves the disk untouched.

A qcow2 disk can grow while the machine is off. Shut down, resize, start again, and the guest sees the new size:

$ ./hmp.sh web1 'system_powerdown' > /dev/null; until [ ! -e vms/web1.pid ]; do sleep 1; done; echo "stopped"
stopped
$ qemu-img resize vms/web1.qcow2 20G
Image resized.
$ ./start.sh web1 2222; until ssh -o ConnectTimeout=2 web1 true 2>/dev/null; do sleep 1; done
$ ssh web1 'lsblk -d -o NAME,SIZE | grep vda; df -h / | tail -n 1'
vda    20G
/dev/vda1        19G  1.7G   17G   9% /

The disk is 20 gigabytes and the root file system grew to match, because cloud images enlarge the partition and the file system at boot. With another operating system you grow the partition and the file system yourself. Shrinking needs the --shrink option, and is dangerous, because the data past the new end is lost.

qemu-img convert changes the format of a disk or makes a copy of it. A copy made this way is a single file that no longer depends on the base image, so you can move it to another machine. -c compresses it. The machine must be off:

$ ./hmp.sh web1 'system_powerdown' > /dev/null; until [ ! -e vms/web1.pid ]; do sleep 1; done; echo "stopped"
stopped
$ qemu-img convert -O qcow2 -c vms/web1.qcow2 vms/web1-flat.qcow2
$ qemu-img info vms/web1-flat.qcow2 | head -n 4
image: vms/web1-flat.qcow2
file format: qcow2
virtual size: 20 GiB (21474836480 bytes)
disk size: 604 MiB
$ ls -lh vms/*.qcow2
-rw-r--r-- 1 tc tc 606M Nov  6 03:47 vms/web1-flat.qcow2
-rw-r--r-- 1 tc tc  89M Nov  6 03:45 vms/web1.qcow2

The flat copy has no backing file line, and it is bigger, because it now includes the 600 megabytes of the base. The thin disk is only the difference. The same command writes other formats. Raw is a plain image of the disk, which dd can write to a real drive, and vmdk, vdi and vhdx are for VMware, VirtualBox and Hyper-V. A raw file is sparse, so ls -s shows the space it really uses:

$ qemu-img convert -O raw vms/web1-flat.qcow2 vms/web1.raw
$ ls -lhs vms/web1.raw
1.8G -rw-r--r-- 1 tc tc 20G Nov  6 03:47 vms/web1.raw
$ file vms/web1.raw | cut -c1-70
vms/web1.raw: DOS/MBR boot sector, extended partition table (last)

A raw image is as large as the disk, 20 gigabytes, but only about two gigabytes are really used. Finally, check a disk for damage, which you can do after a crash or a full file system on the host:

$ qemu-img check vms/web1.qcow2
No errors were found on the image.
1304/327680 = 0.40% allocated, 16.41% fragmented, 0.00% compressed clusters
Image end offset: 92930048

Most problems come in a few kinds. The first is starting a second machine on a disk that another one is using. QEMU locks the image to stop it:

$ ./start.sh web1 2222; until ssh -o ConnectTimeout=2 web1 true 2>/dev/null; do sleep 1; done
$ qemu-system-x86_64 -enable-kvm -m 256 -display none -drive file=vms/web1.qcow2,if=virtio -nic user,hostfwd=tcp::2223-:22; echo "exit status $?"
qemu-system-x86_64: -drive file=vms/web1.qcow2,if=virtio: Failed to get "write" lock
Is another process using the image [vms/web1.qcow2]?
exit status 1

The first start worked. The second command was refused with Failed to get write lock, which means another process has the disk. Find that process with the pid file, and do not remove the lock by hand. The second kind is a port that is taken. Starting another machine on 2222 while web1 uses it fails like this:

$ qemu-img create -q -f qcow2 vms/web2.qcow2 1G
$ qemu-system-x86_64 -enable-kvm -m 256 -display none -drive file=vms/web2.qcow2,if=virtio -nic user,hostfwd=tcp::2222-:22; echo "exit status $?"
qemu-system-x86_64: -nic user,hostfwd=tcp::2222-:22: Could not set up host forwarding rule 'tcp::2222-:22'
exit status 1

Give each machine its own port. The third is a missing permission on /dev/kvm. A user who is not in the kvm group is refused. Here the command runs as nobody, a user that is not in the group:

$ sudo -n -u nobody qemu-system-x86_64 -enable-kvm -m 256 -display none; echo "exit status $?"
Could not access KVM kernel module: Permission denied
qemu-system-x86_64: failed to initialize kvm: Permission denied
exit status 1

Add yourself to the group, as at the start. The fourth is a machine that starts and never answers. Look at the console log, which is why we kept it:

$ tail -n 3 vms/web1.log | cut -c1-70
[   10.923284] cloud-init[833]: Cloud-init v. 26.1-0ubuntu1~24.04.1 fi
[  OK  ] Finished cloud-final.service - Cloud-
[  OK  ] Reached target cloud-init.target - Cl

When a machine will not boot, switch -display none -serial file:... to -nographic and watch the console. Last, to clean up: turn the machine off, and delete its files. The base image stays for the next one:

$ ./hmp.sh web1 'system_powerdown' > /dev/null; until [ ! -e vms/web1.pid ]; do sleep 1; done; echo "stopped"
stopped
$ rm vms/web1.* vms/web2.qcow2
$ ls vms
meta-data
noble-server-cloudimg-amd64.img
seed.img
user-data
web1-flat.qcow2

From here, libvirt (virsh and virt-manager) adds named machines, networks and storage pools on top of the same QEMU, and Proxmox adds a web interface and clusters. Everything they do is options like the ones above.