☸️ Kubernetes on Proxmox Guide

Kubernetes on Proxmox VE: one control plane node and two workers, built with kubeadm
qm clone 9000 301 --name k8s-cp1 --full
sudo kubeadm init --apiserver-advertise-address=10.10.10.11 --pod-network-cidr=10.244.0.0/16
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
kubectl get nodes

Proxmox VE runs the virtual machines, Kubernetes runs the containers inside them. This page builds the smallest cluster that behaves like a real one: three virtual machines (one control plane node, two workers) made from a single cloud-init template, joined with kubeadm, with containerd as the container runtime and Flannel as the pod network.

The order is: a NAT network for the nodes, an Ubuntu 24.04 cloud image turned into a template, three full clones with fixed addresses, the preparation every node needs (swap off, two kernel modules, three network settings, containerd, the Kubernetes packages), kubeadm init on the first node, kubectl access, the pod network, joining the workers, and then a real application: nginx with three replicas, a NodePort, scaling and draining a node. The last section takes a Proxmox snapshot.

Everything ran for real in a lab: Proxmox VE 9.2.2 (itself a virtual machine with nested KVM, 8 vCPU and 16 GB of memory), Ubuntu Server 24.04 cloud image, containerd 2.2.1 from the Ubuntu repository and Kubernetes v1.37.2 from pkgs.k8s.io. The commands are the ones in the video. The Proxmox guide and the Proxmox web interface guide on this site cover the host; the Kubernetes and kubectl guides cover the cluster once it exists.

Three virtual machines on one Proxmox node:

k8s-cp1   10.10.10.11   4 GB, 2 vCPU   control plane (API server, etcd, scheduler, controller manager)
k8s-w1    10.10.10.12   3 GB, 2 vCPU   worker
k8s-w2    10.10.10.13   3 GB, 2 vCPU   worker

All three sit on a private NAT bridge, vmbr1 (10.10.10.0/24, gateway 10.10.10.1 on the Proxmox host). They reach the internet through the host, and nothing outside can reach them directly. Pods use 10.244.0.0/16.

The Kubernetes documentation asks for 2 GB and 2 CPUs per machine as a minimum for kubeadm. The control plane gets 4 GB here because it also runs etcd and the API server, and the images are pulled on it first.

This was set up before the recording. On the Proxmox node, add to /etc/network/interfaces:

auto vmbr1
iface vmbr1 inet static
	address 10.10.10.1/24
	bridge-ports none
	bridge-stp off
	bridge-fd 0
	post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
	post-up   iptables -t nat -A POSTROUTING -s 10.10.10.0/24 -o vmbr0 -j MASQUERADE
	post-down iptables -t nat -D POSTROUTING -s 10.10.10.0/24 -o vmbr0 -j MASQUERADE

# ifreload -a
# ip -br addr show vmbr1
vmbr1            DOWN           10.10.10.1/24 fe80::c55:f3ff:fe59:68d2/64

DOWN is normal for a bridge without ports until the first VM is plugged in. vmbr0 is the bridge of the installer with the uplink. If your uplink bridge has another name, change it in the MASQUERADE line.

A cloud image is a ready-made, small disk image that expects cloud-init to configure it on first boot. Ubuntu publishes one for every release:

# wget -q --show-progress -P /var/lib/vz/template/iso https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
noble-server-cloudimg-amd6 100%[========================================>] 592.50M  27.5MB/s    in 32s
# ls -lh /var/lib/vz/template/iso
total 593M
-rw-r--r-- 1 root root 593M Oct 17 04:17 noble-server-cloudimg-amd64.img

The file goes to the ISO directory of the local storage only because that is a convenient place. It is a disk image, not an installer.

Create an empty VM, import the image as its disk, add the cloud-init drive and convert it to a template:

# qm create 9000 --name ubuntu-2404-tpl --memory 3072 --cores 2 --cpu host --net0 virtio,bridge=vmbr1 --scsihw virtio-scsi-pci --ostype l26 --agent 1 --serial0 socket --vga serial0
# qm set 9000 --scsi0 local-lvm:0,import-from=/var/lib/vz/template/iso/noble-server-cloudimg-amd64.img
update VM 9000: -scsi0 local-lvm:0,import-from=/var/lib/vz/template/iso/noble-server-cloudimg-amd64.img
  Logical volume "vm-9000-disk-0" created.
...
scsi0: successfully created disk 'local-lvm:vm-9000-disk-0,size=3584M'
# qm disk resize 9000 scsi0 20G
  Size of logical volume pve/vm-9000-disk-0 changed from 3.50 GiB (896 extents) to 20.00 GiB (5120 extents).
# qm set 9000 --ide2 local-lvm:cloudinit --boot order=scsi0
# qm set 9000 --ciuser ubuntu --sshkeys /root/.ssh/authorized_keys --ipconfig0 ip=dhcp
# qm template 9000
  Renamed "vm-9000-disk-0" to "base-9000-disk-0" in volume group "pve"
# qm list
      VMID NAME                 STATUS     MEM(MB)    BOOTDISK(GB) PID
      9000 ubuntu-2404-tpl      stopped    3072              20.00 0

What each option does:
--cpu host        the VM sees the real CPU model, which gives the best speed (and is required for nested virtualization)
--net0 ...vmbr1   VirtIO network on the NAT bridge
--agent 1         enable the QEMU guest agent channel (the image does not contain the agent, see the gotchas)
--serial0 socket --vga serial0   the cloud image has no graphical console, so give it a serial one
import-from        copies the image into the storage as a VM disk (3.5 GiB), then the disk is grown to 20 GB
--ciuser/--sshkeys   the user ubuntu gets the SSH keys of root on the host. There is no password

A full clone copies the disk, so each node is independent of the template. A small shell function does the clone and the per-node settings (memory, fixed address, DNS server):

# mk() { qm clone 9000 $1 --name $2 --full > /dev/null && qm set $1 --memory $4 --ipconfig0 ip=$3/24,gw=10.10.10.1 --nameserver 1.1.1.1 > /dev/null; }
# mk 301 k8s-cp1 10.10.10.11 4096
# mk 302 k8s-w1 10.10.10.12 3072
# mk 303 k8s-w2 10.10.10.13 3072
# for v in 301 302 303; do qm start $v; done
generating cloud-init ISO
generating cloud-init ISO
generating cloud-init ISO
# qm list
      VMID NAME                 STATUS     MEM(MB)    BOOTDISK(GB) PID
       301 k8s-cp1              running    4096              20.00 16834
       302 k8s-w1               running    3072              20.00 16894
       303 k8s-w2               running    3072              20.00 16953
      9000 ubuntu-2404-tpl      stopped    3072              20.00 0

Cloud-init sets the host name (from the VM name), the user, the SSH key and the address on the first boot. Each clone also gets its own SSH host keys and machine ID (checked: the three nodes show different values).

Make the three nodes easy to reach from the Proxmox host:

# printf '10.10.10.11 cp1\n10.10.10.12 w1\n10.10.10.13 w2\n' >> /etc/hosts
# printf 'Host cp1 w1 w2\n  User ubuntu\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n  LogLevel ERROR\n' > ~/.ssh/config
# for n in cp1 w1 w2; do until ssh -o ConnectTimeout=3 $n hostname 2>/dev/null; do sleep 4; done; done
k8s-cp1
k8s-w1
k8s-w2

The loop waits until each VM has booted and accepts the key. StrictHostKeyChecking no and the empty known-hosts file are for a throwaway lab, where clones are recreated with the same addresses and new host keys. Do not use them for machines you care about.

The same machines appear in the Proxmox web interface (port 8006 of the node). Select k8s-cp1:

Summary     CPU, memory and disk. "IPs" reads "Guest Agent not running" because the cloud image does not include qemu-guest-agent
Hardware    2 cores, 4 GiB, a 20 GiB disk, the network on vmbr1 and the cloud-init drive
Cloud-Init  User ubuntu, DNS server 1.1.1.1, the SSH public key, "Upgrade packages: Yes" and IP Config ip=10.10.10.11/24,gw=10.10.10.1

The template 9000 has its own icon in the tree and cannot be started.

The video shows the web interface after the clones are running. The Proxmox web interface guide on this site tours the rest of the screens.

Kubernetes needs the same preparation on all three machines. This is shown on the first node:

# ssh cp1
ubuntu@k8s-cp1:~$ sudo swapoff -a
ubuntu@k8s-cp1:~$ sudo sed -i '/ swap /d' /etc/fstab
ubuntu@k8s-cp1:~$ free -h | grep -i swap
Swap:             0B          0B          0B

1. Swap off. The Kubernetes documentation says the kubelet does not run with swap on by default. The sed line removes a swap entry from /etc/fstab so swap stays off after a reboot (harmless if there is none).

ubuntu@k8s-cp1:~$ printf 'overlay\nbr_netfilter\n' | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
ubuntu@k8s-cp1:~$ sudo modprobe overlay && sudo modprobe br_netfilter
ubuntu@k8s-cp1:~$ lsmod | grep -E "^(overlay|br_netfilter)"
br_netfilter           32768  0
overlay               212992  0

2. Two kernel modules. overlay is used by the container runtime for image layers, br_netfilter lets iptables see traffic that crosses a Linux bridge. The file in /etc/modules-load.d loads them at every boot.

ubuntu@k8s-cp1:~$ printf 'net.bridge.bridge-nf-call-iptables=1\nnet.bridge.bridge-nf-call-ip6tables=1\nnet.ipv4.ip_forward=1\n' | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
ubuntu@k8s-cp1:~$ sudo sysctl --system > /dev/null
ubuntu@k8s-cp1:~$ sysctl net.ipv4.ip_forward net.bridge.bridge-nf-call-iptables
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1

3. Three kernel settings: bridged traffic goes through iptables (so Services work), and the node forwards packets between pods and the network.

containerd is the container runtime. Ubuntu 24.04 has it in its own repository:

ubuntu@k8s-cp1:~$ sudo apt-get update -qq
ubuntu@k8s-cp1:~$ sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq containerd
...
Setting up containerd (2.2.1-0ubuntu1~24.04.4) ...
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /usr/lib/systemd/system/containerd.service.
...
No VM guests are running outdated hypervisor (qemu) binaries on this host.
ubuntu@k8s-cp1:~$ systemctl is-active containerd
active

The lines about "Scanning processes" and "Service restarts being deferred" are needrestart, which Ubuntu Server runs after every apt install. DEBIAN_FRONTEND=noninteractive stops it from asking questions.

Now the one setting that matters. The kubelet and containerd must use the same cgroup driver, systemd. containerd's generated default says false:

ubuntu@k8s-cp1:~$ sudo mkdir -p /etc/containerd
ubuntu@k8s-cp1:~$ containerd config default | sudo tee /etc/containerd/config.toml > /dev/null
ubuntu@k8s-cp1:~$ sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
ubuntu@k8s-cp1:~$ grep SystemdCgroup /etc/containerd/config.toml
            SystemdCgroup = true
ubuntu@k8s-cp1:~$ sudo systemctl restart containerd

Checked on the lab before the sed: "containerd config default | grep SystemdCgroup" printed SystemdCgroup = false.

Kubernetes publishes one package repository per minor version on pkgs.k8s.io. This one is v1.37, the current stable release in the lab:

ubuntu@k8s-cp1:~$ K=https://pkgs.k8s.io/core:/stable:/v1.37/deb
ubuntu@k8s-cp1:~$ curl -fsSL $K/Release.key | sudo gpg --dearmor --yes -o /etc/apt/keyrings/k8s.gpg
ubuntu@k8s-cp1:~$ echo "deb [signed-by=/etc/apt/keyrings/k8s.gpg] $K/ /" | sudo tee /etc/apt/sources.list.d/k8s.list
deb [signed-by=/etc/apt/keyrings/k8s.gpg] https://pkgs.k8s.io/core:/stable:/v1.37/deb/ /
ubuntu@k8s-cp1:~$ sudo apt-get update -qq
ubuntu@k8s-cp1:~$ sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubelet kubeadm kubectl
...
Setting up kubeadm (1.37.2-1.1) ...
Setting up kubectl (1.37.2-1.1) ...
Setting up kubernetes-cni (1.9.1-1.1) ...
Setting up kubelet (1.37.2-1.1) ...
ubuntu@k8s-cp1:~$ sudo apt-mark hold kubelet kubeadm kubectl
kubelet set on hold.
kubeadm set on hold.
kubectl set on hold.
ubuntu@k8s-cp1:~$ kubeadm version -o short
v1.37.2

kubeadm builds the cluster, the kubelet is the agent on every node, kubectl is the command line client. apt-mark hold stops a routine apt upgrade from moving the cluster to a new version by accident: upgrade it on purpose, with the kubeadm upgrade procedure. To use another minor version, change v1.37 in the URL.

The same commands, as one script (prep-node.sh on the Proxmox host):

#!/bin/bash
set -e
sudo swapoff -a
sudo sed -i '/ swap /d' /etc/fstab
printf 'overlay\nbr_netfilter\n' | sudo tee /etc/modules-load.d/k8s.conf
sudo modprobe overlay && sudo modprobe br_netfilter
printf 'net.bridge.bridge-nf-call-iptables=1\nnet.bridge.bridge-nf-call-ip6tables=1\nnet.ipv4.ip_forward=1\n' | sudo tee /etc/sysctl.d/k8s.conf
sudo sysctl --system > /dev/null
sudo apt-get update -qq
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq containerd
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml > /dev/null
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd
K=https://pkgs.k8s.io/core:/stable:/v1.37/deb
curl -fsSL $K/Release.key | sudo gpg --dearmor --yes -o /etc/apt/keyrings/k8s.gpg
echo "deb [signed-by=/etc/apt/keyrings/k8s.gpg] $K/ /" | sudo tee /etc/apt/sources.list.d/k8s.list
sudo apt-get update -qq
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
kubeadm version -o short

Run it on both workers at the same time and wait:

# for n in w1 w2; do ssh $n bash -s < prep-node.sh > /tmp/$n.log 2>&1 & done; wait
[1]-  Done                    ssh $n bash -s < prep-node.sh > /tmp/$n.log 2>&1
[2]+  Done                    ssh $n bash -s < prep-node.sh > /tmp/$n.log 2>&1
# tail -qn 1 /tmp/w1.log /tmp/w2.log
v1.37.2
v1.37.2

With a script like this on three nodes, the preparation took about three minutes in the lab. For more nodes use cloud-init user data or configuration management instead (the Ansible guide on this site).

On the first node, create the cluster. --apiserver-advertise-address is the address the other nodes use to reach the API server, --pod-network-cidr is the range pods will get (it must match the pod network plug-in, Flannel below uses 10.244.0.0/16):

ubuntu@k8s-cp1:~$ sudo kubeadm init --apiserver-advertise-address=10.10.10.11 --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.37.2
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
...
[kubeconfig] Writing "admin.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Starting the kubelet
[control-plane-check] kube-apiserver is healthy after 2.503748891s
...
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!
...
Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.10.10.11:6443 --token iqdffc.oj0xudjidv15r8np \
        --discovery-token-ca-cert-hash sha256:893a582ec9634eb24e77efcb6fa11b05f4808e1bcd2315fda5551ac51ee74af4

What happened: kubeadm checked the node, pulled the control plane images through containerd, made the certificates, wrote a kubeconfig for each component and started etcd, the API server, the controller manager and the scheduler as static pods that the kubelet runs from /etc/kubernetes/manifests. The token and hash above belong to this lab, yours differ.

kubectl reads ~/.kube/config. kubeadm leaves the admin credentials in /etc/kubernetes/admin.conf, readable only by root:

ubuntu@k8s-cp1:~$ mkdir -p ~/.kube
ubuntu@k8s-cp1:~$ sudo cp /etc/kubernetes/admin.conf ~/.kube/config
ubuntu@k8s-cp1:~$ sudo chown $(id -u):$(id -g) ~/.kube/config
ubuntu@k8s-cp1:~$ kubectl get nodes
NAME      STATUS     ROLES           AGE   VERSION
k8s-cp1   NotReady   control-plane   21s   v1.37.2

NotReady is expected: there is no pod network yet. Look at the pods of the system namespace:

ubuntu@k8s-cp1:~$ kubectl get pods -A
NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE
kube-system   coredns-559f6c778d-qhvxp          0/1     Pending   0          14s
kube-system   coredns-559f6c778d-vrb2c          0/1     Pending   0          14s
kube-system   etcd-k8s-cp1                      1/1     Running   0          24s
kube-system   kube-apiserver-k8s-cp1            1/1     Running   0          24s
kube-system   kube-controller-manager-k8s-cp1   1/1     Running   0          24s
kube-system   kube-proxy-drbng                  1/1     Running   0          14s
kube-system   kube-scheduler-k8s-cp1            1/1     Running   0          24s

The two coredns pods stay Pending until a network plug-in exists. The admin.conf credentials are cluster-admin. In a real cluster create a limited user for daily work instead of sharing this file.

Flannel is a small pod network that gives every node a slice of 10.244.0.0/16 and carries pod traffic between nodes:

ubuntu@k8s-cp1:~$ kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
namespace/kube-flannel created
serviceaccount/flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
ubuntu@k8s-cp1:~$ kubectl wait --for=condition=Ready node/k8s-cp1 --timeout=180s
node/k8s-cp1 condition met
ubuntu@k8s-cp1:~$ kubectl get pods -A
NAMESPACE      NAME                              READY   STATUS              RESTARTS   AGE
kube-flannel   kube-flannel-ds-zj6q8             1/1     Running             0          13s
kube-system    coredns-559f6c778d-qhvxp          0/1     ContainerCreating   0          39s
kube-system    coredns-559f6c778d-vrb2c          0/1     ContainerCreating   0          39s
...

The node becomes Ready first, then coredns leaves Pending and starts. "latest" in the URL is convenient for a lab; pin a release in anything you keep. Other plug-ins (Calico, Cilium) work the same way: one manifest or Helm chart, with the pod range matching --pod-network-cidr.

A worker needs the API server address, a token and the hash of the cluster certificate. kubeadm prints all of it as one command. The one printed by kubeadm init works too, but a new one can be made at any time on the control plane. From the Proxmox host:

# J=$(ssh cp1 sudo kubeadm token create --print-join-command)
# echo $J
kubeadm join 10.10.10.11:6443 --token dzr3x0.y6fwon6f99k9h1le --discovery-token-ca-cert-hash sha256:893a582ec9634eb24e77efcb6fa11b05f4808e1bcd2315fda5551ac51ee74af4
# ssh w1 sudo $J | tail -n 4
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
# ssh w2 sudo $J | tail -n 4
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

Back on the control plane:

# ssh cp1
ubuntu@k8s-cp1:~$ kubectl wait --for=condition=Ready node --all --timeout=180s
node/k8s-cp1 condition met
node/k8s-w1 condition met
node/k8s-w2 condition met
ubuntu@k8s-cp1:~$ kubectl get nodes
NAME      STATUS   ROLES           AGE   VERSION
k8s-cp1   Ready    control-plane   92s   v1.37.2
k8s-w1    Ready    <none>          24s   v1.37.2
k8s-w2    Ready    <none>          17s   v1.37.2

ROLES is empty for the workers. That is only a label, kubectl label node k8s-w1 node-role.kubernetes.io/worker= fills it and changes nothing else (not run in the lab).

A Deployment keeps a number of identical pods running. Three copies of nginx; a small alias keeps the pod list short (name, state, node):

ubuntu@k8s-cp1:~$ alias kp='kubectl get pods -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName'
ubuntu@k8s-cp1:~$ kubectl create deployment web --image=nginx --replicas=3
deployment.apps/web created
ubuntu@k8s-cp1:~$ kubectl rollout status deployment web
Waiting for deployment "web" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "web" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "web" rollout to finish: 2 of 3 updated replicas are available...
deployment "web" successfully rolled out
ubuntu@k8s-cp1:~$ kp
NAME                   STATUS    NODE
web-5fc9f4bf66-2wjm5   Running   k8s-w1
web-5fc9f4bf66-bx4gw   Running   k8s-w2
web-5fc9f4bf66-qkh28   Running   k8s-w2

No pod runs on k8s-cp1: kubeadm puts a NoSchedule taint on the control plane node, so ordinary pods go to the workers. The image is pulled from Docker Hub, which needs the internet access the NAT bridge provides.

A Service gives the pods one stable address. Type NodePort also opens the same port, from 30000 to 32767, on every node:

ubuntu@k8s-cp1:~$ kubectl expose deployment web --port=80 --type=NodePort
service/web exposed
ubuntu@k8s-cp1:~$ kubectl get svc web
NAME   TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
web    NodePort   10.99.38.255   <none>        80:30409/TCP   4s
ubuntu@k8s-cp1:~$ P=$(kubectl get svc web -o jsonpath="{.spec.ports[0].nodePort}"); echo $P
30409
ubuntu@k8s-cp1:~$ curl -s 10.10.10.12:$P | grep title
<title>Welcome to nginx!</title>
ubuntu@k8s-cp1:~$ curl -s 10.10.10.13:$P | grep title
<title>Welcome to nginx!</title>
ubuntu@k8s-cp1:~$ curl -s localhost:$P | grep title
<title>Welcome to nginx!</title>

The same page comes from w1, from w2 and from the control plane itself, which runs no nginx pod: kube-proxy on every node forwards the port to a pod wherever it is. Your port differs. From outside the NAT network you would add a port forward on the Proxmox host, or use a load balancer such as MetalLB, which this page did not run.

Scale to six replicas. One command, three new pods:

ubuntu@k8s-cp1:~$ kubectl scale deployment web --replicas=6
deployment.apps/web scaled
ubuntu@k8s-cp1:~$ kubectl rollout status deployment web
deployment "web" successfully rolled out
ubuntu@k8s-cp1:~$ kp
NAME                   STATUS    NODE
web-5fc9f4bf66-2wjm5   Running   k8s-w1
web-5fc9f4bf66-6qsz9   Running   k8s-w1
web-5fc9f4bf66-928lb   Running   k8s-w1
web-5fc9f4bf66-bx4gw   Running   k8s-w2
web-5fc9f4bf66-cx9kr   Running   k8s-w2
web-5fc9f4bf66-qkh28   Running   k8s-w2

Before you reboot or upgrade a node, drain it: cordon (no new pods) and evict the pods it runs. DaemonSet pods such as kube-proxy and Flannel stay, so the flag says to ignore them:

ubuntu@k8s-cp1:~$ kubectl drain k8s-w2 --ignore-daemonsets --delete-emptydir-data
node/k8s-w2 cordoned
Warning: ignoring DaemonSet-managed Pods: kube-flannel/kube-flannel-ds-frggg, kube-system/kube-proxy-l9kjk
evicting pod default/web-5fc9f4bf66-qkh28
evicting pod default/web-5fc9f4bf66-bx4gw
evicting pod default/web-5fc9f4bf66-cx9kr
pod/web-5fc9f4bf66-qkh28 evicted
pod/web-5fc9f4bf66-cx9kr evicted
pod/web-5fc9f4bf66-bx4gw evicted
node/k8s-w2 drained
ubuntu@k8s-cp1:~$ kp
NAME                   STATUS    NODE
web-5fc9f4bf66-2wjm5   Running   k8s-w1
web-5fc9f4bf66-6qsz9   Running   k8s-w1
web-5fc9f4bf66-8fvhq   Running   k8s-w1
web-5fc9f4bf66-928lb   Running   k8s-w1
web-5fc9f4bf66-pn9hm   Running   k8s-w1
web-5fc9f4bf66-vpsg5   Running   k8s-w1
ubuntu@k8s-cp1:~$ kubectl get nodes
NAME      STATUS                     ROLES           AGE    VERSION
k8s-cp1   Ready                      control-plane   3m     v1.37.2
k8s-w1    Ready                      <none>          112s   v1.37.2
k8s-w2    Ready,SchedulingDisabled   <none>          105s   v1.37.2

All six pods moved to k8s-w1 and the Deployment kept its six replicas. Now bring the node back:

ubuntu@k8s-cp1:~$ kubectl uncordon k8s-w2
node/k8s-w2 uncordoned
ubuntu@k8s-cp1:~$ kubectl get nodes
NAME      STATUS   ROLES           AGE    VERSION
k8s-cp1   Ready    control-plane   3m8s   v1.37.2
k8s-w1    Ready    <none>          2m     v1.37.2
k8s-w2    Ready    <none>          113s   v1.37.2

uncordon only lets the scheduler use the node again. The pods that moved stay where they are, new pods will use w2 too.

A Proxmox snapshot is a restore point for a VM, a good thing to take before you upgrade Kubernetes or change the pod network. In the web interface: select the VM, Snapshots, Take Snapshot:

Name         cluster-ready
Include RAM  (unticked for this lab, the disk state is enough)
Description  (optional)

The snapshot task finishes with status OK, and the Snapshots tab lists cluster-ready above NOW ("You are here!"). Rollback returns the VM to that state. The video takes the snapshot of k8s-cp1 only: take one of every node at about the same moment, so the three stay consistent with each other (etcd lives on the control plane, so its snapshot matters most). The same from the shell, per node:

# qm snapshot 301 cluster-ready
# qm listsnapshot 301

(the shell form was not run in the lab, the web interface was)

swap is on               the kubelet does not run with swap on by default (Kubernetes documentation; the lab nodes had it off). swapoff -a and remove the swap line from /etc/fstab.
SystemdCgroup = false    containerd's generated default. Kubernetes documents that the kubelet and containerd must use the same cgroup driver, systemd on a systemd distribution; a mismatch is a known cause of unstable pods (not provoked in the lab). Change it to true before kubeadm init.
NotReady, coredns Pending   normal until the pod network is installed. Not a fault.
Guest Agent not running  the Ubuntu cloud image does not contain qemu-guest-agent, although the template says --agent 1. Install it (apt install qemu-guest-agent) in the template if you want IPs in the web interface and clean shutdowns.
needrestart output       after every apt install on Ubuntu Server. Harmless. DEBIAN_FRONTEND=noninteractive keeps it from asking.
hold                     the packages are on hold, so apt upgrade skips kubelet, kubeadm and kubectl. To upgrade, apt-mark unhold them on purpose and follow the kubeadm upgrade procedure, one minor version at a time.
join token               kubeadm join tokens expire; by default after 24 hours (kubeadm documentation, not waited for in the lab). Make a new one with kubeadm token create --print-join-command.
ROLES is <none>          for a worker this is normal, see above.
no pod on the control plane   the NoSchedule taint, on purpose. For a one-node lab remove it: kubectl taint nodes k8s-cp1 node-role.kubernetes.io/control-plane- (not run).
duplicate machine IDs    always make clones with --full from a clean template that has never booted, otherwise nodes share a machine ID and host keys (the template here never started, which is why each clone is different).

One control plane only. A highly available control plane (three of them behind a virtual IP or load balancer), etcd backups and restores.
Upgrading the cluster to the next version with kubeadm upgrade.
Persistent storage: this page has no volumes. Proxmox storage (LVM-thin, ZFS, Ceph) can back volumes through a CSI driver, which was not run.
Ingress controllers and load balancers (MetalLB, ingress-nginx), TLS certificates.
Other pod networks than Flannel (Calico, Cilium), network policies.
Another distribution: k3s is a lighter way to get a cluster (see the Kubernetes guide on this site). Debian, Rocky Linux and others need different package commands.
Running Proxmox on real hardware with several nodes: the lab was a single Proxmox node, itself a virtual machine with nested virtualization, so the speed numbers are not representative.
Terraform or Ansible to automate the VM creation: the Terraform and Ansible guides on this site cover them.

Make a template once, clone often: the whole cluster from step "Clone three nodes" to "Join the workers" takes a few minutes.
Give every node a fixed address (cloud-init ipconfig0). kubeadm and the certificates know the control plane by its address.
Keep the template free of Kubernetes. Install the packages on the clones, or build a second template with them for faster new nodes.
Take a Proxmox snapshot of every node before an upgrade, and after a clean install.
Run sudo kubeadm init phase preflight before the real init when something looks wrong (not run in the lab).
Start again quickly: kubeadm reset on a node (not run here) or destroy the VM and clone it again, which is what the lab did between the recordings.
Memory: three nodes with 4 GB, 3 GB and 3 GB ran this page comfortably in the lab, with six nginx pods.