A how-to manual: how a cluster works, running and updating workloads, services and ingress, configuration, storage, resource limits, probes, scheduling, autoscaling, network policy, access control, backups and upgrades, with every example run for real on a two-node lab.
▶Prefer to watch? This short video runs this page's main commands on a real two-node k3s cluster, with on-screen captions and no voiceover. Click to play, or watch it on YouTube.
kubectl: talk to a Kubernetes cluster. Describe what you want, and the cluster makes it so.
kubectl apply -f app.yaml
kubectl get pods -o wide
kubectl describe pod NAME
kubectl rollout undo deployment NAME
Kubernetes runs containers across a group of machines and keeps them in the state you declared. You write down what you want (three copies of this web server, reachable on port 80) and a set of controllers keeps making that true: they start the copies, replace the ones that die, move them off a machine that fails and spread the traffic. It is a very large system, and most of the trouble comes from a few places: a pod that never gets scheduled, a Service that finds no pods, a limit that kills a container, a volume that lives on one node, a certificate that quietly expires, an upgrade done in the wrong order.
This guide is how to run it. It installs a cluster, runs workloads, gives them configuration, storage and a network, limits and protects them, schedules and scales them, and operates the cluster: backup and restore, node maintenance, certificates and upgrades. The kubectl Day-to-Day page on this site covers the command line and the failure gallery in depth, and the cheat sheets for Helm, Kustomize, K9s and Popeye cover those tools. The older cheat sheet that stood on this page is replaced by this guide, and its one-line commands are in the cheat sheet section.
Every example ran for real, on two Ubuntu 24.04 machines (containers with their own container runtime inside) running k3s, a small certified Kubernetes distribution: first v1.35.9, then upgraded to v1.36.5 in the upgrade section. One node was the server (control plane with embedded etcd), and one was a worker. Real clusters have more control-plane nodes, separate machines and a real load balancer. Those, cloud-managed clusters and everything that needs them were not run, and the section "What was not run in the lab" lists them.
What Kubernetes is, and the lab used here
Kubernetes runs containers across a group of machines and keeps them in the state you declared. You write down what you want ("three copies of this web server, reachable on port 80"), and Kubernetes keeps making that true: it starts the copies, replaces the ones that die, moves them off a machine that fails, and spreads traffic over them. You do not log in to a machine and start things. You describe, and a set of controllers does the work, continuously.
The parts, in the order a request meets them:
API server the front door. Every kubectl command, and every other component, talks to it. It checks who you are and what you may do.
etcd the database behind the API server. It holds the whole cluster state: if etcd is lost, the cluster has forgotten everything.
scheduler picks a node for every new pod, from its resource requests, its rules and the node's room.
controller manager runs the loops that make reality match the declared state (the Deployment controller, the node controller and so on).
kubelet an agent on every node. It starts the containers the API says should run on that node, and reports back.
container runtime what the kubelet asks to start containers (containerd, in most clusters).
kube-proxy / CNI the network: a flat pod network, and the rules that make Services work.
control plane the first four together. Worker nodes run the last three.
The words you meet next:
pod the smallest thing Kubernetes runs: one or more containers that share a network address and storage.
deployment keeps N identical pods running and does rolling updates.
service a stable name and address in front of a changing set of pods.
namespace a folder for objects, with its own names, quotas and permissions.
label a key and value on an object. Selectors (in services, deployments, policies) find objects by label.
manifest the YAML that describes an object.
This guide is how to run it: install a cluster, run workloads, give them configuration, storage and a network, limit and protect them, schedule them, scale them, and operate the cluster (backup, upgrade, drain a node). The tool for reading and changing objects day to day is covered by the kubectl Day-to-Day page on this site, which has the failure gallery (CrashLoopBackOff, Pending, ImagePullBackOff and the others, one by one). The older cheat sheet that stood on this page is replaced by this guide, and its one-line commands are in the cheat sheet section.
Every example below ran for real, on a lab of two Ubuntu 24.04 machines (containers, with their own container runtime inside) running k3s, a small certified Kubernetes distribution from Rancher/SUSE. The lab started on Kubernetes v1.35.9 (k3s v1.35.9+k3s1), and the upgrade section moves it to v1.36.5. One node is the server (control plane with embedded etcd, named k8s) and one is a worker (named k8w1). Real clusters have three or five control-plane nodes, separate machines and a real load balancer. Those, cloud-managed clusters (EKS, AKS, GKE) and everything that needs them were not run, and the section "What was not run in the lab" lists them.
$ kubectl version
Client Version: v1.35.9+k3s1
Kustomize Version: v5.7.1
Server Version: v1.35.9+k3s1
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s Ready control-plane,etcd 21s v1.35.9+k3s1 172.25.0.2 <none> Ubuntu 24.04.5 LTS 7.0.0-38-generic containerd://2.2.7-k3s1
k8w1 Ready <none> 15s v1.35.9+k3s1 172.25.0.3 <none> Ubuntu 24.04.5 LTS 7.0.0-38-generic containerd://2.2.7-k3s1
$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-b644fd95b-rfzv8 1/1 Running 0 16s
kube-system helm-install-traefik-crd-758p4 0/1 Completed 0 16s
kube-system helm-install-traefik-nxwfm 0/1 Completed 2 16s
kube-system local-path-provisioner-d9d8769b6-582zr 1/1 Running 0 16s
kube-system metrics-server-7f4b6d9bd7-mcfwk 0/1 Running 0 16s
kube-system svclb-traefik-5192b2cf-4xrns 2/2 Running 0 2s
kube-system svclb-traefik-5192b2cf-dlvkh 2/2 Running 0 2s
kube-system traefik-5869c87968-fszlq 1/1 Running 0 2s
A cluster on Ubuntu with k3s
There are many ways to get a cluster. Use the one that fits the job:
k3s one binary, installs in a minute, runs on a Raspberry Pi or a VM. Used here. Bundles Traefik (ingress), a local storage provisioner, CoreDNS and metrics-server.
kubeadm the upstream tool for building a cluster from separate machines. You install the runtime, the kubelet and the network add-on yourself. More work, nothing hidden. Not run in the lab.
kind, minikube clusters inside containers or a VM, for development and tests. Not run in the lab.
EKS, AKS, GKE the cloud provider runs the control plane for you; you manage the nodes (or not, in autopilot modes). Not run in the lab.
The k3s way, from its own documentation, is one command on each machine. On the first one (the server):
$ curl -sfL https://get.k3s.io | sh -
$ sudo kubectl get nodes
and on a machine that should join as a worker, with the server's address and its token:
$ curl -sfL https://get.k3s.io | K3S_URL=https://SERVER:6443 K3S_TOKEN=TOKEN sh -
$ sudo cat /var/lib/rancher/k3s/server/node-token # on the server: where the token is
The script installs a systemd service. The lab machines are containers with no systemd, so it could not be used (it stopped with "Can not find systemd or openrc"), and the lab ran the same binary by hand instead: k3s server and k3s agent, which is exactly what the service runs. Everything after the install is the same. The server was started with --cluster-init, so that it uses embedded etcd (the default is SQLite on a single server, which has no snapshot command, and the backup section needs etcd).
$ k3s --version
k3s version v1.35.9+k3s1 (58877f27)
go version go1.26.8
$ ps -eo args | grep -E '^k3s (server|agent)' # run as root on the server node
k3s server
$ ps -eo args | grep -E '^k3s (server|agent)' | cut -c1-70 # run as root on the worker node
k3s agent
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s Ready control-plane,etcd 22s v1.35.9+k3s1 172.25.0.2 <none> Ubuntu 24.04.5 LTS 7.0.0-38-generic containerd://2.2.7-k3s1
k8w1 Ready <none> 16s v1.35.9+k3s1 172.25.0.3 <none> Ubuntu 24.04.5 LTS 7.0.0-38-generic containerd://2.2.7-k3s1
$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-b644fd95b-rfzv8 1/1 Running 0 17s
kube-system helm-install-traefik-crd-758p4 0/1 Completed 0 17s
kube-system helm-install-traefik-nxwfm 0/1 Completed 2 17s
kube-system local-path-provisioner-d9d8769b6-582zr 1/1 Running 0 17s
kube-system metrics-server-7f4b6d9bd7-mcfwk 0/1 Running 0 17s
kube-system svclb-traefik-5192b2cf-4xrns 2/2 Running 0 3s
kube-system svclb-traefik-5192b2cf-dlvkh 2/2 Running 0 3s
kube-system traefik-5869c87968-fszlq 1/1 Running 0 3s
Four things are running that you did not ask for: coredns (cluster DNS), metrics-server (what kubectl top reads), local-path-provisioner (storage) and traefik (the ingress controller, with an svclb pod per node that opens port 80 and 443 on the node). The two helm-install-traefik pods are Jobs that installed Traefik from a chart, and they show Completed, which is what a finished Job looks like.
kubectl needs to know where the API server is and who you are. That comes from a kubeconfig file. k3s writes one at /etc/rancher/k3s/k3s.yaml (readable by everyone here, because the server was started with --write-kubeconfig-mode 644):
$ echo $KUBECONFIG
/etc/rancher/k3s/k3s.yaml
$ kubectl config view --minify | grep -E 'server:|name:|user:'
server: https://127.0.0.1:6443
name: default
user: default
name: default
- name: default
user:
$ kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* default default default
$ kubectl cluster-info
Kubernetes control plane is running at https://127.0.0.1:6443
CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
A kubeconfig holds clusters, users (credentials) and contexts, which pair one cluster with one user and a default namespace. With several clusters, kubectl config use-context NAME switches, and the safest habit is to show the current context in your shell prompt. Copy the file to ~/.kube/config on your own workstation (and replace 127.0.0.1 in it with the server's address) to use the cluster from there. The kubeconfig is a credential: anyone with it is as powerful as it says.
The API itself is just HTTPS. kubectl is one client, and curl works the same way (with the credentials). Ask what kinds of objects this cluster knows:
$ kubectl api-resources | head -12
NAME SHORTNAMES APIVERSION NAMESPACED KIND
bindings v1 true Binding
componentstatuses cs v1 false ComponentStatus
configmaps cm v1 true ConfigMap
endpoints ep v1 true Endpoints
events ev v1 true Event
limitranges limits v1 true LimitRange
namespaces ns v1 false Namespace
nodes no v1 false Node
persistentvolumeclaims pvc v1 true PersistentVolumeClaim
persistentvolumes pv v1 false PersistentVolume
pods po v1 true Pod
$ kubectl api-resources | wc -l
105
$ kubectl get --raw /version | head -5
{
"major": "1",
"minor": "35",
"emulationMajor": "1",
"emulationMinor": "35",
Pods, ReplicaSets and Deployments
A pod runs one or more containers. The simplest thing is to start one by hand:
$ kubectl run solo --image=nginx:alpine
pod/solo created
$ kubectl get pod solo -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
solo 0/1 ContainerCreating 0 1s <none> k8w1 <none> <none>
$ kubectl delete pod solo
pod "solo" deleted from default namespace
$ kubectl get pod solo
Error from server (NotFound): pods "solo" not found
(exit status 1)
The pod ran on a node, and when it was deleted it was gone for good, because nothing owned it. That is why you almost never create pods yourself. A Deployment owns a ReplicaSet, which owns the pods and keeps the count right:
$ kubectl create deployment web --image=nginx:alpine --replicas=2
deployment.apps/web created
$ kubectl get deploy,rs,pods -o wide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/web 2/2 2 2 2s nginx nginx:alpine app=web
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
replicaset.apps/web-5d4f7c4d94 2 2 2 2s nginx nginx:alpine app=web,pod-template-hash=5d4f7c4d94
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/web-5d4f7c4d94-hpdcg 1/1 Running 0 2s 10.42.0.8 k8s <none> <none>
pod/web-5d4f7c4d94-qpsp7 1/1 Running 0 2s 10.42.1.5 k8w1 <none> <none>
$ kubectl get pod -l app=web -o jsonpath='{range .items[*]}{.metadata.name}{" owned by "}{.metadata.ownerReferences[0].kind}{"/"}{.metadata.ownerReferences[0].name}{"\n"}{end}'
web-5d4f7c4d94-hpdcg owned by ReplicaSet/web-5d4f7c4d94
web-5d4f7c4d94-qpsp7 owned by ReplicaSet/web-5d4f7c4d94
The chain is deployment web, then ReplicaSet web-HASH, then pods web-HASH-SUFFIX, and each object records its owner. Now delete a pod, and look at what happens to the count:
$ kubectl delete pod -l app=web --wait=false | head -2
pod "web-5d4f7c4d94-hpdcg" deleted from default namespace
pod "web-5d4f7c4d94-qpsp7" deleted from default namespace
$ kubectl get pods -l app=web
NAME READY STATUS RESTARTS AGE
web-5d4f7c4d94-d4g7n 0/1 ContainerCreating 0 0s
web-5d4f7c4d94-hpdcg 1/1 Terminating 0 3s
web-5d4f7c4d94-qpsp7 1/1 Terminating 0 3s
web-5d4f7c4d94-xzpgl 0/1 ContainerCreating 0 0s
$ kubectl get pods -l app=web
NAME READY STATUS RESTARTS AGE
web-5d4f7c4d94-d4g7n 1/1 Running 0 6s
web-5d4f7c4d94-xzpgl 1/1 Running 0 6s
Two new pods took their place within seconds: the ReplicaSet saw "2 wanted, 0 running" and made new ones. This is the loop that Kubernetes is built from: compare what is wanted with what exists, and act on the difference. Scale by changing the number you want:
$ kubectl scale deployment web --replicas=4
deployment.apps/web scaled
$ kubectl get pods -l app=web -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
web-5d4f7c4d94-d4g7n 1/1 Running 0 8s 10.42.1.6 k8w1 <none> <none>
web-5d4f7c4d94-sc6p2 1/1 Running 0 1s 10.42.0.10 k8s <none> <none>
web-5d4f7c4d94-xzpgl 1/1 Running 0 8s 10.42.0.9 k8s <none> <none>
web-5d4f7c4d94-zcl7c 1/1 Running 0 1s 10.42.1.7 k8w1 <none> <none>
$ kubectl scale deployment web --replicas=2
deployment.apps/web scaled
$ kubectl get deploy web
NAME READY UP-TO-DATE AVAILABLE AGE
web 2/2 2 2 12s
The pods were spread over the two nodes by the scheduler (the NODE column). The labels app=web on the pods are what the ReplicaSet and, later, a Service use to find them:
$ kubectl get pods --show-labels | head -3
NAME READY STATUS RESTARTS AGE LABELS
web-5d4f7c4d94-d4g7n 1/1 Running 0 9s app=web,pod-template-hash=5d4f7c4d94
web-5d4f7c4d94-sc6p2 1/1 Terminating 0 2s app=web,pod-template-hash=5d4f7c4d94
$ kubectl get pods -l app=web --no-headers | wc -l
4
$ kubectl get pods -l 'app in (web,other)' --no-headers | wc -l
4
Manifests: declare the state and apply it
kubectl create deployment is good for a first look. Real work is kept as YAML files in git, and applied. A manifest has four top-level parts: apiVersion and kind (what sort of object), metadata (its name, namespace and labels) and spec (what you want).
$ cat shop.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: shop
labels:
app: shop
spec:
replicas: 2
selector:
matchLabels:
app: shop
template:
metadata:
labels:
app: shop
spec:
containers:
- name: nginx
image: nginx:1.27-alpine
ports:
- containerPort: 80
$ kubectl apply -f shop.yaml
deployment.apps/shop created
$ kubectl get deploy shop
NAME READY UP-TO-DATE AVAILABLE AGE
shop 2/2 2 2 1s
Note that spec.selector must match the labels in spec.template, because that is how the Deployment finds its pods. apply is declarative: it compares the file with the object, and changes only the difference. Run it again, and nothing happens:
$ kubectl apply -f shop.yaml
deployment.apps/shop unchanged
Change the file, and look at what would change before you change it. kubectl diff shows a diff against the live object, and --dry-run=server sends the change to the API server to be validated (including by admission controllers) without saving it:
$ sed -i 's/replicas: 2/replicas: 3/' shop.yaml
$ kubectl diff -f shop.yaml | grep -E '^[+-] '
- generation: 1
+ generation: 2
- replicas: 2
+ replicas: 3
$ kubectl apply -f shop.yaml --dry-run=server
deployment.apps/shop configured (server dry run)
$ kubectl apply -f shop.yaml
deployment.apps/shop configured
$ kubectl get deploy shop
NAME READY UP-TO-DATE AVAILABLE AGE
shop 2/3 3 2 3s
Use kubectl explain when you do not remember a field, instead of searching the web. It reads the schema from the cluster itself, so it matches the version you run:
$ kubectl explain deployment.spec.strategy | head -14
GROUP: apps
KIND: Deployment
VERSION: v1
FIELD: strategy <DeploymentStrategy>
DESCRIPTION:
The deployment strategy to use to replace existing pods with new ones.
DeploymentStrategy describes how to replace existing pods with new ones.
FIELDS:
rollingUpdate <RollingUpdateDeployment>
Rolling update config params. Present only if DeploymentStrategyType =
$ kubectl explain pod.spec.containers.resources --recursive | head -8
KIND: Pod
VERSION: v1
FIELD: resources <ResourceRequirements>
DESCRIPTION:
Compute Resources required by this container. Cannot be updated. More info:
A validation error is caught before anything changes. Spell a field wrong:
$ sed 's/name: shop/name: shop2/; s/replicas: 3/replicas: three/' shop.yaml | kubectl apply -f - 2>&1 | head -3 | cut -c1-170
Error from server (BadRequest): error when creating "STDIN": Deployment in version "v1" cannot be handled as a Deployment: json: cannot unmarshal string into Go struct fi
$ sed 's/kind: Deployment/kind: Deployent/' shop.yaml | kubectl apply -f - 2>&1 | head -3
error: resource mapping not found for name: "shop" namespace: "" from "STDIN": no matches for kind "Deployent" in version "apps/v1"
ensure CRDs are installed first
The differences between the commands that write:
kubectl create -f fails if the object already exists. For things you make once.
kubectl apply -f creates, or changes only what differs. The normal way.
kubectl replace -f swaps the whole object for the file (it must exist).
kubectl delete -f deletes what the file describes.
kubectl apply -k DIR applies a kustomization (a later section).
The file is the truth, not the cluster. Anything changed by hand with kubectl edit or scale and not put back in the file is lost at the next apply of the old file, or worse, kept and forgotten. Keep the YAML in git, and change the cluster only by applying it. Delete what the file describes:
$ kubectl delete -f shop.yaml
deployment.apps "shop" deleted from default namespace
$ kubectl delete deployment web
deployment.apps "web" deleted from default namespace
Rolling updates and rollbacks
When the pod template changes (a new image, a new environment variable), a Deployment replaces its pods gradually, so the application stays up. Two numbers control the pace: maxSurge (how many extra pods may exist during the update) and maxUnavailable (how many may be missing).
$ cat site.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: site
spec:
replicas: 4
selector:
matchLabels:
app: site
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
template:
metadata:
labels:
app: site
spec:
containers:
- name: nginx
image: nginx:1.27-alpine
$ kubectl apply -f site.yaml
deployment.apps/site created
$ kubectl get deploy site
NAME READY UP-TO-DATE AVAILABLE AGE
site 4/4 4 4 1s
maxUnavailable 0 and maxSurge 1 mean: start one new pod, wait until it is ready, then stop one old one, and repeat. Never fewer than 4 pods ready. Start an update and watch it:
$ kubectl set image deployment/site nginx=nginx:1.28-alpine
deployment.apps/site image updated
$ kubectl rollout status deployment site --timeout=120s | tail -2
Waiting for deployment "site" rollout to finish: 1 old replicas are pending termination...
deployment "site" successfully rolled out
$ kubectl get rs -l app=site
NAME DESIRED CURRENT READY AGE
site-557fdfc647 4 4 4 3s
site-fc4cccc76 0 0 0 5s
$ kubectl rollout history deployment site
deployment.apps/site
REVISION CHANGE-CAUSE
1 <none>
2 <none>
The old ReplicaSet is kept at zero pods. That is what makes a rollback instant: it is only changing the counts back. Now an update that cannot work, because the image tag does not exist:
$ kubectl set image deployment/site nginx=nginx:no-such-tag
deployment.apps/site image updated
$ kubectl get pods -l app=site
NAME READY STATUS RESTARTS AGE
site-557fdfc647-bk7zl 1/1 Running 0 19s
site-557fdfc647-gtdrt 1/1 Running 0 17s
site-557fdfc647-h5dc6 1/1 Running 0 18s
site-557fdfc647-sld8h 1/1 Running 0 19s
site-65f5cc8fcb-fv2df 0/1 ImagePullBackOff 0 15s
$ kubectl rollout status deployment site --timeout=10s
Waiting for deployment "site" rollout to finish: 1 out of 4 new replicas have been updated...
error: timed out waiting for the condition
(exit status 1)
The new pod is in ErrImagePull or ImagePullBackOff and will never be ready, and because maxUnavailable is 0 Kubernetes did not remove any good pod: the site is still served by the four old ones. That is the point of a rolling update. Roll back:
$ kubectl rollout undo deployment site
deployment.apps/site rolled back
$ kubectl rollout status deployment site --timeout=120s
deployment "site" successfully rolled out
$ kubectl get pods -l app=site
NAME READY STATUS RESTARTS AGE
site-557fdfc647-bk7zl 1/1 Running 0 30s
site-557fdfc647-gtdrt 1/1 Running 0 28s
site-557fdfc647-h5dc6 1/1 Running 0 29s
site-557fdfc647-sld8h 1/1 Running 0 30s
site-65f5cc8fcb-fv2df 0/1 Terminating 0 26s
$ kubectl get deployment site -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
nginx:1.28-alpine
Two settings you will want to know. minReadySeconds makes a pod count as ready only after it has run that long without crashing. progressDeadlineSeconds (600 by default) is how long a stuck update may stay stuck before the Deployment reports "ProgressDeadlineExceeded". Neither stops the update by itself: a stuck rollout waits until you roll it back, which is why the check in a pipeline is kubectl rollout status with a timeout, and an undo on failure. Restart every pod without changing anything (to pick up a changed ConfigMap, say) with:
$ kubectl rollout restart deployment site
deployment.apps/site restarted
$ kubectl rollout status deployment site --timeout=120s | tail -1
deployment "site" successfully rolled out
$ kubectl delete deployment site
deployment.apps "site" deleted from default namespace
Services and cluster DNS
Pods come and go, and their addresses change every time. A Service gives a set of pods one stable address and one name, and spreads the traffic over whichever pods match its selector.
$ cat app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: app
spec:
replicas: 2
selector:
matchLabels:
app: app
template:
metadata:
labels:
app: app
spec:
containers:
- name: nginx
image: nginx:alpine
---
apiVersion: v1
kind: Service
metadata:
name: app
spec:
selector:
app: app
ports:
- port: 80
targetPort: 80
$ kubectl apply -f app.yaml
deployment.apps/app created
service/app created
$ kubectl get svc app
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
app ClusterIP 10.43.145.78 <none> 80/TCP 1s
$ kubectl get endpointslices -l kubernetes.io/service-name=app
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
app-rwcvt IPv4 80 10.42.0.18,10.42.1.17 2s
$ kubectl get pods -l app=app -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
app-5497957c-ljmzb 1/1 Running 0 2s 10.42.0.18 k8s <none> <none>
app-5497957c-lwfvq 1/1 Running 0 2s 10.42.1.17 k8w1 <none> <none>
The Service has a virtual address (the CLUSTER-IP), which belongs to no machine. Its endpoints are the real pod addresses, and the list is kept up to date by Kubernetes: a pod that is not ready is taken out of it. A pod reaches the Service by name, because the cluster runs DNS. Run a test pod:
$ kubectl run dns --image=busybox:1.36 --restart=Never -- sleep 3600
pod/dns created
$ kubectl exec dns -- nslookup app 2>&1 | grep -A1 '^Name'
Name: app.default.svc.cluster.local
Address: 10.43.145.78
$ kubectl exec dns -- wget -qO- http://app | grep -o '<title>.*</title>'
<title>Welcome to nginx!</title>
$ kubectl exec dns -- cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.43.0.10
options ndots:5
The other lines that nslookup printed (not shown) were NXDOMAIN answers for the search-domain guesses, such as app.svc.cluster.local, which do not exist. The short name app works because the pod's search domains include its own namespace (default.svc.cluster.local), so app is expanded to app.default.svc.cluster.local. From another namespace, use app.default or the full name. The five pieces of the name are service.namespace.svc.cluster-domain.
Types of Service:
ClusterIP the default. A virtual address reachable only inside the cluster.
NodePort also opens the same port (30000-32767) on every node. Reachable from outside at NODEIP:PORT.
LoadBalancer asks the environment for an external load balancer. In k3s, a small built-in one (svclb) opens the port on the nodes. In a cloud, a real one is made (and billed).
ExternalName a DNS alias (a CNAME) to a name outside the cluster. No proxying.
headless clusterIP: None. No virtual address: DNS returns the pod addresses themselves. Used with StatefulSets (a later section).
Make the Service a NodePort and use it from the worker node, which is not where all the pods are:
$ kubectl patch svc app -p '{"spec":{"type":"NodePort","ports":[{"port":80,"nodePort":30080}]}}'
service/app patched
$ kubectl get svc app
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
app NodePort 10.43.145.78 <none> 80:30080/TCP 4s
$ curl -s -o /dev/null -w 'HTTP %{http_code}\n' http://k8s:30080 # run as root on the worker node
HTTP 200
The worker node reached the Service through the other node's address. NodePort opens the port on every node, whichever node the pods are on. A Service that finds no pods, because the selector and the labels disagree, is the most common networking fault, and the sign is an empty endpoint list:
$ kubectl patch svc app -p '{"spec":{"selector":{"app":"typo"}}}'
service/app patched
$ kubectl get endpointslices -l kubernetes.io/service-name=app
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
app-rwcvt IPv4 <unset> <unset> 5s
$ kubectl exec dns -- wget -qO- -T 3 http://app 2>&1 | head -1
wget: can't connect to remote host (10.43.133.144): Connection refused
$ kubectl patch svc app -p '{"spec":{"selector":{"app":"app"},"type":"ClusterIP"}}'
service/app patched
$ kubectl get svc app
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
app ClusterIP 10.43.145.78 <none> 80/TCP 12s
Ingress: one front door for many sites
A Service of type NodePort or LoadBalancer opens one port per Service. For websites that is wasteful: you want one address on ports 80 and 443 that sends app.example.com to one Service and blog.example.com to another. That is what an Ingress is. It is a set of rules, and an ingress controller (a program running in the cluster, here Traefik, which k3s installs) reads them and does the routing. An Ingress without a controller does nothing at all.
$ kubectl get ingressclass
NAME CONTROLLER PARAMETERS AGE
traefik traefik.io/ingress-controller <none> 74s
$ kubectl get pods -n kube-system -l app.kubernetes.io/name=traefik
NAME READY STATUS RESTARTS AGE
traefik-5869c87968-fszlq 1/1 Running 0 74s
$ kubectl get svc -n kube-system traefik
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
traefik LoadBalancer 10.43.216.143 172.25.0.2,172.25.0.3 80:32521/TCP,443:31548/TCP 74s
Traefik is exposed as a LoadBalancer Service, and in k3s the small svclb pods on each node answer on ports 80 and 443 of the node itself (the EXTERNAL-IP column shows the node addresses). The Service named app is still there from the last section. Add a second site, and the Ingress that routes by host name:
$ cat sites.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: blog
spec:
replicas: 1
selector:
matchLabels:
app: blog
template:
metadata:
labels:
app: blog
spec:
containers:
- name: echo
image: hashicorp/http-echo:1.0
args: ["-text=this is the blog"]
---
apiVersion: v1
kind: Service
metadata:
name: blog
spec:
selector:
app: blog
ports:
- port: 80
targetPort: 5678
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: sites
spec:
rules:
- host: app.lab
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: app
port:
number: 80
- host: blog.lab
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: blog
port:
number: 80
$ kubectl apply -f sites.yaml
deployment.apps/blog created
service/blog created
ingress.networking.k8s.io/sites created
$ kubectl get ingress sites
NAME CLASS HOSTS ADDRESS PORTS AGE
sites traefik app.lab,blog.lab 172.25.0.2,172.25.0.3 80 7s
The ADDRESS column fills in with the node addresses once Traefik has picked the Ingress up. Now ask the front door (port 80 of this node) for each site. There is no DNS for app.lab and blog.lab in the lab, so the Host header is sent by hand, which is exactly what a browser does with a real name:
$ curl -s -o /dev/null -w 'no host name: HTTP %{http_code}\n' http://localhost/
no host name: HTTP 404
$ curl -s -H 'Host: app.lab' http://localhost/ | grep -o '<title>.*</title>'
<title>Welcome to nginx!</title>
$ curl -s -H 'Host: blog.lab' http://localhost/
this is the blog
$ curl -s -H 'Host: blog.lab' http://k8s/ # run as root on the worker node
this is the blog
The first answer is a 404 from Traefik, because no rule matches a request for no host. The next two reach different Services through the same port. The last one came from the worker node. Any node answers, because every node runs the front door.
For real names, point the DNS records of app.example.com and blog.example.com at the node (or load balancer) addresses. HTTPS needs a certificate. The usual way is cert-manager with Let's Encrypt, and the Ingress then carries a tls: section naming a Secret. That was not run in the lab, since it needs real DNS names and a reachable address. Traefik and most other controllers read extra settings from annotations on the Ingress, and those differ from controller to controller, so keep to the standard fields where you can. The newer Gateway API (HTTPRoute and friends) is replacing Ingress, and was not run either.
$ kubectl delete -f sites.yaml
deployment.apps "blog" deleted from default namespace
service "blog" deleted from default namespace
ingress.networking.k8s.io "sites" deleted from default namespace
$ kubectl delete -f app.yaml
deployment.apps "app" deleted from default namespace
service "app" deleted from default namespace
$ kubectl delete pod dns --wait=false
pod "dns" deleted from default namespace
ConfigMaps and Secrets
A container image should not carry settings that change between environments. A ConfigMap holds non-secret configuration (a file, a few keys), and a Secret holds the confidential kind (passwords, tokens, keys). Both are objects in the API, and a pod gets them as environment variables or as files.
$ kubectl create configmap app-config --from-literal=GREETING=hello --from-literal=LEVEL=info
configmap/app-config created
$ cat settings.conf
mode=production
retries=3
$ kubectl create configmap app-files --from-file=settings.conf
configmap/app-files created
$ kubectl get configmap app-config -o yaml | sed -n '1,10p'
apiVersion: v1
data:
GREETING: hello
LEVEL: info
kind: ConfigMap
metadata:
creationTimestamp: "2026-10-19T21:29:51Z"
name: app-config
namespace: default
resourceVersion: "1792"
A pod uses the first as environment variables (every key becomes a variable) and the second as a mounted directory (every key becomes a file):
$ cat config-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: cfg
spec:
containers:
- name: main
image: busybox:1.36
command: ["sh", "-c", "sleep 3600"]
envFrom:
- configMapRef:
name: app-config
volumeMounts:
- name: files
mountPath: /etc/app
volumes:
- name: files
configMap:
name: app-files
$ kubectl apply -f config-pod.yaml
pod/cfg created
$ kubectl exec cfg -- sh -c 'echo "GREETING=$GREETING LEVEL=$LEVEL"; ls /etc/app; cat /etc/app/settings.conf'
GREETING=hello LEVEL=info
settings.conf
mode=production
retries=3
Change the ConfigMap, and look at which of the two sees the change:
$ kubectl patch configmap app-config -p '{"data":{"GREETING":"goodbye"}}'
configmap/app-config patched
$ echo 'mode=staging' > settings.conf && kubectl create configmap app-files --from-file=settings.conf --dry-run=client -o yaml | kubectl apply -f -
Warning: resource configmaps/app-files is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
configmap/app-files configured
$ kubectl exec cfg -- sh -c 'echo "GREETING=$GREETING"; cat /etc/app/settings.conf'
GREETING=hello
mode=staging
The environment variable still says hello, and the file shows the new value. Environment variables are fixed when the container starts, and a changed ConfigMap reaches them only when the pod is recreated (kubectl rollout restart for a Deployment). The mounted files are refreshed by the kubelet in the background, after a delay (a minute or so: 63 seconds in a separate timing test, and the example above waited for it). A file that is mounted with subPath is never refreshed. Many applications do not re-read their files anyway, so a restart is the dependable way in both cases.
Secrets are used the same way. Create one, and look at it:
$ kubectl create secret generic db --from-literal=username=app --from-literal=password='S3cr3t-lab-only'
secret/db created
$ kubectl get secret db
NAME TYPE DATA AGE
db Opaque 2 0s
$ kubectl get secret db -o yaml | sed -n '1,8p'
apiVersion: v1
data:
password: UzNjcjN0LWxhYi1vbmx5
username: YXBw
kind: Secret
metadata:
creationTimestamp: "2026-10-19T21:31:12Z"
name: db
$ kubectl get secret db -o jsonpath='{.data.password}' | base64 -d; echo
S3cr3t-lab-only
The values are base64 encoded, which is a way to carry binary data and is not encryption. Anyone who can read the Secret object can read the password in one command. What protects Secrets is who may read them (RBAC, a later section) and whether the database underneath them is encrypted:
$ k3s secrets-encrypt status # run as root on the server node
Encryption Status: Disabled, no configuration file found
In k3s, encryption of Secrets at rest in the database is off by default, as the status shows, and it is turned on with the server flag --secrets-encryption (not run in the lab). In other distributions it is a setting of the API server (an EncryptionConfiguration) or the cloud provider's key service. Even with it on, the Secret is plain text inside the pod, and anyone with exec access to the pod can read it.
Use the Secret in a pod as variables from one key, or as files:
$ cat secret-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: sec
spec:
containers:
- name: main
image: busybox:1.36
command: ["sh", "-c", "sleep 3600"]
env:
- name: DB_USER
valueFrom:
secretKeyRef:
name: db
key: username
volumeMounts:
- name: creds
mountPath: /etc/db
readOnly: true
volumes:
- name: creds
secret:
secretName: db
defaultMode: 0400
$ kubectl apply -f secret-pod.yaml
pod/sec created
$ kubectl exec sec -- sh -c 'echo "DB_USER=$DB_USER"; ls -l /etc/db; cat /etc/db/password; echo'
DB_USER=app
total 0
lrwxrwxrwx 1 root root 15 Oct 19 21:31 password -> ..data/password
lrwxrwxrwx 1 root root 15 Oct 19 21:31 username -> ..data/username
S3cr3t-lab-only
The files in the listing are links into a hidden directory that Kubernetes swaps in one step when the Secret changes. Files from a Secret are held in memory (tmpfs) on the node and not written to its disk. Never put a Secret in a manifest that goes into git. The usual answers are sealed secrets (Sealed Secrets), an operator that reads from a secrets manager (External Secrets, Vault), or SOPS-encrypted files, and none of them were run in the lab. Clean up:
$ kubectl delete pod cfg sec --wait=false
pod "cfg" deleted from default namespace
pod "sec" deleted from default namespace
$ kubectl delete configmap app-config app-files
configmap "app-config" deleted from default namespace
configmap "app-files" deleted from default namespace
$ kubectl delete secret db
secret "db" deleted from default namespace
Storage: volumes, claims and what survives
A container's own file system disappears with the container. Data that must survive lives in a volume. For durable data the pod does not name a disk. It asks for storage with a PersistentVolumeClaim (PVC): "10 GB, readable and writable by one node", and the cluster finds or makes a PersistentVolume (PV) that fits, through a StorageClass. The same manifest then works on a laptop, in a data centre and in a cloud, with different StorageClasses behind it.
$ kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 3m3s
$ kubectl get pv,pvc
No resources found
k3s ships the local-path provisioner: a PV is a directory on the node's disk. It is not replicated, so the data lives and dies with that node, which is fine for a lab and wrong for a database you care about (use a network or cloud disk class for that, not run here). Make a claim:
$ cat claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 1Gi
$ kubectl apply -f claim.yaml
persistentvolumeclaim/data created
$ kubectl get pvc data
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
data Pending local-path <unset> 3s
The claim is Pending, and that is correct: the StorageClass has volumeBindingMode WaitForFirstConsumer, which means the volume is only made once a pod that uses the claim is scheduled, so it can be put on that pod's node. Start a pod that uses it:
$ cat writer.yaml
apiVersion: v1
kind: Pod
metadata:
name: writer
spec:
containers:
- name: main
image: busybox:1.36
command: ["sh", "-c", "echo 'written by the first pod' > /data/note.txt; sleep 3600"]
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: data
$ kubectl apply -f writer.yaml
pod/writer created
$ kubectl get pvc data
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
data Bound pvc-dc67754a-7e9b-4e5f-b418-ea792fa94803 1Gi RWO local-path <unset> 9s
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
pvc-dc67754a-7e9b-4e5f-b418-ea792fa94803 1Gi RWO Delete Bound default/data local-path <unset> 3s
$ kubectl get pod writer -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
writer 1/1 Running 0 5s 10.42.1.22 k8w1 <none> <none>
The claim is Bound to a new volume. Now delete the pod, and start another one that mounts the same claim:
$ kubectl delete pod writer
pod "writer" deleted from default namespace
$ cat reader.yaml
apiVersion: v1
kind: Pod
metadata:
name: reader
spec:
containers:
- name: main
image: busybox:1.36
command: ["sh", "-c", "sleep 3600"]
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: data
$ kubectl apply -f reader.yaml
pod/reader created
$ kubectl exec reader -- cat /data/note.txt
written by the first pod
$ kubectl get pod reader -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
reader 1/1 Running 0 1s 10.42.1.23 k8w1 <none> <none>
The file survived, and the new pod ran on the same node as the old one, because a local-path volume exists on one node only. The PV says so (nodeAffinity), and the pod can only be scheduled where the data is. The directory on that node:
$ kubectl get pv -o jsonpath='{.items[0].spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]}{"\n"}'
k8w1
$ kubectl get pv -o jsonpath='{.items[0].spec.local.path}{.items[0].spec.hostPath.path}{"\n"}'
/var/lib/rancher/k3s/storage/pvc-dc67754a-7e9b-4e5f-b418-ea792fa94803_default_data
Access modes say how many nodes may mount the volume at the same time: ReadWriteOnce (one node), ReadOnlyMany, ReadWriteMany (needs a file system such as NFS or CephFS, not run here) and ReadWriteOncePod (one pod). What happens to the data when the claim is deleted is the reclaim policy of the StorageClass. For local-path it is Delete:
$ kubectl get storageclass local-path -o jsonpath='{.reclaimPolicy}{"\n"}'
Delete
$ kubectl delete pod reader
pod "reader" deleted from default namespace
$ kubectl delete pvc data
persistentvolumeclaim "data" deleted from default namespace
$ kubectl get pv,pvc
No resources found
The volume and its data are gone together with the claim. For data that matters, use a class with reclaimPolicy: Retain (the PV is kept and needs a human), and back it up, because no replica exists. Deleting a Namespace deletes its claims too.
StatefulSet: pods with a name and a disk of their own
A Deployment treats pods as interchangeable cattle: any of them can be replaced by any other. Some programs are not like that, such as a database with a primary and replicas, or a cluster that needs member 0 to start first. A StatefulSet gives each pod a stable name (web-0, web-1), a stable network name through a headless Service, its own volume that follows it, and an orderly start and stop.
$ cat stateful.yaml
apiVersion: v1
kind: Service
metadata:
name: db
spec:
clusterIP: None
selector:
app: db
ports:
- port: 80
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: db
spec:
serviceName: db
replicas: 2
selector:
matchLabels:
app: db
template:
metadata:
labels:
app: db
spec:
containers:
- name: main
image: busybox:1.36
command: ["sh", "-c", "echo \"I am $(hostname)\" > /data/id; sleep 36000"]
volumeMounts:
- name: data
mountPath: /data
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 100Mi
$ kubectl apply -f stateful.yaml
service/db created
statefulset.apps/db created
$ kubectl get pods -l app=db -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
db-0 1/1 Running 0 9s 10.42.1.26 k8w1 <none> <none>
db-1 1/1 Running 0 4s 10.42.0.21 k8s <none> <none>
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
data-db-0 Bound pvc-efa8129f-90bf-4055-98c7-051cfa4b59db 100Mi RWO local-path <unset> 9s
data-db-1 Bound pvc-54b2833c-71e5-4522-b797-5c258942b964 100Mi RWO local-path <unset> 4s
The pods came up in order, db-0 first, and each has its own claim named after the template and the pod (data-db-0, data-db-1). The headless Service (clusterIP: None) has no virtual address: its DNS name returns the pod addresses, and every pod also gets its own name (busybox's nslookup does not apply the search list to a name that has a dot in it, so it needs the full name, while ping and most programs use the system resolver and accept the short db-0.db):
$ kubectl run dns2 --image=busybox:1.36 --restart=Never -- sleep 3600
pod/dns2 created
$ kubectl exec dns2 -- nslookup db 2>&1 | grep -E '^(Name|Address)' | head -6
Address: 10.43.0.10:53
Name: db.default.svc.cluster.local
Address: 10.42.0.21
Name: db.default.svc.cluster.local
Address: 10.42.1.26
$ kubectl exec dns2 -- nslookup db-0.db.default.svc.cluster.local 2>&1 | grep -A1 '^Name'
Name: db-0.db.default.svc.cluster.local
Address: 10.42.1.26
$ kubectl exec dns2 -- ping -c1 db-0.db | head -1
PING db-0.db (10.42.1.26): 56 data bytes
Now delete db-0, and see what comes back:
$ kubectl exec db-0 -- cat /data/id
I am db-0
$ kubectl get pod db-0 -o jsonpath='{.status.podIP}{"\n"}'
10.42.1.26
$ kubectl delete pod db-0
pod "db-0" deleted from default namespace
$ kubectl exec db-0 -- cat /data/id
I am db-0
$ kubectl get pod db-0 -o jsonpath='{.status.podIP}{"\n"}'
10.42.1.28
The pod came back with the same name and the same disk (the file is still there), but not necessarily the same IP address, which is why you talk to db-0.db and never to a pod IP. Scaling down removes the highest number first, and keeps the volume:
$ kubectl scale statefulset db --replicas=1
statefulset.apps/db scaled
$ kubectl get pods -l app=db
NAME READY STATUS RESTARTS AGE
db-0 1/1 Running 0 7s
db-1 1/1 Terminating 0 45s
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
data-db-0 Bound pvc-efa8129f-90bf-4055-98c7-051cfa4b59db 100Mi RWO local-path <unset> 50s
data-db-1 Bound pvc-54b2833c-71e5-4522-b797-5c258942b964 100Mi RWO local-path <unset> 45s
The claim data-db-1 is still there, so that scaling back up gives db-1 its data again. It is also how a forgotten claim keeps costing money in a cloud: when you remove a StatefulSet for good, delete its claims too.
$ kubectl delete statefulset db
statefulset.apps "db" deleted from default namespace
$ kubectl delete pvc -l app=db
persistentvolumeclaim "data-db-0" deleted from default namespace
persistentvolumeclaim "data-db-1" deleted from default namespace
$ kubectl delete svc db
service "db" deleted from default namespace
$ kubectl delete pod dns2 --wait=false
pod "dns2" deleted from default namespace
A StatefulSet gives the identity and the disks. It does not give you replication, failover or backups, and a database running in one still needs an operator or the database's own clustering to do that. Updates go one pod at a time, from the highest number down, and the partition field of updateStrategy holds back an update to test it on one pod first (not run).
Jobs, CronJobs and DaemonSets
Not everything runs forever. A Job runs a task to completion, retries it if it fails, and keeps the record. A CronJob makes a Job on a schedule. A DaemonSet runs one pod on every node (log collectors, monitoring agents, node tools).
$ cat job.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: count
spec:
completions: 3
parallelism: 2
template:
spec:
restartPolicy: Never
containers:
- name: main
image: busybox:1.36
command: ["sh", "-c", "echo working on it; sleep 3; echo done"]
$ kubectl apply -f job.yaml
job.batch/count created
$ kubectl get job count
NAME STATUS COMPLETIONS DURATION AGE
count Complete 3/3 12s 12s
$ kubectl get pods -l job-name=count
NAME READY STATUS RESTARTS AGE
count-7nc5s 0/1 Completed 0 6s
count-rhmhz 0/1 Completed 0 12s
count-v9hc7 0/1 Completed 0 12s
$ kubectl logs -l job-name=count --tail=1
done
done
done
Three pods ran to completion, two at a time, and stayed (Completed) so that their logs can be read. They are removed with the Job, or by ttlSecondsAfterFinished. A Job that fails is retried up to backoffLimit times, with growing pauses, and then marked failed:
$ cat failing.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: broken
spec:
backoffLimit: 2
template:
spec:
restartPolicy: Never
containers:
- name: main
image: busybox:1.36
command: ["sh", "-c", "echo trying; exit 1"]
$ kubectl apply -f failing.yaml
job.batch/broken created
$ kubectl get job broken
NAME STATUS COMPLETIONS DURATION AGE
broken Failed 0/1 32s 32s
$ kubectl get pods -l job-name=broken
NAME READY STATUS RESTARTS AGE
broken-kkgh5 0/1 Error 0 23s
broken-nrp2h 0/1 Error 0 3s
broken-xsds9 0/1 Error 0 33s
$ kubectl get job broken -o jsonpath='{.status.conditions[0].reason}{": "}{.status.conditions[0].message}{"\n"}'
BackoffLimitExceeded: Job has reached the specified backoff limit
Three pods (the first try and two retries) all failed, and the Job gave up with BackoffLimitExceeded. Now a CronJob. It makes a Job on a schedule written like crontab (minute, hour, day of month, month, day of week), in UTC unless timeZone is set:
$ cat cron.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: hello
spec:
schedule: "* * * * *"
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 2
failedJobsHistoryLimit: 1
jobTemplate:
spec:
template:
spec:
restartPolicy: Never
containers:
- name: main
image: busybox:1.36
command: ["sh", "-c", "date; echo hello from a cron job"]
$ kubectl apply -f cron.yaml
cronjob.batch/hello created
$ kubectl create job manual --from=cronjob/hello
job.batch/manual created
$ kubectl get cronjob hello
NAME SCHEDULE TIMEZONE SUSPEND ACTIVE LAST SCHEDULE AGE
hello * * * * * <none> False 0 58s 71s
$ kubectl get jobs
NAME STATUS COMPLETIONS DURATION AGE
broken Failed 0/1 104s 104s
count Complete 3/3 12s 117s
hello-29874095 Complete 1/1 3s 58s
manual Complete 1/1 3s 71s
$ kubectl logs job/manual
Mon Oct 19 21:34:48 UTC 2026
hello from a cron job
kubectl create job --from=cronjob/NAME runs a CronJob once now, which is the way to test it. After a minute the schedule itself made a Job (named hello-NUMBER). concurrencyPolicy: Forbid skips a run if the last one is still going, and the two history limits keep the number of old Jobs down. A CronJob that is suspended (spec.suspend: true) makes nothing. If the cluster was down at the time, a run that was missed by more than startingDeadlineSeconds is skipped, so a critical schedule needs that set and the job should be able to catch up.
A DaemonSet has no replica count, because the count is the number of matching nodes:
$ cat daemon.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: agent
spec:
selector:
matchLabels:
app: agent
template:
metadata:
labels:
app: agent
spec:
containers:
- name: main
image: busybox:1.36
command: ["sh", "-c", "echo agent on $NODE; sleep 36000"]
env:
- name: NODE
valueFrom:
fieldRef:
fieldPath: spec.nodeName
$ kubectl apply -f daemon.yaml
daemonset.apps/agent created
$ kubectl get daemonset agent
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
agent 2 2 2 2 2 <none> 1s
$ kubectl get pods -l app=agent -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
agent-2rg4p 1/1 Running 0 2s 10.42.1.34 k8w1 <none> <none>
agent-bq4zb 1/1 Running 0 2s 10.42.0.27 k8s <none> <none>
$ kubectl logs -l app=agent --tail=1
agent on k8w1
agent on k8s
One pod ran on each of the two nodes, and a third node that joined later would get one automatically. Clean up:
$ kubectl delete job count broken manual
job.batch "count" deleted from default namespace
job.batch "broken" deleted from default namespace
job.batch "manual" deleted from default namespace
$ kubectl delete cronjob hello
cronjob.batch "hello" deleted from default namespace
$ kubectl delete daemonset agent
daemonset.apps "agent" deleted from default namespace
Requests, limits and what happens when they are wrong
Every container can say two things about CPU and memory. A request is the amount it needs: the scheduler places the pod on a node that has that much unclaimed, and the kubelet reserves it. A limit is the most it may use: past it, CPU is throttled, and memory use is fatal (the kernel kills the process). The two together decide the pod's quality of service (QoS) class, which decides who is killed first when a node runs out of memory:
Guaranteed every container has requests equal to limits (for CPU and memory). Killed last.
Burstable at least one request or limit is set, but not as above. The common case.
BestEffort nothing set. Killed first, and the scheduler counts it as asking for nothing.
Make one of each and look at the class Kubernetes gave it:
$ cat qos.yaml
apiVersion: v1
kind: Pod
metadata:
name: guaranteed
spec:
containers:
- name: main
image: busybox:1.36
command: ["sleep", "3600"]
resources:
requests: {cpu: 100m, memory: 64Mi}
limits: {cpu: 100m, memory: 64Mi}
---
apiVersion: v1
kind: Pod
metadata:
name: burstable
spec:
containers:
- name: main
image: busybox:1.36
command: ["sleep", "3600"]
resources:
requests: {cpu: 50m, memory: 32Mi}
---
apiVersion: v1
kind: Pod
metadata:
name: besteffort
spec:
containers:
- name: main
image: busybox:1.36
command: ["sleep", "3600"]
$ kubectl apply -f qos.yaml
pod/guaranteed created
pod/burstable created
pod/besteffort created
$ kubectl get pods guaranteed burstable besteffort -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass
NAME QOS
guaranteed Guaranteed
burstable Burstable
besteffort BestEffort
CPU is measured in cores, and 100m means a tenth of a core ("millicores"). Memory is in bytes (Mi is mebibytes). Now a limit that is too low. This container allocates about 50 MB, and its limit is 20 MiB:
$ cat oom.yaml
apiVersion: v1
kind: Pod
metadata:
name: oom
spec:
restartPolicy: OnFailure
containers:
- name: main
image: busybox:1.36
command: ["sh", "-c", "x=$(head -c 50000000 /dev/zero | tr '\\0' a); sleep 3600"]
resources:
limits:
memory: 20Mi
$ kubectl apply -f oom.yaml
pod/oom created
$ kubectl get pod oom
NAME READY STATUS RESTARTS AGE
oom 0/1 OOMKilled 2 (23s ago) 25s
$ kubectl get pod oom -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}{" exit code "}{.status.containerStatuses[0].lastState.terminated.exitCode}{"\n"}'
OOMKilled exit code 137
$ kubectl describe pod oom | grep -E 'Reason:|Exit Code|Restart Count|Limits|memory'
Reason: OOMKilled
Exit Code: 137
Reason: OOMKilled
Exit Code: 137
Restart Count: 2
Limits:
memory: 20Mi
memory: 20Mi
OOMKilled with exit code 137 (128 plus the signal 9) is the memory limit doing its job, and the pod restarts, grows, and is killed again, with growing pauses (CrashLoopBackOff). The fix is a bigger limit, or fixing the program. A CPU limit never kills, it only slows the container down.
The other side is a request that cannot be met. Ask for more CPU than any node has:
$ cat big.yaml
apiVersion: v1
kind: Pod
metadata:
name: big
spec:
containers:
- name: main
image: busybox:1.36
command: ["sleep", "3600"]
resources:
requests:
cpu: "64"
$ kubectl apply -f big.yaml
pod/big created
$ kubectl get pod big
NAME READY STATUS RESTARTS AGE
big 0/1 Pending 0 5s
$ kubectl describe pod big | grep -A3 -E '^Events:'
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 5s default-scheduler 0/2 nodes are available: 2 Insufficient cpu. no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
The pod is Pending, and the event says FailedScheduling and why ("Insufficient cpu"). A pod stays Pending until something changes, with no error, so Pending is the status to look at first when something does not start. How much room each node has, and how much of it is claimed by requests:
$ kubectl describe node k8w1 | grep -A8 'Allocated resources'
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 150m (1%) 100m (1%)
memory 116Mi (0%) 84Mi (0%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
$ kubectl top nodes
NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%)
k8s 712m 8% 5764Mi 18%
k8w1 684m 8% 5765Mi 18%
$ kubectl top pods -A --sort-by=memory | head -5
NAMESPACE NAME CPU(cores) MEMORY(bytes)
kube-system metrics-server-7f4b6d9bd7-mcfwk 8m 22Mi
kube-system traefik-5869c87968-fszlq 1m 18Mi
kube-system coredns-b644fd95b-rfzv8 3m 14Mi
kube-system local-path-provisioner-d9d8769b6-582zr 1m 9Mi
The numbers in "Allocated resources" are requests and limits, which is what the scheduler counts. kubectl top shows what is really being used, from metrics-server. A node can be full of requests and nearly idle, or the other way round, and both are problems. Set requests from what the program really uses (look at kubectl top over a week), set a memory limit a little above the peak, and think hard before setting a CPU limit (throttling can slow a program that has plenty of idle CPU on the node).
$ kubectl delete pod guaranteed burstable besteffort oom big --wait=false
pod "guaranteed" deleted from default namespace
pod "burstable" deleted from default namespace
pod "besteffort" deleted from default namespace
pod "oom" deleted from default namespace
pod "big" deleted from default namespace
Probes: tell Kubernetes when a container is alive and ready
A container that is running is not necessarily working. Three probes let Kubernetes ask, and each answers a different question:
readiness "can it take traffic now?" If it fails, the pod is removed from the Service's endpoints, and not restarted.
liveness "is it stuck for good?" If it fails, the container is restarted.
startup "has it finished starting?" Holds off the other two for slow starters.
A probe is a command to run in the container, an HTTP request, or a TCP connection. This pod is ready only while /tmp/ready exists, and alive only while /tmp/alive exists, and a Service selects it:
$ cat probes.yaml
apiVersion: v1
kind: Pod
metadata:
name: probed
labels:
app: probed
spec:
containers:
- name: main
image: busybox:1.36
command: ["sh", "-c", "touch /tmp/alive /tmp/ready; sleep 36000"]
readinessProbe:
exec:
command: ["test", "-f", "/tmp/ready"]
periodSeconds: 3
livenessProbe:
exec:
command: ["test", "-f", "/tmp/alive"]
periodSeconds: 3
failureThreshold: 2
---
apiVersion: v1
kind: Service
metadata:
name: probed
spec:
selector:
app: probed
ports:
- port: 80
$ kubectl apply -f probes.yaml
pod/probed created
service/probed created
$ kubectl get pod probed
NAME READY STATUS RESTARTS AGE
probed 1/1 Running 0 2s
$ kubectl get endpointslices -l kubernetes.io/service-name=probed -o jsonpath='{.items[0].endpoints[0].conditions.ready}{"\n"}'
true
Break readiness, and look at the pod and the Service:
$ kubectl exec probed -- rm /tmp/ready
$ kubectl get pod probed
NAME READY STATUS RESTARTS AGE
probed 0/1 Running 0 11s
$ kubectl get endpointslices -l kubernetes.io/service-name=probed -o jsonpath='{.items[0].endpoints[0].conditions.ready}{"\n"}'
false
$ kubectl describe pod probed | grep -E 'Readiness|Unhealthy' | head -3
Readiness: exec [test -f /tmp/ready] delay=0s timeout=1s period=3s #success=1 #failure=3
Warning Unhealthy 2s (x4 over 8s) kubelet spec.containers{main}: Readiness probe failed:
The pod is Running but 0/1 ready, and the Service no longer sends it traffic: the address is still in the endpoint list, but its ready condition is now false (the second answer above), and the old v1 Endpoints object shows an empty list. The container was not touched. That is the right response for "busy" or "warming up": take it out of rotation until it is ready again. Put the file back, and it returns:
$ kubectl exec probed -- touch /tmp/ready
$ kubectl get pod probed
NAME READY STATUS RESTARTS AGE
probed 1/1 Running 0 14s
Now break liveness. This time Kubernetes restarts the container:
$ kubectl exec probed -- rm /tmp/alive
$ kubectl get pod probed
NAME READY STATUS RESTARTS AGE
probed 1/1 Running 1 (1s ago) 49s
$ kubectl describe pod probed | grep -E 'Liveness probe failed|Killing|Restart Count' | head -4
Restart Count: 1
Warning Unhealthy 32s (x2 over 35s) kubelet spec.containers{main}: Liveness probe failed:
Normal Killing 32s kubelet spec.containers{main}: Container main failed liveness probe, will be restarted
The restart count went to 1, and the events say that the liveness probe failed and the container was killed and started again. Setting probes badly causes outages of its own, so keep to these rules:
a liveness probe checks only that the process itself is not stuck, never a dependency (if the database is down, restarting every web pod does not help, and it makes the outage bigger)
a readiness probe may check dependencies, because "not ready" only removes a pod from traffic
make probes cheap and fast, set timeoutSeconds for slow ones, and leave the failureThreshold at 3 or more so that one slow answer does not restart a pod
use a startupProbe for programs that take minutes to start, instead of a long initialDelaySeconds on the liveness probe
a pod with no readiness probe is "ready" as soon as its process starts, which sends it traffic too early
$ kubectl delete -f probes.yaml
pod "probed" deleted from default namespace
service "probed" deleted from default namespace
Scheduling: where pods go, and moving them off a node
The scheduler puts each pod on a node by two steps: it throws out the nodes that cannot run it (not enough room, a taint it does not tolerate, a label it does not match), and then it picks the best of the rest. You steer it with labels and rules.
Label the nodes, and ask for a node by label with nodeSelector. The simplest control of all:
$ kubectl label node k8w1 disk=ssd
node/k8w1 labeled
$ kubectl get nodes -L disk
NAME STATUS ROLES AGE VERSION DISK
k8s Ready control-plane,etcd 9m50s v1.35.9+k3s1
k8w1 Ready <none> 9m44s v1.35.9+k3s1 ssd
$ cat pinned.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: pinned
spec:
replicas: 3
selector:
matchLabels:
app: pinned
template:
metadata:
labels:
app: pinned
spec:
nodeSelector:
disk: ssd
containers:
- name: main
image: busybox:1.36
command: ["sleep", "3600"]
$ kubectl apply -f pinned.yaml
deployment.apps/pinned created
$ kubectl get pods -l app=pinned -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pinned-5596c6bbdf-bwzc9 1/1 Running 0 2s 10.42.1.42 k8w1 <none> <none>
pinned-5596c6bbdf-rt9dm 1/1 Running 0 2s 10.42.1.41 k8w1 <none> <none>
pinned-5596c6bbdf-xdgh2 1/1 Running 0 2s 10.42.1.40 k8w1 <none> <none>
All three pods went to the one node with the label. Richer rules exist (nodeAffinity with "prefer" and "require" and expressions, podAffinity and podAntiAffinity to keep pods together or apart), and topologySpreadConstraints, which spread replicas evenly across zones or nodes, are usually the better tool for "do not put all my replicas in one place":
$ cat spread.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: spread
spec:
replicas: 4
selector:
matchLabels:
app: spread
template:
metadata:
labels:
app: spread
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: spread
containers:
- name: main
image: busybox:1.36
command: ["sleep", "3600"]
$ kubectl apply -f spread.yaml
deployment.apps/spread created
$ kubectl get pods -l app=spread -o wide --no-headers | awk '{print $7}' | sort | uniq -c
2 k8s
2 k8w1
$ kubectl delete -f pinned.yaml -f spread.yaml
deployment.apps "pinned" deleted from default namespace
deployment.apps "spread" deleted from default namespace
maxSkew 1 means that the number of pods on any two nodes may differ by at most one: two and two. The opposite of a label is a taint. A taint on a node repels every pod that does not tolerate it, which reserves nodes for special work (GPUs, a database tier, the control plane):
$ kubectl taint node k8w1 dedicated=db:NoSchedule
node/k8w1 tainted
$ kubectl describe node k8w1 | grep Taints
Taints: dedicated=db:NoSchedule
$ kubectl create deployment plain --image=busybox:1.36 --replicas=3 -- sleep 3600
deployment.apps/plain created
$ kubectl get pods -l app=plain -o wide --no-headers | awk '{print $7}' | sort | uniq -c
3 k8s
No pod landed on k8w1. Pods that tolerate the taint may go there (and are not forced to):
$ cat tolerant.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: tolerant
spec:
replicas: 4
selector:
matchLabels:
app: tolerant
template:
metadata:
labels:
app: tolerant
spec:
tolerations:
- key: dedicated
operator: Equal
value: db
effect: NoSchedule
nodeSelector:
disk: ssd
containers:
- name: main
image: busybox:1.36
command: ["sleep", "3600"]
$ kubectl apply -f tolerant.yaml
deployment.apps/tolerant created
$ kubectl get pods -l app=tolerant -o wide --no-headers | awk '{print $7}' | sort | uniq -c
4 k8w1
$ kubectl taint node k8w1 dedicated=db:NoSchedule-
node/k8w1 untainted
$ kubectl delete deployment plain tolerant
deployment.apps "plain" deleted from default namespace
deployment.apps "tolerant" deleted from default namespace
Taint effects are NoSchedule (new pods are kept away), PreferNoSchedule (a soft version) and NoExecute (running pods that do not tolerate it are evicted as well). The same mechanism is what Kubernetes uses on its own: a node that is not ready gets a taint, and pods are moved off it after a delay.
For maintenance, take a node out of service on purpose. cordon only stops new pods, and drain also moves the existing ones, politely (they are deleted and the controllers make new ones elsewhere):
$ kubectl create deployment keep --image=nginx:alpine --replicas=4
deployment.apps/keep created
$ kubectl get pods -l app=keep -o wide --no-headers | awk '{print $7}' | sort | uniq -c
2 k8s
2 k8w1
$ kubectl cordon k8w1
node/k8w1 cordoned
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s Ready control-plane,etcd 9m59s v1.35.9+k3s1
k8w1 Ready,SchedulingDisabled <none> 9m53s v1.35.9+k3s1
$ kubectl drain k8w1 --ignore-daemonsets --delete-emptydir-data 2>&1 | tail -4
pod/tolerant-9587fdc6-kx266 evicted
pod/tolerant-9587fdc6-74dd5 evicted
pod/tolerant-9587fdc6-f48hd evicted
node/k8w1 drained
$ kubectl get pods -l app=keep -o wide --no-headers | awk '{print $7}' | sort | uniq -c
4 k8s
All four pods are now on k8s, and the worker is free to be rebooted, upgraded or replaced. drain refuses to remove pods that no controller owns (a bare pod would be lost) unless you force it, and the DaemonSet pods are left, because a DaemonSet would put them straight back. When the work is done, bring it back with uncordon. Nothing moves back by itself:
$ kubectl uncordon k8w1
node/k8w1 uncordoned
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s Ready control-plane,etcd 10m v1.35.9+k3s1
k8w1 Ready <none> 10m v1.35.9+k3s1
$ kubectl get pods -l app=keep -o wide --no-headers | awk '{print $7}' | sort | uniq -c
4 k8s
A PodDisruptionBudget limits how many pods of an application a voluntary disruption (a drain, a cluster upgrade) may take down at once. Without one, a drain can remove every replica of a service together, if they all sit on that node. With one, the drain waits:
$ cat pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: keep
spec:
minAvailable: 3
selector:
matchLabels:
app: keep
$ kubectl apply -f pdb.yaml
poddisruptionbudget.policy/keep created
$ kubectl get pdb keep
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
keep 3 N/A 1 0s
$ kubectl scale deployment keep --replicas=3
deployment.apps/keep scaled
$ kubectl get pdb keep
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
keep 3 N/A 0 0s
$ kubectl get pods -l app=keep -o wide --no-headers | awk '{print $7}' | sort | uniq -c
4 k8s
$ kubectl cordon k8s
node/k8s cordoned
$ timeout 25 kubectl drain k8s --ignore-daemonsets --delete-emptydir-data 2>&1 | grep -iE 'cannot evict|disruption budget|evicting' | sort | uniq -c | head -4
5 error when evicting pods/"keep-9459bb8ff-7mdmj" -n "default" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
5 error when evicting pods/"keep-9459bb8ff-8rf2p" -n "default" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
5 error when evicting pods/"keep-9459bb8ff-fzc98" -n "default" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
5 evicting pod default/keep-9459bb8ff-7mdmj
The drain was stopped by the budget: it tried to evict a pod, was told that doing so would leave fewer than 3 available, and kept retrying until the timeout cut it off. That is the budget protecting the service. Scale up so that the budget allows it, or relax it, and the drain goes through. Put everything back:
$ kubectl uncordon k8s
node/k8s uncordoned
$ kubectl delete -f pdb.yaml
poddisruptionbudget.policy "keep" deleted from default namespace
$ kubectl delete deployment keep
deployment.apps "keep" deleted from default namespace
Autoscaling with the HPA and metrics-server
The HorizontalPodAutoscaler (HPA) changes the number of replicas of a Deployment from a measured value, usually CPU. It needs two things: metrics (metrics-server, which k3s runs) and a CPU request on the pods, because the percentage it uses is a percentage of the request. A pod with no request gives the HPA nothing to divide by.
$ kubectl top nodes
NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%)
k8s 1401m 17% 5699Mi 17%
k8w1 1243m 15% 5704Mi 17%
$ kubectl get apiservice v1beta1.metrics.k8s.io
NAME SERVICE AVAILABLE AGE
v1beta1.metrics.k8s.io kube-system/metrics-server True 10m
$ cat hpa-app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: hpa
spec:
replicas: 1
selector:
matchLabels:
app: hpa
template:
metadata:
labels:
app: hpa
spec:
containers:
- name: main
image: registry.k8s.io/hpa-example
ports:
- containerPort: 80
resources:
requests:
cpu: 100m
limits:
cpu: 200m
---
apiVersion: v1
kind: Service
metadata:
name: hpa
spec:
selector:
app: hpa
ports:
- port: 80
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: hpa
minReplicas: 1
maxReplicas: 4
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
$ kubectl apply -f hpa-app.yaml
deployment.apps/hpa created
service/hpa created
horizontalpodautoscaler.autoscaling/hpa created
$ kubectl get hpa hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa Deployment/hpa cpu: <unknown>/50% 1 4 1 42s
The target shows the current use over the target (cpu: 1%/50% here, once the first measurement is in; <unknown> before that). Now send load to the Service from a pod in the cluster, and watch the HPA react:
$ kubectl run load --image=busybox:1.36 --restart=Never -- sh -c 'while true; do wget -q -O- http://hpa > /dev/null; done'
pod/load created
$ kubectl get hpa hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa Deployment/hpa cpu: 54%/50% 1 4 4 2m13s
$ kubectl top pods -l app=hpa
NAME CPU(cores) MEMORY(bytes)
hpa-75ff459d5b-556js 85m 12Mi
hpa-75ff459d5b-74hll 41m 11Mi
hpa-75ff459d5b-qrd9x 37m 11Mi
hpa-75ff459d5b-stxj5 56m 11Mi
$ kubectl get pods -l app=hpa
NAME READY STATUS RESTARTS AGE
hpa-75ff459d5b-556js 1/1 Running 0 58s
hpa-75ff459d5b-74hll 1/1 Running 0 58s
hpa-75ff459d5b-qrd9x 1/1 Running 0 2m13s
hpa-75ff459d5b-stxj5 1/1 Running 0 58s
The measured CPU went well over 50% of the request, so the HPA raised the replica count, to 4, which is the maximum. It checks every 15 seconds, and scales up quickly. Scaling down is slow on purpose: by default it waits 5 minutes of lower load before removing pods, so that a short dip does not cause flapping. That wait was not run in the lab. Stop the load. The measured value falls only when the next metrics window is in (about a minute), and the replicas then stay for the stabilisation window:
$ kubectl delete pod load --wait=false
pod "load" deleted from default namespace
$ kubectl get hpa hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa Deployment/hpa cpu: 5%/50% 1 4 4 3m33s
$ kubectl describe hpa hpa | grep -E 'Min replicas|Max replicas|Deployment pods|ScalingActive' | head -4
Min replicas: 1
Max replicas: 4
Deployment pods: 4 current / 4 desired
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
Notes for real use. The HPA and a fixed replicas: number in a manifest fight each other, so leave replicas out of a Deployment that has an HPA, or the next apply resets it. Scaling on CPU only suits programs whose load shows in CPU. For queue length or request rate you need custom metrics (Prometheus adapter or KEDA), which were not run. The HPA only adds pods: if the nodes are full, the new pods stay Pending, and adding nodes is the job of the cluster autoscaler or Karpenter in a cloud (not run). The vertical autoscaler, which changes requests, is a separate add-on and was not run either.
$ kubectl delete -f hpa-app.yaml
deployment.apps "hpa" deleted from default namespace
service "hpa" deleted from default namespace
horizontalpodautoscaler.autoscaling "hpa" deleted from default namespace
NetworkPolicy: who may talk to whom
By default every pod can reach every other pod in the cluster, in any namespace. A NetworkPolicy limits that. It selects pods and lists what may come in (ingress) and go out (egress). Two facts first. A policy only works if the network plug-in enforces it (k3s does, with an embedded kube-router controller, as the next test shows). And as soon as any policy selects a pod for a direction, everything else in that direction is denied for it.
$ cat np-demo.yaml
apiVersion: v1
kind: Namespace
metadata:
name: shopping
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
namespace: shopping
spec:
replicas: 1
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: nginx
image: nginx:alpine
---
apiVersion: v1
kind: Service
metadata:
name: web
namespace: shopping
spec:
selector:
app: web
ports:
- port: 80
---
apiVersion: v1
kind: Pod
metadata:
name: client
namespace: shopping
labels:
role: client
spec:
containers:
- name: main
image: busybox:1.36
command: ["sleep", "3600"]
---
apiVersion: v1
kind: Pod
metadata:
name: stranger
namespace: shopping
spec:
containers:
- name: main
image: busybox:1.36
command: ["sleep", "3600"]
$ kubectl apply -f np-demo.yaml
namespace/shopping created
deployment.apps/web created
service/web created
pod/client created
pod/stranger created
$ kubectl -n shopping exec client -- wget -qO- -T 3 http://web | grep -o '<title>.*</title>'
<title>Welcome to nginx!</title>
$ kubectl -n shopping exec stranger -- wget -qO- -T 3 http://web | grep -o '<title>.*</title>'
<title>Welcome to nginx!</title>
Both pods can reach the web server. First deny everything coming into the namespace, which is the usual starting point:
$ cat np-deny.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-ingress
namespace: shopping
spec:
podSelector: {}
policyTypes: ["Ingress"]
$ kubectl apply -f np-deny.yaml
networkpolicy.networking.k8s.io/default-deny-ingress created
$ kubectl -n shopping exec client -- wget -qO- -T 3 http://web 2>&1 | head -1
wget: can't connect to remote host (10.43.64.74): Connection refused
$ kubectl -n shopping exec stranger -- wget -qO- -T 3 http://web 2>&1 | head -1
wget: can't connect to remote host (10.43.64.74): Connection refused
podSelector: {} selects every pod in the namespace, and the policy lists no allowed sources, so nothing may reach them. Then allow only what is needed: web accepts traffic on port 80 from pods labelled role=client.
$ cat np-allow.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-client-to-web
namespace: shopping
spec:
podSelector:
matchLabels:
app: web
policyTypes: ["Ingress"]
ingress:
- from:
- podSelector:
matchLabels:
role: client
ports:
- protocol: TCP
port: 80
$ kubectl apply -f np-allow.yaml
networkpolicy.networking.k8s.io/allow-client-to-web created
$ kubectl -n shopping exec client -- wget -qO- -T 3 http://web | grep -o '<title>.*</title>'
<title>Welcome to nginx!</title>
$ kubectl -n shopping exec stranger -- wget -qO- -T 3 http://web 2>&1 | head -1
wget: can't connect to remote host (10.43.64.74): Connection refused
$ kubectl -n shopping get networkpolicy
NAME POD-SELECTOR AGE
allow-client-to-web app=web 8s
default-deny-ingress <none> 17s
The client works again and the stranger is still blocked. Policies add up: a connection is allowed if any policy allows it. Remember these points. A denied connection fails with an error from wget (the first line printed above), and which error depends on the plug-in. Egress is not limited unless you write an egress policy, and one that denies egress must also allow DNS (UDP and TCP 53 to the cluster DNS), or names stop resolving. namespaceSelector allows whole namespaces. And a policy cannot match by host name or by anything above TCP, UDP and SCTP, so it is a coarse tool: for finer control, use a service mesh or a plug-in with its own policy language (Cilium, Calico), not run in the lab.
$ kubectl delete namespace shopping
namespace "shopping" deleted
RBAC, service accounts, quotas and namespaces
Who may do what in the cluster is decided by RBAC (role-based access control). Four objects make it: a Role (a list of allowed verbs on resources, in one namespace), a ClusterRole (the same, cluster-wide), and the bindings that attach them to a user, group or service account (RoleBinding and ClusterRoleBinding). Nothing is allowed unless a role says so, and there is no "deny".
A service account is the identity a pod runs as, and what a program inside the cluster uses to call the API. Make a namespace, a service account that may only read pods in it, and test it with can-i:
$ cat rbac.yaml
apiVersion: v1
kind: Namespace
metadata:
name: team-a
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: reader
namespace: team-a
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: pod-reader
namespace: team-a
rules:
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: reader-reads-pods
namespace: team-a
subjects:
- kind: ServiceAccount
name: reader
namespace: team-a
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
$ kubectl apply -f rbac.yaml
namespace/team-a created
serviceaccount/reader created
role.rbac.authorization.k8s.io/pod-reader created
rolebinding.rbac.authorization.k8s.io/reader-reads-pods created
$ kubectl auth can-i list pods -n team-a --as=system:serviceaccount:team-a:reader
yes
$ kubectl auth can-i delete pods -n team-a --as=system:serviceaccount:team-a:reader
no
(exit status 1)
$ kubectl auth can-i list pods -n default --as=system:serviceaccount:team-a:reader
no
(exit status 1)
$ kubectl auth can-i list secrets -n team-a --as=system:serviceaccount:team-a:reader
no
(exit status 1)
$ kubectl auth can-i --list -n team-a --as=system:serviceaccount:team-a:reader | head -6
Resources Non-Resource URLs Resource Names Verbs
selfsubjectreviews.authentication.k8s.io [] [] [create]
selfsubjectaccessreviews.authorization.k8s.io [] [] [create]
selfsubjectrulesreviews.authorization.k8s.io [] [] [create]
pods/log [] [] [get list watch]
pods [] [] [get list watch]
The account may read pods in team-a and nothing else, not even the same pods in another namespace, and not Secrets. auth can-i, with --as, is how you test a role before you hand it out (it prints yes or no, and its exit status is 1 for no, which the listing above shows). Use the identity for real, with a short-lived token:
$ kubectl run sample -n team-a --image=nginx:alpine
pod/sample created
$ TOKEN=$(kubectl create token reader -n team-a --duration=10m); curl -sk -H "Authorization: Bearer $TOKEN" https://127.0.0.1:6443/api/v1/namespaces/team-a/pods | jq -r '.items[].metadata.name'
sample
$ TOKEN=$(kubectl create token reader -n team-a --duration=10m); curl -sk -H "Authorization: Bearer $TOKEN" https://127.0.0.1:6443/api/v1/namespaces/default/pods | jq -r '.reason, .message'
Forbidden
pods is forbidden: User "system:serviceaccount:team-a:reader" cannot list resource "pods" in API group "" in the namespace "default"
The first request worked and the second was refused with Forbidden, and the message names the account, the verb and the namespace. This is how to read an RBAC denial: "User X cannot VERB RESOURCE in the namespace Y" says exactly which rule to add. Kubernetes never makes a token that lives for ever by itself anymore: kubectl create token gives one that expires, and a pod's own token is refreshed and bound to that pod. Rules of thumb:
give each application its own service account, and only the verbs it needs (never use the default account, and never bind cluster-admin to an application)
prefer namespaced Roles to ClusterRoles, and groups to individual users in bindings
the verbs list, get on Secrets and create on pods/exec are as powerful as they sound: whoever has them can read credentials or run code
audit regularly: kubectl auth can-i --list --as=..., or tools such as kubectl-who-can and rbac-lookup (not run)
Users (people) are not objects in Kubernetes. They come from the client certificate in the kubeconfig or from an external identity provider (OIDC), and the lab used the certificate that k3s made for its admin. Setting up OIDC login was not run.
A namespace is also where quotas live. A ResourceQuota caps what a namespace may consume in total, and a LimitRange gives containers defaults and limits so that nothing runs without requests:
$ cat quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
name: compute
namespace: team-a
spec:
hard:
pods: "3"
requests.cpu: "1"
requests.memory: 1Gi
---
apiVersion: v1
kind: LimitRange
metadata:
name: defaults
namespace: team-a
spec:
limits:
- type: Container
defaultRequest:
cpu: 100m
memory: 64Mi
default:
cpu: 200m
memory: 128Mi
$ kubectl apply -f quota.yaml
resourcequota/compute created
limitrange/defaults created
$ kubectl delete pod sample -n team-a
pod "sample" deleted from team-a namespace
$ kubectl create deployment filler -n team-a --image=nginx:alpine --replicas=5
deployment.apps/filler created
$ kubectl get deployment filler -n team-a
NAME READY UP-TO-DATE AVAILABLE AGE
filler 3/5 3 3 15s
$ kubectl describe quota compute -n team-a | tail -5
Resource Used Hard
-------- ---- ----
pods 3 3
requests.cpu 300m 1
requests.memory 192Mi 1Gi
$ kubectl get events -n team-a --field-selector reason=FailedCreate | tail -2 | cut -c1-180
15s Warning FailedCreate replicaset/filler-65bcb6646 Error creating: pods "filler-65bcb6646-pzjgb" is forbidden: exceeded quota: compute, requested: pods=1, used: pod
14s Warning FailedCreate replicaset/filler-65bcb6646 (combined from similar events): Error creating: pods "filler-65bcb6646-5knjl" is forbidden: exceeded quota: compu
$ kubectl get pod -n team-a -o jsonpath='{.items[0].spec.containers[0].resources}{"\n"}'
{"limits":{"cpu":"200m","memory":"128Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}
The quota let three pods run and refused the other two, and the Deployment's events say "exceeded quota". The LimitRange put its default requests and limits on the pods that had none (the last line), which also means they count against the cpu and memory quota. A namespace is a boundary for names, quotas and permissions. It is not a security boundary for the network (policies do that) or for the nodes (pods of every namespace share them), so tenants that do not trust each other need more than namespaces.
$ kubectl delete namespace team-a
namespace "team-a" deleted
Operating the cluster: etcd backups and restore
Everything the cluster knows is in etcd: every object, every Secret, every ConfigMap, the state of every Deployment. The pods and their volumes are not in it, but without etcd the cluster does not know they exist. So the backup that matters is a snapshot of etcd, taken regularly and kept off the machine.
k3s takes snapshots by itself on a schedule (by default every 12 hours, keeping 5) and can take one on demand. It can only do that when it runs the embedded etcd database, which is why the lab server was started with --cluster-init. The default on a single server is SQLite, which is backed up by copying its file. Make something worth saving, take a snapshot, and then "lose" it:
$ kubectl create namespace precious
namespace/precious created
$ kubectl create deployment keepme --image=nginx:alpine -n precious
deployment.apps/keepme created
$ kubectl create configmap important -n precious --from-literal=answer=42
configmap/important created
$ k3s etcd-snapshot save --name before-disaster 2>&1 | grep -E 'saved' # run as root on the server node
time="2026-10-19T21:44:12Z" level=info msg="Snapshot before-disaster-k8s-1792446253 saved."
$ k3s etcd-snapshot ls 2>&1 | tail -2 | cut -c1-150 # run as root on the server node
Name Location Size Created
before-disaster-k8s-1792446253 file:///var/lib/rancher/k3s/server/db/snapshots/before-disaster-k8s-1792446253 14245920 2026-10-19T21:44:13Z
$ ls -l /var/lib/rancher/k3s/server/db/snapshots/ # run as root on the server node
total 13916
-rw------- 1 root root 14245920 Oct 19 21:44 before-disaster-k8s-1792446253
A snapshot is one file. Copy it to another machine (the same machine's disk does not survive the same disaster) and test restoring it regularly: a backup that has never been restored is a hope. k3s can send snapshots to S3 storage with the --etcd-s3 flags, which was not run in the lab. Now the disaster, someone deletes the namespace:
$ kubectl delete namespace precious
namespace "precious" deleted
$ kubectl get ns | grep precious || echo "namespace precious is gone"
namespace precious is gone
To restore, stop k3s on the server, run it once with the cluster-reset flags that point at the snapshot (it restores, resets the etcd membership and exits), then start it normally:
$ pkill -x k3s-server; sleep 6; pgrep -x k3s-server || echo "k3s server stopped" # run as root on the server node
k3s server stopped
$ k3s server --cluster-reset --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/$(ls /var/lib/rancher/k3s/server/db/snapshots | head -1) --node-name k8s 2>&1 | grep -E 'Managed etcd cluster membership has been reset' | cut -c1-120 # run as root on the server node
time="2026-10-19T21:44:37Z" level=error msg="Shutdown request received: \"Managed etcd cluster membership has been reset
$ kubectl get ns | grep precious
precious Active 35s
$ kubectl get deployment,configmap -n precious
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/keepme 0/1 1 0 35s
NAME DATA AGE
configmap/important 1 35s
configmap/kube-root-ca.crt 1 35s
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s Ready control-plane,etcd 16m v1.35.9+k3s1
k8w1 Ready <none> 16m v1.35.9+k3s1
The namespace, the Deployment and the ConfigMap are back, with everything else that was in the cluster at the time of the snapshot, and what was created after it is gone. While the server was down the worker kept running its pods: the data plane carries on without the control plane, you just cannot change anything. With several servers, the others must be stopped, the restore is done on one, and the rest are removed from etcd and rejoin (see the k3s documentation for the exact steps, not run in the lab). In other distributions the same job is etcdctl snapshot save and restore, or the vendor's tool. In a managed cloud cluster the provider backs up etcd and you do not get to, so back up your manifests (in git) and your volumes instead.
What a snapshot does not hold: the contents of volumes, the state of anything outside the cluster (DNS records, load balancers, cloud disks), and the node-level files such as the server token and certificates under /var/lib/rancher/k3s/server/tls (copy those too).
$ kubectl delete namespace precious
namespace "precious" deleted
Operating the cluster: certificates, nodes and the container runtime
Kubernetes uses TLS certificates between its own parts and for the admin login, and they expire. This is the classic outage of an unattended cluster: after a year, kubectl and the nodes suddenly say "x509: certificate has expired". Look at the dates:
$ k3s certificate check --output table 2>&1 | grep -E 'FILENAME|client-admin|serving-kube-apiserver|server-ca' | cut -c1-150 # run as root on the server node
FILENAME SUBJECT USAGES EXPIRES RESIDUAL TIME STATUS
kube-controller-manager.crt k3s-server-ca@1792445280 CertSign Oct 16, 2036 20:28 UTC 10 years OK
kube-scheduler.crt k3s-server-ca@1792445280 CertSign Oct 16, 2036 20:28 UTC 10 years OK
serving-kube-apiserver.crt kube-apiserver ServerAuth Oct 19, 2027 20:28 UTC 1 year OK
serving-kube-apiserver.crt k3s-server-ca@1792445280 CertSign Oct 16, 2036 20:28 UTC 10 years OK
client-admin.crt system:admin ClientAuth Oct 19, 2027 20:28 UTC 1 year OK
client-admin.crt k3s-client-ca@1792445280 CertSign Oct 16, 2036 20:28 UTC 10 years OK
client.crt etcd-server-ca@1792445280 CertSign Oct 16, 2036 20:28 UTC 10 years OK
server-client.crt etcd-server-ca@1792445280 CertSign Oct 16, 2036 20:28 UTC 10 years OK
serving-kubelet.crt k3s-server-ca@1792445280 CertSign Oct 16, 2036 20:28 UTC 10 years OK
The client and serving certificates last one year, and the certificate authorities ten years. k3s renews certificates on its own at start-up when they have less than 90 days left (from its documentation, not run in the lab), so a server that is restarted for upgrades at least once a year never runs into this. A server that is never restarted can. Put the expiry in your monitoring: k3s certificate check is one command, and the Prometheus metric apiserver_client_certificate_expiration_seconds (kube-apiserver) is another. For a kubeadm cluster the command is kubeadm certs check-expiration (not run).
Adding a worker was the join command from the second section: the same k3s agent command with the server's address and token. To take a worker out for good, do it in order. Drain it, delete the node object, and uninstall k3s on the machine:
$ kubectl drain k8w1 --ignore-daemonsets --delete-emptydir-data 2>&1 | tail -1
node/k8w1 drained
$ kubectl delete node k8w1
node "k8w1" deleted
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s Ready control-plane,etcd 17m v1.35.9+k3s1
The node object is gone from the list, and it did not come back within 40 seconds, although the k3s agent was still running on the worker machine. Deleting the object only removes the record: the machine and its agent are untouched, and the pods it was running were already moved by the drain. To retire a machine for good, also stop and remove k3s on it, using the uninstall script that the installer makes (k3s-agent-uninstall.sh, not available in the lab because the installer was not used). To bring a machine back, restart its agent. A node that joins again is a new object: it is not cordoned and has none of the labels you added. The join command is the one from the second section, with the server's address and the token that sits on the server in /var/lib/rancher/k3s/server/node-token (a credential, so treat it like a password):
$ k3s agent --server https://k8s:6443 --token TOKEN --node-name k8w1
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s Ready control-plane,etcd 18m v1.35.9+k3s1
k8w1 Ready <none> 4s v1.35.9+k3s1
On a node, kubectl is not the only tool. The container runtime can be asked directly with crictl, which shows the containers that the kubelet started, even when the API server is down. That is how you look at a node whose control plane is gone:
$ k3s crictl ps | head -4 | cut -c1-120 # run as root on the worker node
CONTAINER IMAGE CREATED STATE NAME ATTEMPT
0d1210026de4b eaa1212a2b456 2 seconds ago Running lb-tcp-443 0
e11c0ff5d6198 eaa1212a2b456 2 seconds ago Running lb-tcp-80 0
$ k3s crictl images | head -4 # run as root on the worker node
IMAGE TAG IMAGE ID SIZE
docker.io/hashicorp/http-echo 1.0 04fa556e62bdd 4.63MB
docker.io/library/busybox 1.36 b116e15507444 2.22MB
docker.io/library/nginx 1.27-alpine 6769dc3a703c7 21MB
The kubelet's own log is the first place to look when a node is NotReady, and so is the machine itself: disk space, memory, the clock and the network. In a normal install they are in journalctl -u k3s and journalctl -u k3s-agent (the lab wrote them to /var/log/k3s.log):
$ grep -c . /var/log/k3s.log # run as root on the server node
1063
$ grep -E 'level=(error|fatal)' /var/log/k3s.log | tail -2 | cut -c1-150 # run as root on the server node
time="2026-10-19T21:44:39Z" level=error msg="Sending HTTP/1.1 503 response to 127.0.0.1:33592: runtime core not ready"
time="2026-10-19T21:44:40Z" level=error msg="Sending HTTP/1.1 503 response to 127.0.0.1:33650: runtime core not ready"
The two error lines are from the moment of the restore in the previous section, when the server was starting and the API answered 503 for a few seconds. They show the kind of line to look for.
Upgrading the cluster
A Kubernetes release comes every few months, and each minor version (1.35, 1.36) is supported for about a year. Staying current is part of running it. The rules that keep an upgrade safe:
one minor version at a time (1.35 to 1.36, never 1.34 to 1.36), and patch releases freely
control plane first, then the nodes. The control plane may be one minor version newer than the kubelets (and recent releases allow up to three), but never older
kubectl may differ from the server by one minor version either way
read the release notes for removed APIs before you start: an API that was deprecated can disappear in a later version, and manifests that use it stop working
back up first: an etcd snapshot (the section above), and your manifests are in git
drain a node, upgrade it, uncordon it, one at a time, so that the workloads keep running
Take the snapshot, and look at where the cluster is:
$ k3s etcd-snapshot save --name before-upgrade 2>&1 | grep -E 'saved' # run as root on the server node
time="2026-10-19T21:46:12Z" level=info msg="Snapshot before-upgrade-k8s-1792446372 saved."
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s Ready control-plane,etcd 18m v1.35.9+k3s1
k8w1 Ready <none> 5s v1.35.9+k3s1
$ kubectl version | grep -E 'Client|Server'
Client Version: v1.35.9+k3s1
Server Version: v1.35.9+k3s1
To find manifests that use an API that is going away, ask the API server which deprecated APIs have been called since it started (the counts are from this server's own metrics):
$ kubectl get --raw /metrics | grep '^apiserver_requested_deprecated_apis' | head -3
apiserver_requested_deprecated_apis{group="",removed_release="",resource="endpoints",subresource="",version="v1"} 1
(The one line says that something called the v1 Endpoints API, which is deprecated in favour of EndpointSlices, once since the server started, and removed_release is empty because no removal is planned yet. k3s's own components make such calls. No lines at all, or lines with a removed_release you are about to reach, are what to act on. Tools such as kubent and pluto scan your manifests and Helm releases for the same thing, and were not run.) With k3s, the installer script upgrades by running again with a new version, which replaces the binary and restarts the service:
$ curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.36.5+k3s1 sh -
The lab has no systemd, so it did the same thing by hand: replaced the binary on the server (k3s v1.36.5+k3s1 is the stable release in the k3s release channel at the time of the test), and restarted the process. First the server:
$ pkill -x k3s-server; sleep 6; mv -f /usr/local/bin/k3s.new /usr/local/bin/k3s; k3s --version | head -1 # run as root on the server node
k3s version v1.36.5+k3s1 (3dd98cc5)
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s Ready control-plane,etcd 18m v1.36.5+k3s1
k8w1 Ready <none> 24s v1.35.9+k3s1
The control plane is on 1.36.5 and the worker is still on 1.35.9, which is allowed and the right order. The pods on the worker kept running during the server's restart. Now the worker: drain it, replace its binary, restart it, and let it take work again:
$ kubectl drain k8w1 --ignore-daemonsets --delete-emptydir-data 2>&1 | tail -1
node/k8w1 drained
$ pkill -x k3s-agent; sleep 6; mv -f /usr/local/bin/k3s.new /usr/local/bin/k3s; k3s --version | head -1 # run as root on the worker node
k3s version v1.36.5+k3s1 (3dd98cc5)
$ kubectl uncordon k8w1
node/k8w1 uncordoned
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s Ready control-plane,etcd 18m v1.36.5+k3s1
k8w1 Ready <none> 37s v1.36.5+k3s1
$ kubectl get pods -A | grep -vE 'Running|Completed|^NAMESPACE' || echo "all pods Running or Completed"
all pods Running or Completed
Both nodes are on v1.36.5 and the system pods are healthy. A drain before an upgrade also tests your PodDisruptionBudgets and your replica counts: an application with one replica goes down during the drain, and no setting changes that. In a managed cloud cluster the provider upgrades the control plane with one click or command, and you upgrade the node pools in the same order. With kubeadm the steps are kubeadm upgrade plan and apply on the first server, kubeadm upgrade node on the others, then the kubelet package on every node (not run in the lab). Rolling back a Kubernetes upgrade is not supported in general: the safe plan is a snapshot restore onto the old version, so test the upgrade on a copy of the cluster first.
Kustomize and Helm: packaging manifests
Plain YAML repeats itself: the same Deployment for dev, test and production, with a different image tag, replica count and name. Two tools deal with that. Kustomize patches plain YAML without templates, and is built into kubectl. Helm packages YAML as templated charts with values, and keeps a numbered history of installs. The site has separate cheat sheets for both (Kustomize, Helm); this is how they fit into running a cluster.
Kustomize. A base holds the plain manifests. An overlay says how one environment differs:
$ cat kz/base/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
replicas: 1
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: nginx
image: nginx:1.27-alpine
$ cat kz/base/kustomization.yaml
resources:
- deployment.yaml
$ cat kz/prod/kustomization.yaml
resources:
- ../base
namePrefix: prod-
namespace: production
replicas:
- name: web
count: 3
images:
- name: nginx
newTag: 1.28-alpine
labels:
- pairs:
env: production
includeSelectors: true
$ kubectl kustomize kz/prod | grep -E 'name:|namespace:|replicas:|image:|env:'
env: production
name: prod-web
namespace: production
replicas: 3
env: production
env: production
- image: nginx:1.28-alpine
name: nginx
$ kubectl create namespace production
namespace/production created
$ kubectl apply -k kz/prod
deployment.apps/prod-web created
$ kubectl get deploy -n production --show-labels
NAME READY UP-TO-DATE AVAILABLE AGE LABELS
prod-web 3/3 3 3 4s env=production
$ kubectl delete -k kz/prod
deployment.apps "prod-web" deleted from production namespace
kubectl kustomize only prints the result, so you can read exactly what an overlay will do. Apply it with -k. The base was not changed, and a "dev" overlay beside "prod" would give a second variant. Kustomize cannot loop or compute, and that limit is also its good point: what you read is what you get.
Helm. A chart is a folder of templates and a values.yaml with the defaults, and an install of a chart is a release with its own name and history. Helm is a separate program. The lab downloaded the release tarball of Helm 4.3.1 from get.helm.sh:
$ curl -sfL https://get.helm.sh/helm-v4.3.1-linux-amd64.tar.gz | tar -xz -C /tmp && install -m 755 /tmp/linux-amd64/helm /usr/local/bin/helm && helm version --short # run as root on the server node
v4.3.1+gc0d08d0
$ helm create mychart > /dev/null && ls mychart mychart/templates | head -20
mychart:
Chart.yaml
charts
templates
values.yaml
mychart/templates:
NOTES.txt
_helpers.tpl
deployment.yaml
hpa.yaml
httproute.yaml
ingress.yaml
service.yaml
serviceaccount.yaml
tests
$ helm template demo ./mychart --set image.tag=1.28-alpine | grep -E '^kind:|image:' | head -8
kind: ServiceAccount
kind: Service
kind: Deployment
image: "nginx:1.28-alpine"
kind: Pod
image: busybox
helm template renders the templates to YAML without touching the cluster, and is the same idea as kubectl kustomize: look before you leap. Install it, change a value, and look at the history:
$ helm install demo ./mychart --set image.tag=1.28-alpine --wait --timeout 120s | head -4
NAME: demo
LAST DEPLOYED: Mon Oct 19 21:46:58 2026
NAMESPACE: default
STATUS: deployed
$ helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
demo default 1 2026-10-19 21:46:58.093794033 +0000 UTC deployed mychart-0.1.0 1.16.0
$ helm upgrade demo ./mychart --set image.tag=1.28-alpine --set replicaCount=2 --wait --timeout 120s | head -2
Release "demo" has been upgraded. Happy Helming!
NAME: demo
$ helm history demo
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Mon Oct 19 21:46:58 2026 superseded mychart-0.1.0 1.16.0 Install complete
2 Mon Oct 19 21:47:00 2026 deployed mychart-0.1.0 1.16.0 Upgrade complete
$ kubectl get deploy demo-mychart
NAME READY UP-TO-DATE AVAILABLE AGE
demo-mychart 2/2 2 2 4s
The release is at revision 2 with two replicas. Roll it back to revision 1, and remove it:
$ helm rollback demo 1 --wait --timeout 120s
Rollback was a success! Happy Helming!
$ helm history demo | tail -2
2 Mon Oct 19 21:47:00 2026 superseded mychart-0.1.0 1.16.0 Upgrade complete
3 Mon Oct 19 21:47:02 2026 deployed mychart-0.1.0 1.16.0 Rollback to 1
$ kubectl get deploy demo-mychart
NAME READY UP-TO-DATE AVAILABLE AGE
demo-mychart 1/1 1 1 5s
$ helm uninstall demo
release "demo" uninstalled
Helm keeps each revision as a Secret in the release's namespace (look with kubectl get secrets), which is why rollback works and why deleting those Secrets loses the history. Charts from public repositories save a great deal of work for common software (ingress controllers, databases, monitoring), and they also bring in templates that you did not write. Pin the chart version, read the values, run helm template and diff it before you install, and never install a chart from a source you have not looked at into a cluster that matters. Public chart repositories were not used in the lab. Helm and Kustomize are often combined (Helm renders, Kustomize patches), and GitOps tools such as Argo CD and Flux apply either from git, none of which were run.
Troubleshooting by symptom
Nearly every problem shows up in the same few places. Work from the outside in: what is the object's status, what do its events say, what do the logs say, and what is the node doing. These are the symptoms met in this guide, with what the real output looked like:
STATUS / SYMPTOM WHAT IT MEANS FIRST COMMANDS
Pending not scheduled: no room, an unmet taint, label kubectl describe pod (the Events: FailedScheduling), kubectl describe node
or unbound volume claim
ContainerCreating (stuck) image pull or volume mount in progress or failing kubectl describe pod, kubectl get events
ImagePullBackOff, ErrImagePull wrong image name or tag, or no access kubectl describe pod, check the name and any imagePullSecrets
CrashLoopBackOff the container starts and exits, again and again kubectl logs POD --previous, kubectl describe pod (exit code)
OOMKilled (exit code 137) the memory limit was exceeded kubectl describe pod, raise the limit or fix the leak
Running but 0/1 Ready the readiness probe fails kubectl describe pod, kubectl logs, curl the probe path
Service answers nothing the selector matches no pod, or no pod is ready kubectl get endpointslices, compare labels with the selector
Ingress gives 404 no rule matches the host or path kubectl describe ingress, curl with the right Host header
Terminating (stuck) a finalizer or a node that is gone kubectl get pod -o yaml (finalizers), the node's status
NotReady node the kubelet cannot reach the API or the runtime kubectl describe node, the kubelet log, disk, memory, time
Forbidden RBAC does not allow it kubectl auth can-i VERB RESOURCE --as=..., read the message
rollout stuck the new pods never become ready kubectl rollout status, kubectl describe, then rollout undo
Look at the events of the whole namespace, newest last, which often names the problem outright, and at the previous container's log when a container keeps restarting:
$ kubectl run crash --image=busybox:1.36 --restart=Always -- sh -c 'echo starting; echo "fatal: config file missing" >&2; exit 3'
pod/crash created
$ kubectl get pod crash
NAME READY STATUS RESTARTS AGE
crash 0/1 Error 2 (24s ago) 25s
$ kubectl logs crash
starting
fatal: config file missing
$ kubectl get pod crash -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}{"\n"}'
3
$ kubectl get events --sort-by=.lastTimestamp | grep crash | tail -3 | cut -c1-150
14s Normal Created pod/crash Container created
14s Normal Pulled pod/crash Container image "busybox:1.36" a
13s Warning BackOff pod/crash Back-off restarting failed conta
$ kubectl delete pod crash --wait=false
pod "crash" deleted from default namespace
The log of the crashed attempt (add --previous to read the one before the latest restart) and its exit code (3) are what the program itself said, and they are the first place to look. The kubectl Day-to-Day page on this site goes through each of these failures in more depth, with the commands to find the cause. A pod you cannot read the cause of is often one that cannot be scheduled (no log exists yet), so describe comes before logs when the pod never started.
Two last checks that save time. Is the node really healthy, and is the system itself running:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s Ready control-plane,etcd 19m v1.36.5+k3s1
k8w1 Ready <none> 83s v1.36.5+k3s1
$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-b644fd95b-kl9h2 1/1 Running 0 2m14s
local-path-provisioner-d9d8769b6-pzcd2 1/1 Running 0 2m14s
metrics-server-7f4b6d9bd7-vtcqp 1/1 Running 0 2m14s
svclb-traefik-5192b2cf-dlvkh 2/2 Running 0 19m
svclb-traefik-5192b2cf-mthvx 2/2 Running 0 82s
traefik-5869c87968-mmqzf 1/1 Running 0 2m14s
$ kubectl get --raw='/readyz?verbose' | tail -4
[+]poststarthook/apiservice-openapi-controller ok
[+]poststarthook/apiservice-openapiv3-controller ok
[+]shutdown ok
readyz check passed
The API server's /readyz endpoint lists its own checks, and the last line says ok when all pass. If kubectl itself cannot connect, the problem is below Kubernetes: the API server process, its certificate, the network or the clock, and the place to look is the server's own log.
Safe habits
What a team that runs Kubernetes for a year ends up doing, and what each habit guards against.
keep every manifest in git, and apply from git the file is the truth; hand edits are lost or forgotten
pin image tags (and digests for important ones) :latest changes under you and cannot be rolled back
set requests on every container, limits on memory the scheduler and the OOM killer have something to work from
add readiness probes to everything that takes traffic traffic goes only to pods that can serve it
run at least two replicas of anything that matters a drain or a node loss does not take the service down
add a PodDisruptionBudget to those voluntary disruptions cannot remove every replica
spread replicas over nodes (topology spread) one node failure is not an outage
use one namespace per team or app, with quotas one bad deployment cannot eat the cluster
one service account per application, least privilege a leaked token is a small problem
default-deny NetworkPolicy, then allow what is needed a compromised pod cannot reach everything
do not run as root; drop capabilities see the security context of the pod (not run in the lab)
never put Secrets in git; encrypt them at rest base64 is not encryption
back up etcd and the volumes, and test a restore the only backup is one that has been restored
take snapshots before an upgrade, upgrade one minor a failed upgrade cannot be undone any other way
watch certificate expiry and node disk space the two quiet causes of most outages
read kubectl diff and --dry-run=server before apply the cluster answers what would change
The list of kubectl commands that bite: kubectl delete with a label selector and no namespace, kubectl delete -f on a file that also holds a Namespace, kubectl apply --force (it deletes and recreates), kubectl drain on a node that holds the only replica, kubectl edit in production (it changes the live object and leaves git behind), and any command typed against the wrong context. Show the context and namespace in your shell prompt, and keep separate kubeconfig files for production.
What was not run in the lab
The lab was a two-node k3s cluster (v1.35.9, then v1.36.5) in two containers on one machine. Everything in the sections above ran there. These did not:
multi-node control planes, high availability, and a restore on a multi-server cluster; separate machines, real networks and real node failures (a node that disappears, a network split)
cloud-managed clusters (EKS, AKS, GKE), their load balancers, disks, identity integration and node pools; kubeadm clusters; kind and minikube
cloud storage classes, ReadWriteMany volumes (NFS, CephFS), volume snapshots, volume expansion, and the CSI drivers behind them; moving a volume between nodes
HTTPS for the Ingress (cert-manager, Let's Encrypt), the Gateway API, other ingress controllers (nginx, HAProxy), and real DNS names
other network plug-ins (Calico, Cilium, Flannel in the real world) and their policies, egress policies, a service mesh
custom metrics for the HPA (KEDA, the Prometheus adapter), the vertical autoscaler, and node autoscaling (Cluster Autoscaler, Karpenter)
OIDC login, admission controllers (Pod Security Admission, Kyverno, OPA Gatekeeper), securityContext and seccomp settings, and encryption of Secrets at rest
operators and custom resources (CRDs) beyond the ones k3s installs, GitOps tools (Argo CD, Flux), and public Helm repositories
monitoring and logging stacks (Prometheus, Grafana and Loki, covered on other pages of this site, were not connected to this cluster)
upgrade paths other than k3s v1.35.9 to v1.36.5, the removal of deprecated APIs, kubeadm and managed upgrades, and the k3s system-upgrade-controller
Windows nodes, GPUs, ARM machines, and Kubernetes versions other than the two above; Ubuntu versions other than 24.04
the details of the old kubectl cheat sheet that this guide replaces are in the next section, and the kubectl command line itself is on the kubectl Day-to-Day page
The cheat sheet
Context and cluster
kubectl config get-contexts list contexts (the * is the current one)
kubectl config use-context NAME switch cluster
kubectl config set-context --current --namespace=NS change the default namespace
kubectl cluster-info where the API server is
kubectl version client and server versions
kubectl api-resources every kind this cluster knows (add --namespaced=false for cluster-wide ones)
kubectl explain pod.spec.containers the schema of any field
Look at things
kubectl get nodes|pods|svc|deploy|ing|pvc -A one kind, every namespace
kubectl get pods -o wide with node and IP
kubectl get pods -l app=web --show-labels filter by label
kubectl get pod NAME -o yaml the whole object
kubectl get pod NAME -o jsonpath='{.status.podIP}' one field
kubectl describe pod NAME status, conditions and Events: the first place to look
kubectl get events --sort-by=.lastTimestamp what just happened
kubectl top nodes|pods real use (needs metrics-server)
kubectl logs POD [-c CONTAINER] [-f] [--previous] [--tail=50]
kubectl exec -it POD -- sh a shell in the container
kubectl port-forward svc/NAME 8080:80 reach a service from your machine
kubectl cp POD:/path ./local copy files
Change things
kubectl apply -f FILE|DIR|URL create or update (the normal way)
kubectl apply -k DIR a kustomization
kubectl diff -f FILE what would change
kubectl apply -f FILE --dry-run=server validate on the server
kubectl create deployment web --image=nginx:alpine --replicas=2
kubectl scale deployment web --replicas=5
kubectl set image deployment/web nginx=nginx:1.28-alpine
kubectl rollout status|history|undo|restart deployment web
kubectl label|annotate pod NAME key=value
kubectl patch deployment web -p '{"spec":{"replicas":3}}'
kubectl delete -f FILE | kubectl delete pod NAME [--grace-period=0 --force]
kubectl autoscale deployment web --cpu=50% --min=2 --max=10
kubectl create job NAME --from=cronjob/CRON run a CronJob now
Nodes and maintenance
kubectl cordon NODE no new pods kubectl uncordon NODE allow them again
kubectl drain NODE --ignore-daemonsets --delete-emptydir-data move the pods off
kubectl taint node NODE key=value:NoSchedule (add a trailing - to remove)
kubectl label node NODE key=value
kubectl describe node NODE capacity, allocated, conditions, taints
Access
kubectl auth can-i VERB RESOURCE [-n NS] [--as=USER]
kubectl create token SERVICEACCOUNT [-n NS] --duration=10m
kubectl create secret generic NAME --from-literal=k=v
kubectl create configmap NAME --from-file=FILE
k3s
k3s etcd-snapshot save --name NAME snapshot (embedded etcd only)
k3s server --cluster-reset --cluster-reset-restore-path=FILE restore
k3s certificate check --output table certificate expiry
k3s crictl ps containers on this node
k3s secrets-encrypt status encryption of Secrets at rest
Object types in one line each
Pod one or more containers Deployment N replicas and rolling updates StatefulSet named pods with their own disks
DaemonSet one pod per node Job / CronJob run to completion / on a schedule Service stable name and address
Ingress HTTP routing in ConfigMap / Secret settings PVC / PV / StorageClass storage Namespace a folder with quotas
NetworkPolicy who may talk to whom Role / RoleBinding who may do what HPA autoscaling PDB limits on voluntary disruptions
Common problems and tips
A pod is Pending and nothing says why. Run kubectl describe pod and read the Events at the bottom. It is nearly always FailedScheduling (not enough room, a taint, a node selector) or a claim that cannot bind.
Changing a ConfigMap did nothing. Environment variables only change when the pod restarts (kubectl rollout restart), and files take about a minute to refresh and are not refreshed at all with subPath. Many applications do not re-read their files either.
A Service has no endpoints. The selector does not match the pods' labels (compare kubectl get svc -o yaml with kubectl get pods --show-labels), or the pods are not ready.
An Ingress returns 404. There is no rule for that host or path, or there is no ingress controller, or the name in the browser is not the one in the rule.
A rollout hangs. The new pods never become ready (image, probe, resources). The old ones keep serving while maxUnavailable is 0, so look at the new ones with kubectl describe, and kubectl rollout undo when in doubt.
A node is NotReady. Look at kubectl describe node (the conditions), then on the node itself: the kubelet and runtime logs, the disk, the memory and the clock.
The cluster stops accepting commands after a year. The certificates expired. Check k3s certificate check (or kubeadm certs check-expiration) well before, and restart the server in time.
The disk of a node fills up. Container logs, unused images and local volumes. Kubernetes cleans up images and dead containers by itself under pressure, but a full disk evicts pods first.
Deleting a namespace hangs in Terminating. Something in it has a finalizer that never finishes (often a custom resource whose controller is gone). Find it with kubectl api-resources --verbs=list --namespaced -o name and look at what is left.
Tips:
Work in a namespace of your own while learning, and delete it afterwards: kubectl delete namespace NAME removes everything in it.
Turn on shell completion (source <(kubectl completion bash)) and the short alias k, and use kubectl explain instead of searching.
Prefer declarative: kubectl apply a file, even for a quick test, so that you can see and repeat what you did.
kubectl get ... -o yaml, then remove the status and the generated fields, is a fair way to start a manifest. kubectl create ... --dry-run=client -o yaml is better.
Read events and logs before restarting anything. A restart that "fixes" it also destroys the evidence.
Keep clusters small and boring: few namespaces, few add-ons, regular upgrades. Most of the pain comes from clusters that nobody dared to touch for two years.
