A how-to manual: using charts from a repository, installing and reading releases, values, writing a chart and its templates, safe upgrades and rollbacks, hooks, CRDs, subcharts, publishing, and the traps, with every example run for real on a two-node lab.
▶Prefer to watch? This short video runs this page's main commands on a real two-node k3s cluster, with on-screen captions and no voiceover. Click to play, or watch it on YouTube.
helm: the package manager for Kubernetes. A chart is the recipe, a release is one installed copy of it.
helm upgrade --install web ./chart -f values.yaml --wait --rollback-on-failure
helm history web
helm rollback web 1
helm template web ./chart | kubectl diff -f -
Helm turns a folder of templated Kubernetes YAML and a file of settings into the objects in a cluster, and keeps a numbered record of every install and upgrade, so that you can look at what is running and go back. It is a small tool with a few sharp edges: an upgrade that forgets the settings of the last one, an install that says "deployed" while the pods fail, a release stuck half-upgraded after a killed pipeline, a ConfigMap change that no pod notices, a CRD that is never upgraded, a secret that sits in clear text in the release record.
This guide is how to use it without meeting those the hard way. It starts with charts that other people publish (repositories, values, installing, reading a release), then writes a chart and its templates from nothing, then covers the operations: upgrades, rollbacks, waiting, hooks, CRDs, subcharts, publishing to a chart repository and to an OCI registry, where Helm keeps its records, adopting existing objects, several environments from one chart, what is new in Helm 4 and how to read a chart before you trust it. The Kubernetes guide on this site covers the cluster, the Kustomize page covers the other way to avoid copied YAML, and the older Helm cheat sheet that stood on this page is replaced by this guide (its one-line commands are in the cheat sheet section).
Every example ran for real, on two Ubuntu 24.04 machines (containers with their own container runtime inside) running k3s on Kubernetes v1.35.9, with Helm 4.3.1. The lab used a real public chart repository, Artifact Hub's chart search and a real plug-in from GitHub, and ran its own chart repository and OCI registry. Private registries, signed charts, the large public chart repositories and GitOps tools were not run, and the section "What was not run in the lab" lists them.
What Helm is, and the lab used here
Kubernetes objects are written as YAML, and a real application is a pile of them: a Deployment, a Service, a ConfigMap, an Ingress, a ServiceAccount, and the same pile again for test and for production with a few things changed. Copying the files and editing them by hand does not scale, and nothing remembers what was installed or lets you go back. Helm is the package manager that fills that gap.
The words, which the rest of the guide uses exactly:
chart a package: a folder (or a .tgz of it) with templates, default values and a description. It is the recipe.
release one installed copy of a chart in a cluster, with a name. You can install the same chart many times under different names.
values the settings that fill the templates. The chart ships defaults, and you override them for each release.
revision a numbered version of a release. Every install, upgrade and rollback makes a new one.
repository a web server that holds charts and an index.yaml listing them. Charts also live in OCI registries, the same kind that hold container images.
template a YAML file with {{ placeholders }} written in the Go template language, rendered into plain Kubernetes YAML before it is sent.
hook a template (usually a Job) that Helm runs at a point in the life of a release, such as before an upgrade.
Helm is a client-side program. There is no server component in the cluster (that was Tiller in Helm 2, long gone). Helm renders the templates on your machine, sends the result to the Kubernetes API, and stores a record of each revision as a Secret in the cluster. That record is what makes helm list, helm history and helm rollback work, and a later section looks inside it.
This guide covers what Helm does day to day: using charts that others published, writing your own, installing and upgrading safely, rolling back, hooks, dependencies, publishing to a repository and to a registry, and the traps. The Kustomize page on this site covers the other way to avoid copy and paste, and the Kubernetes guide covers the cluster itself. The older Helm cheat sheet that stood on this page is replaced by this guide, and its one-line commands are in the cheat sheet section.
Every example ran for real, on a lab of two Ubuntu 24.04 machines (containers with their own container runtime inside) running k3s, a small certified Kubernetes distribution, at Kubernetes v1.35.9 with Helm 4.3.1. One node is the server (named k8s) and one is a worker (k8w1). The lab had internet access, so it used a real public chart repository (the one the Helm project publishes for examples), real Artifact Hub, and a real plug-in downloaded from GitHub. It also ran its own chart repository and its own OCI registry, so that publishing could be tried without touching anyone else's. Not run: private registries with logins, signed charts, charts from the large public repositories, GitOps tools, and everything that needs a cloud. The section "What was not run in the lab" lists them.
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s Ready control-plane,etcd 98s v1.35.9+k3s1
k8w1 Ready <none> 6s v1.35.9+k3s1
Install Helm and see what it keeps
Helm is one static binary. Use the method your platform prefers:
$ sudo snap install helm --classic # Ubuntu and other snap systems
$ brew install helm # macOS and Linux with Homebrew
$ winget install Helm.Helm # Windows
or download the tarball from get.helm.sh and put the binary on the path, which is what the lab did. Pin the version you tested, and read any install script before you run it:
$ curl -sSL -o /tmp/helm.tgz https://get.helm.sh/helm-v4.3.1-linux-amd64.tar.gz && tar -xz -C /tmp -f /tmp/helm.tgz && install -m 755 /tmp/linux-amd64/helm /usr/local/bin/helm && helm version --short # run as root on the server node
v4.3.1+gc0d08d0
$ helm version
version.BuildInfo{Version:"v4.3.1", GitCommit:"c0d08d0d6d9424d4a14febb5e8a3e51b0dd00dff", GitTreeState:"clean", GoVersion:"go1.27.2", KubeClientVersion:"v1.37"}
Helm finds the cluster the same way kubectl does, from the KUBECONFIG file and its current context. It needs no other login. That means the permissions Helm has are exactly the permissions of that kubeconfig: if kubectl may not create a Deployment, neither may Helm. Tab completion for bash, zsh and fish comes from helm completion SHELL.
Where Helm keeps its own files (the cache, the repository list, the plug-ins) depends on the system, and helm env prints it all:
$ helm env | grep -E 'CACHE_HOME|CONFIG_HOME|DATA_HOME|NAMESPACE|MAX_HISTORY|DRIVER'
HELM_CACHE_HOME="/home/ubuntu/.cache/helm"
HELM_CONFIG_HOME="/home/ubuntu/.config/helm"
HELM_DATA_HOME="/home/ubuntu/.local/share/helm"
HELM_MAX_HISTORY="10"
HELM_NAMESPACE="default"
HELM_NAMESPACE is the default namespace for the commands, and the -n flag overrides it. HELM_MAX_HISTORY is how many revisions of each release are kept (10 unless you change it). The environment variable HELM_DRIVER chooses where release records are stored: the default is Secrets, and configmap, memory and sql also exist (the SQL one needs a database, and none of the others were run).
One surprise on this cluster. It already has Helm releases, because k3s installs its bundled Traefik and its CRDs with Helm:
$ helm list -A
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
traefik kube-system 1 2026-10-20 00:31:33.218621329 +0000 UTC deployed traefik-40.1.5+up40.1.0 v3.7.1
traefik-crd kube-system 1 2026-10-20 00:31:19.055204504 +0000 UTC deployed traefik-crd-40.1.5+up40.1.0 v3.7.1
Those two are managed by k3s (it re-applies them), so do not upgrade or remove them by hand.
Charts from a repository: add, search, look, download
A repository is a plain web server with an index.yaml file that lists the charts and where their packages are. Add one, refresh your copy of its index, and search:
$ helm repo add examples https://helm.github.io/examples
"examples" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "examples" chart repository
Update Complete. ⎈Happy Helming!⎈
$ helm repo list
NAME URL
examples https://helm.github.io/examples
$ helm search repo examples
NAME CHART VERSION APP VERSION DESCRIPTION
examples/hello-world 0.1.0 1.16.0 A Helm chart for Kubernetes
$ helm search repo examples --versions
NAME CHART VERSION APP VERSION DESCRIPTION
examples/hello-world 0.1.0 1.16.0 A Helm chart for Kubernetes
helm search repo only searches the repositories you added, using your local copy of their indexes, so run helm repo update first, or you search a stale list. To find charts in the whole public world, helm search hub QUERY searches Artifact Hub (artifacthub.io). It needs internet access and its results change daily, so its output is not shown here:
$ helm search hub wordpress
A hub hit gives a URL with the chart's page and the repository to add. Before installing anything, look at it. These commands read the chart without touching the cluster:
$ helm show chart examples/hello-world
apiVersion: v2
appVersion: 1.16.0
description: A Helm chart for Kubernetes
name: hello-world
type: application
version: 0.1.0
$ helm show values examples/hello-world
# Default values for hello-world.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
image:
repository: nginx
pullPolicy: IfNotPresent
# Overrides the image tag whose default is the chart appVersion.
tag: ""
nameOverride: ""
fullnameOverride: ""
serviceAccount:
# Specifies whether a service account should be created
create: true
# Annotations to add to the service account
annotations: {}
# The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
name: ""
service:
type: ClusterIP
port: 80
$ helm show readme examples/hello-world # the chart's README, if it has one
$ helm show all examples/hello-world # chart, values and readme in one
helm show values is the most useful of them, because the values file is the chart's interface: every setting you may change, with its default. To read the templates too, download the chart and unpack it:
$ helm pull examples/hello-world --untar && ls hello-world hello-world/templates
hello-world:
Chart.yaml
README.md
templates
values.yaml
hello-world/templates:
NOTES.txt
_helpers.tpl
deployment.yaml
service.yaml
serviceaccount.yaml
$ helm template hw hello-world | grep -E 'kind:|image:'
kind: ServiceAccount
kind: Service
kind: Deployment
image: "nginx:1.16.0"
helm pull without --untar saves the .tgz file. Keep in mind that a chart is code that runs with your permissions: it decides what objects get created. Reading the rendered kinds and images, as the last command did, takes ten seconds and is worth it before installing anything from a source you have not used before.
Repositories can be removed with helm repo remove NAME. Other kinds of source (an OCI registry, a local folder, a .tgz file, a URL) need no repo add, and appear in the packaging sections below.
Install a release and read it
Install with a release name and a chart. The name is yours to choose, it must be unique in the namespace, and it is what you use for every later command. --wait makes Helm wait until the pods are ready, with --timeout as the limit (a later section explains what the default does):
$ helm install hw examples/hello-world --wait --timeout 120s
NAME: hw
LAST DEPLOYED: Tue Oct 20 00:31:41 2026
NAMESPACE: default
STATUS: deployed
REVISION: 1
DESCRIPTION: Install complete
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=hello-world,app.kubernetes.io/instance=hw" -o jsonpath="{.items[0].metadata.name}")
export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT
$ kubectl get pods,svc -l app.kubernetes.io/instance=hw
NAME READY STATUS RESTARTS AGE
pod/hw-hello-world-cdccb945b-m978b 1/1 Running 0 1s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/hw-hello-world ClusterIP 10.43.171.235 <none> 80/TCP 1s
The output is a summary of the release plus the chart's own NOTES, which is text the chart author wrote to say what to do next (this chart's notes show how to reach the pod with port-forward). Everything about a release can be read back later:
$ helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
hw default 1 2026-10-20 00:31:41.85290552 +0000 UTC deployed hello-world-0.1.0 1.16.0
$ helm status hw
NAME: hw
LAST DEPLOYED: Tue Oct 20 00:31:41 2026
NAMESPACE: default
STATUS: deployed
REVISION: 1
DESCRIPTION: Install complete
RESOURCES:
==> v1/ServiceAccount
NAME AGE
hw-hello-world 2s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hw-hello-world ClusterIP 10.43.171.235 <none> 80/TCP 2s
==> v1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
hw-hello-world 1/1 1 1 2s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
hw-hello-world-cdccb945b-m978b 1/1 Running 0 2s
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=hello-world,app.kubernetes.io/instance=hw" -o jsonpath="{.items[0].metadata.name}")
export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT
$ helm get values hw
USER-SUPPLIED VALUES:
null
$ helm get manifest hw | grep -E '^kind:|^ name:'
kind: ServiceAccount
name: hw-hello-world
kind: Service
name: hw-hello-world
kind: Deployment
name: hw-hello-world
$ helm history hw
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Tue Oct 20 00:31:41 2026 deployed hello-world-0.1.0 1.16.0 Install complete
What each one answers: helm list shows releases and their status and chart version. helm status shows the state and the notes again. helm get values shows only what YOU set (nothing so far, which Helm prints as null). helm get manifest shows the exact YAML that was sent to the cluster, after rendering, which is the thing to read when a result is not what you expected. helm get all prints everything, including the hooks and the notes. helm history shows each revision.
The objects carry labels that say who owns them, which is also how you find everything a release made:
$ kubectl get deployment hw-hello-world -o jsonpath='{.metadata.labels}{"\n"}'
{"app.kubernetes.io/instance":"hw","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"hello-world","app.kubernetes.io/version":"1.16.0","helm.sh/chart":"hello-world-0.1.0"}
app.kubernetes.io/managed-by: Helm is set by Helm itself. The release name and namespace are recorded as annotations (meta.helm.sh/release-name and meta.helm.sh/release-namespace) on every object. A later section shows what happens when an object has no such mark.
Removing a release deletes what it made:
$ helm uninstall hw
release "hw" uninstalled
$ kubectl get pods -l app.kubernetes.io/instance=hw
No resources found in default namespace.
Persistent volumes and claims created by a StatefulSet's volume template are not deleted by uninstall, as in Kubernetes itself, so look with kubectl get pvc after removing a database.
Values: the layers and how they combine
A chart's defaults live in its values.yaml. You change them for one release in three ways, and they stack. From lowest to highest priority: the chart's own defaults, then each -f file in the order given, then --set flags. Later wins, and maps are merged key by key, so you only name the keys you change.
The chart in the next section (a small web front end called shop) is used here, because it has a few simple values. To see layers, render it without installing, with helm template:
$ cat shop/Chart.yaml
apiVersion: v2
name: shop
description: A small web shop front end
type: application
version: 0.1.0
appVersion: "1.27"
$ cat shop/values.yaml
replicaCount: 2
image:
repository: nginx
tag: ""
pullPolicy: IfNotPresent
service:
type: ClusterIP
port: 80
message: "Welcome to the shop"
resources: {}
ingress:
enabled: false
host: shop.lab
$ cat shop/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Release.Name }}-shop
data:
message: {{ .Values.message | quote }}
replicas: {{ .Values.replicaCount | quote }}
image: {{ printf "%s:%s" .Values.image.repository (.Values.image.tag | default .Chart.AppVersion) | quote }}
$ helm template t shop | grep -E 'message|replicas|image'
message: "Welcome to the shop"
replicas: "2"
image: "nginx:1.27"
Now override. The last one wins:
$ cat prod.yaml
replicaCount: 4
message: "Welcome to the shop (production)"
$ helm template t shop -f prod.yaml | grep -E 'message|replicas|image'
message: "Welcome to the shop (production)"
replicas: "4"
image: "nginx:1.27"
$ helm template t shop -f prod.yaml --set replicaCount=6 --set image.tag=1.28 | grep -E 'message|replicas|image'
message: "Welcome to the shop (production)"
replicas: "6"
image: "nginx:1.28"
The file changed two values, the flag then changed replicaCount again and added an image tag. Nothing else moved. Put settings that belong to an environment into a file kept in git, and keep --set for one-off changes, so that the record of what is running is a file and not a shell history.
--set has a small syntax of its own. Dots go down into maps, commas separate pairs, and square brackets or curly braces make lists. Quote anything with braces or commas, or the shell and Helm will both take a piece of it (an unquoted --set tags={a,b,c} is split by bash into three words, and Helm then fails with "expected at most two arguments"). Three variants are worth knowing:
--set-string KEY=VALUE always a string, so 1234 or true stay text instead of becoming a number or a boolean
--set-json KEY=JSON a whole structure written as JSON, for lists of maps that --set cannot say
--set-file KEY=FILE the value is the content of a file
A comma ends a value, which catches messages written in plain English. The first of these fails, and the second escapes the comma with a backslash (a values file avoids the problem altogether):
$ helm template t shop --set message="Welcome, spring sale" 2>&1 | head -1
Error: failed parsing --set data: key " spring sale" has no value
$ helm template t shop --set 'message=Welcome\, spring sale' | grep message
message: "Welcome, spring sale"
$ helm install t shop --set-json 'resources={"limits":{"cpu":"200m"}}'
$ helm install t shop --set-file message=./message.txt
Both are shown without running, because this small chart does not use the values they set. They are tested in the next section, where the chart has a resources block. A trap with --set-file: a file ends with a newline, and that newline becomes part of the value, which breaks YAML when the value is placed inside a line of a template. Use printf without a newline to write such files, or trim in the template. When the release is installed, helm get values shows what you supplied, and helm get values -a shows the computed result of all layers together, which is what to read when a value is not what you expected.
Write your own chart
helm create makes a starter chart with everything a typical web app needs. Look at what it makes:
$ helm create scratch > /dev/null; find scratch | sort
scratch
scratch/.helmignore
scratch/Chart.yaml
scratch/charts
scratch/templates
scratch/templates/NOTES.txt
scratch/templates/_helpers.tpl
scratch/templates/deployment.yaml
scratch/templates/hpa.yaml
scratch/templates/httproute.yaml
scratch/templates/ingress.yaml
scratch/templates/service.yaml
scratch/templates/serviceaccount.yaml
scratch/templates/tests
scratch/templates/tests/test-connection.yaml
scratch/values.yaml
The starter is long, because it covers ingress, autoscaling, a ServiceAccount and the newer HTTPRoute at once, with many if-blocks. It is a fine reference, but a poor way to learn, so this guide writes a small chart by hand and adds to it. The pieces of a chart:
Chart.yaml name, version (the chart's own), appVersion (the version of the software inside), description, dependencies
values.yaml the defaults, and the documentation of every setting
templates/ the YAML with placeholders, rendered in alphabetical order of file name (Helm sorts the result by object kind when it installs, so a Namespace or ConfigMap exists before the pod that needs it)
templates/_helpers.tpl files that start with an underscore make no objects; they hold named snippets
templates/NOTES.txt text printed after install, also rendered as a template
templates/tests/ pods that helm test runs
charts/ subcharts (a later section)
crds/ CRDs installed once (a later section)
values.schema.json optional check of the values (a later section)
.helmignore files left out of the package
The chart is called shop: an nginx that serves a page from a ConfigMap, with a Service, and an Ingress that is off by default. Two numbers in Chart.yaml are easy to confuse. version is the version of the chart and must change whenever the chart changes (repositories identify a chart by name and version). appVersion is only information about what the chart deploys, and here the template uses it as the default image tag.
$ cat shop/Chart.yaml
apiVersion: v2
name: shop
description: A small web shop front end
type: application
version: 0.1.0
appVersion: "1.27"
$ cat shop/values.yaml
replicaCount: 2
image:
repository: nginx
tag: ""
pullPolicy: IfNotPresent
service:
type: ClusterIP
port: 80
message: "Welcome to the shop"
resources: {}
ingress:
enabled: false
host: shop.lab
$ cat shop/templates/_helpers.tpl
{{- define "shop.fullname" -}}
{{- printf "%s-%s" .Release.Name .Chart.Name | trunc 63 | trimSuffix "-" -}}
{{- end }}
{{- define "shop.selectorLabels" -}}
app.kubernetes.io/name: {{ .Chart.Name }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
{{- define "shop.labels" -}}
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
{{ include "shop.selectorLabels" . }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}
$ cat shop/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "shop.fullname" . }}
labels:
{{- include "shop.labels" . | nindent 4 }}
data:
index.html: |
<html><body><h1>{{ .Values.message }}</h1></body></html>
$ cat shop/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "shop.fullname" . }}
labels:
{{- include "shop.labels" . | nindent 4 }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
{{- include "shop.selectorLabels" . | nindent 6 }}
template:
metadata:
annotations:
checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
labels:
{{- include "shop.selectorLabels" . | nindent 8 }}
spec:
containers:
- name: nginx
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- containerPort: 80
volumeMounts:
- name: html
mountPath: /usr/share/nginx/html
{{- with .Values.resources }}
resources:
{{- toYaml . | nindent 10 }}
{{- end }}
volumes:
- name: html
configMap:
name: {{ include "shop.fullname" . }}
$ cat shop/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: {{ include "shop.fullname" . }}
labels:
{{- include "shop.labels" . | nindent 4 }}
spec:
type: {{ .Values.service.type }}
selector:
{{- include "shop.selectorLabels" . | nindent 4 }}
ports:
- port: {{ .Values.service.port }}
targetPort: 80
$ cat shop/templates/ingress.yaml
{{- if .Values.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "shop.fullname" . }}
labels:
{{- include "shop.labels" . | nindent 4 }}
spec:
rules:
- host: {{ .Values.ingress.host | quote }}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: {{ include "shop.fullname" . }}
port:
number: {{ .Values.service.port }}
{{- end }}
$ cat shop/templates/NOTES.txt
{{ include "shop.fullname" . }} is installed: {{ .Values.replicaCount }} replica(s) of nginx:{{ .Values.image.tag | default .Chart.AppVersion }}.
Look at the page with:
kubectl port-forward svc/{{ include "shop.fullname" . }} 8080:{{ .Values.service.port }}
$ cat shop/templates/tests/test-connection.yaml
apiVersion: v1
kind: Pod
metadata:
name: {{ include "shop.fullname" . }}-test
labels:
{{- include "shop.labels" . | nindent 4 }}
annotations:
"helm.sh/hook": test
spec:
restartPolicy: Never
containers:
- name: wget
image: busybox:1.36
command: ["wget", "-qO-", "http://{{ include "shop.fullname" . }}:{{ .Values.service.port }}"]
Check the chart for mistakes, look at the rendered output of one file, then install it:
$ helm lint shop
==> Linting shop
[INFO] Chart.yaml: icon is recommended
1 chart(s) linted, 0 chart(s) failed
$ helm template demo shop --show-only templates/service.yaml
---
# Source: shop/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: demo-shop
labels:
helm.sh/chart: shop-0.1.0
app.kubernetes.io/name: shop
app.kubernetes.io/instance: demo
app.kubernetes.io/version: "1.27"
app.kubernetes.io/managed-by: Helm
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: shop
app.kubernetes.io/instance: demo
ports:
- port: 80
targetPort: 80
$ helm install demo shop --wait --timeout 120s
NAME: demo
LAST DEPLOYED: Tue Oct 20 00:31:56 2026
NAMESPACE: default
STATUS: deployed
REVISION: 1
DESCRIPTION: Install complete
NOTES:
demo-shop is installed: 2 replica(s) of nginx:1.27.
Look at the page with:
kubectl port-forward svc/demo-shop 8080:80
$ helm test demo | grep -E 'TEST SUITE|Phase'
TEST SUITE: demo-shop-test
Phase: Succeeded
$ kubectl logs demo-shop-test
<html><body><h1>Welcome to the shop</h1></body></html>
helm lint reads the chart and complains about mistakes (the one INFO line about an icon is only a suggestion). helm test ran the pod from templates/tests and it fetched the page through the Service, which proves the pieces are connected and not only created. The test pod stays after the run so you can read its log. Delete it when you are done, or give it a hook-delete-policy.
Two values were left for this section. --set-json sets a whole structure, here the resources block that the Deployment template places only when it is set, and a file used with --set-file must not end with a newline, or the value does:
$ helm template t shop --set-json 'resources={"limits":{"cpu":"200m"}}' --show-only templates/deployment.yaml | grep -A3 ' resources:'
resources:
limits:
cpu: 200m
volumes:
$ printf 'Spring sale' > msg.txt; helm template t shop --set-file message=msg.txt --show-only templates/configmap.yaml | grep h1
<html><body><h1>Spring sale</h1></body></html>
$ echo 'Spring sale' > msg.txt; helm template t shop --set-file message=msg.txt --show-only templates/configmap.yaml 2>&1 | head -2
Error: YAML parse error on shop/templates/configmap.yaml: error converting YAML to JSON: yaml: line 15: could not find expected ':'
$ rm msg.txt
Templates: the language you need
Templates are plain YAML with actions between double braces, in the Go template language with some extra functions from the Sprig library. Rendering only ever produces text, and the text must then be valid YAML, which explains most errors. A short chart called play is used to try each piece, and every output below is real:
$ cat play/Chart.yaml
apiVersion: v2
name: play
version: 0.1.0
$ cat play/values.yaml
name: demo
port: 8080
debug: false
tags:
- web
- blue
labels:
team: ops
tier: front
resources:
limits:
cpu: 100m
memory: 64Mi
db:
password: ""
$ cat play/templates/cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Release.Name }}-{{ .Values.name }}
data:
port: {{ .Values.port | quote }}
missing: {{ .Values.nothing | default "fallback" | quote }}
upper: {{ .Values.name | upper | quote }}
joined: {{ join "," .Values.tags | quote }}
{{- if .Values.debug }}
level: debug
{{- else }}
level: info
{{- end }}
tags: |
{{- range .Values.tags }}
- {{ . }}
{{- end }}
labels: |
{{- range $k, $v := .Values.labels }}
{{ $k }}={{ $v }}
{{- end }}
limits: |
{{- toYaml .Values.resources.limits | nindent 4 }}
template: {{ tpl "release is {{ .Release.Name }}" . | quote }}
$ helm template r play | sed -n '4,40p'
kind: ConfigMap
metadata:
name: r-demo
data:
port: "8080"
missing: "fallback"
upper: "DEMO"
joined: "web,blue"
level: info
tags: |
- web
- blue
labels: |
team=ops
tier=front
limits: |
cpu: 100m
memory: 64Mi
template: "release is r"
What each line did:
{{ .Values.port }} a value. The leading dot is "the current context", and .Values is everything from values.yaml plus your overrides.
| quote a pipeline: the left side becomes the last argument of the function on the right. quote wraps text in quotes, which stops YAML from turning 8080 into a number or no into a boolean.
| default "fallback" used when the value is missing, empty, zero or false. Note that false and 0 count as empty, so a setting that may legitimately be false needs a different test.
{{- if ... }} {{- else }} {{- end }} conditions. A value is false when it is false, 0, empty or missing.
{{- range .Values.tags }} {{ . }} a loop. Inside it the dot is each item. With $k, $v := it loops over a map in sorted key order.
toYaml | nindent 4 writes a block of values as YAML and indents it. This pair is how every chart passes resources, nodeSelector, tolerations and similar blocks through. nindent starts with a newline, and indent does not, so using the wrong one gives a hard-to-read parse error.
tpl "text" . renders a string as a template, for values that themselves contain {{ }}.
include "name" . renders a named template (see the helpers section) and returns text, which can be piped. The older template keyword prints but cannot be piped, so use include.
Whitespace is the cause of most confusion. A dash inside the braces ({{- or -}}) trims the whitespace and newlines on that side, and the lines of the if and range above all start with {{- so that no blank lines are left. Without the dash the line break stays in the output, and in YAML that can change the meaning.
Where the values come from can be seen by changing them. A boolean and a list from the command line:
$ helm template r play --set debug=true | grep level
level: debug
$ helm template r play --set 'tags={a,b,c}' | grep joined
joined: "a,b,c"
$ helm template r play --set labels.team=dev --set labels.new=x | grep -A3 ' labels:'
labels: |
new=x
team=dev
tier=front
The built-in objects that templates can read:
.Values the merged values
.Release.Name the release name .Release.Namespace the namespace .Release.Revision the revision number
.Release.IsInstall and .Release.IsUpgrade true during an install or an upgrade
.Chart.Name .Chart.Version .Chart.AppVersion the fields of Chart.yaml
.Template.Name and .Template.BasePath the file being rendered
.Capabilities.KubeVersion and .Capabilities.APIVersions.Has "batch/v1" what the cluster supports, used to choose an API version
.Files the non-template files in the chart folder (files in the chart outside templates/ can be read with .Files.Get "name")
lookup reads live objects from the cluster during a real install or upgrade (for example to reuse an existing Secret instead of generating a new password on every upgrade). It returns nothing under helm template, where there is no cluster, which makes charts that depend on it hard to test. It was not run in the lab.
Helpers, labels and the checksum trick
A file whose name starts with an underscore, conventionally templates/_helpers.tpl, makes no objects. It holds named templates, which are functions you call with include. The shop chart has three. One builds the object name, one the labels that select the pods, and one the full set of labels. Look at the file again:
$ cat shop/templates/_helpers.tpl
{{- define "shop.fullname" -}}
{{- printf "%s-%s" .Release.Name .Chart.Name | trunc 63 | trimSuffix "-" -}}
{{- end }}
{{- define "shop.selectorLabels" -}}
app.kubernetes.io/name: {{ .Chart.Name }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
{{- define "shop.labels" -}}
helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
{{ include "shop.selectorLabels" . }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}
The first line of a define creates the name. printf, trunc 63 and trimSuffix keep the name valid for Kubernetes (names are limited to 63 characters in many places). include "shop.labels" . passes the current context (the dot) into the helper, which is why it can read .Release.Name. Names of helpers share one global space across the chart and all its subcharts, so prefix them with the chart name, as above, or two charts will overwrite each other's helpers.
A rule worth keeping: put the pod selector labels in one helper and use it for both the Deployment's selector and the pod template. A Deployment's selector cannot be changed after it is created, so if a later chart version adds a label to the selector, the upgrade fails. Keep the selector small and stable (the name and the instance), and put anything that changes, such as the chart version, only in the plain labels.
The second trick solves a real problem. Pods read a ConfigMap when they start. If you change the ConfigMap with an upgrade, nothing about the Deployment changes, so Kubernetes does nothing, and the pods keep serving the old content. The usual fix is already in the Deployment template of the shop chart: an annotation on the pod template that holds a hash of the ConfigMap's rendered text:
checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
Change the message and the hash changes, the pod template changes, and the Deployment rolls out new pods. Try it, and look at the annotation and the rollout history before and after:
$ kubectl get deployment demo-shop -o jsonpath='{.spec.template.metadata.annotations}{"\n"}'
{"checksum/config":"6cc7b58b438ccbb5826b556e9bdd4f54865da923be376cd65a3febef4398e557"}
$ kubectl rollout history deployment demo-shop
deployment.apps/demo-shop
REVISION CHANGE-CAUSE
1 <none>
$ helm upgrade demo shop --set message="Spring sale now on" --wait --timeout 90s | grep -E 'REVISION|STATUS'
STATUS: deployed
REVISION: 2
$ kubectl get deployment demo-shop -o jsonpath='{.spec.template.metadata.annotations}{"\n"}'
{"checksum/config":"14d17a94ac051df8d0dbcd487d5847cda06fee40f07980c507976704a24c7c80"}
$ kubectl rollout history deployment demo-shop
deployment.apps/demo-shop
REVISION CHANGE-CAUSE
1 <none>
2 <none>
The hash changed, and the Deployment has a second rollout revision, so the pods were replaced and now serve the new page. Without the annotation the ConfigMap would have been updated and the pods would not have noticed, and a Secret needs the same trick (with its own file name in the include). The same idea works for any file that pods read only at start. Put it back with helm rollback, which a later section covers, or just carry on.
Stop bad values: required, schema and the usual template errors
Helm fails early and with a message, if you ask it to. Two tools do that: required in the template, and a JSON Schema file next to values.yaml. They are worth the few minutes, because a chart gets used by people who never read its templates.
required stops with your own text when a value is missing or empty. It is the right tool for a value that has no sensible default, such as a password or a host name:
$ cat play/templates/bad.yaml
data:
pw: {{ required "db.password is required" .Values.db.password | quote }}
$ helm template r play 2>&1 | head -3
Error: execution error at (play/templates/bad.yaml:2:9): db.password is required
Use --debug flag to render out invalid YAML
$ helm template r play --set db.password=s3cret 2>&1 | grep pw
pw: "s3cret"
A schema checks types and ranges for every value, whether it comes from a file or from --set. Write it as values.schema.json in the chart:
$ cat play/values.schema.json
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"required": ["port"],
"properties": {
"port": {"type": "integer", "minimum": 1, "maximum": 65535},
"debug": {"type": "boolean"}
}
}
$ rm play/templates/bad.yaml
$ helm template r play --set port=99999 2>&1 | head -4
Error: values don't meet the specifications of the schema(s) in the following chart(s):
play:
- at '/port': maximum: got 99,999, want 65,535
$ helm template r play --set port=abc 2>&1 | head -4
Error: values don't meet the specifications of the schema(s) in the following chart(s):
play:
- at '/port': got string, want integer
$ helm template r play --set debug=maybe 2>&1 | head -4
Error: values don't meet the specifications of the schema(s) in the following chart(s):
play:
- at '/debug': got string, want boolean
The schema runs on install, upgrade, template and lint, so a typo is caught before anything reaches the cluster (--skip-schema-validation turns it off, and should not be needed). The messages name the path of the bad value.
The mistakes you will meet most, with the real messages. A path through a map that does not exist:
$ cat play/templates/bad.yaml
data:
pw: {{ .Values.db.pass.word }}
$ helm template r play 2>&1 | head -4
Error: play/templates/bad.yaml:2:16
executing "play/templates/bad.yaml" at <.Values.db.pass.word>:
nil pointer evaluating interface {}.word
A "nil pointer" means a value in the path is missing: here db has no key called pass. For a key that may be absent, test it first with if, or use default, or the safer dig and hasKey functions. The other family is the output not being valid YAML, which is nearly always indentation:
$ cat play/templates/bad.yaml
a: 1
b: 2
$ helm template r play 2>&1 | head -2
Error: YAML parse error on play/templates/bad.yaml: error converting YAML to JSON: yaml: line 2: mapping values are not allowed in this context
$ helm lint play 2>&1 | head -5
==> Linting play
[INFO] Chart.yaml: icon is recommended
[ERROR] templates/bad.yaml: unable to parse YAML: error converting YAML to JSON: yaml: line 2: mapping values are not allowed in this context
Error: 1 chart(s) linted, 1 chart(s) failed
$ rm play/templates/bad.yaml
Both messages name the file and the line of the rendered output. To see the broken YAML itself, add --debug to helm template, which prints what was rendered even when it is not valid. A missing nindent, or one with the wrong number, is the usual cause.
Look before you apply: template, dry run, lint and diff
Every Helm command that changes the cluster has a way to look first. Use them in this order, from cheapest to most thorough.
helm lint checks the chart's structure and renders it. It needs no cluster:
$ helm lint shop --strict
==> Linting shop
[INFO] Chart.yaml: icon is recommended
1 chart(s) linted, 0 chart(s) failed
helm template renders the YAML on your machine. It needs no cluster either, so it works in CI, but it cannot know what the cluster supports or already has. Add --show-only to see one file, and --debug to see invalid YAML:
$ helm template demo shop --show-only templates/service.yaml | head -8
---
# Source: shop/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: demo-shop
labels:
helm.sh/chart: shop-0.1.0
A dry run goes further. --dry-run=client (the default meaning of plain --dry-run) renders only. In Helm 4, --dry-run=server sends the objects to the API server for validation without saving them, so it catches things the client cannot: an unknown field, a wrong apiVersion, a missing CRD, a webhook that would reject the object:
$ helm install check shop --dry-run=server | head -7
NAME: check
LAST DEPLOYED: Tue Oct 20 00:32:10 2026
NAMESPACE: default
STATUS: pending-install
REVISION: 1
DESCRIPTION: Dry run complete
HOOKS:
$ helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
demo default 2 2026-10-20 00:32:04.446065254 +0000 UTC deployed shop-0.1.0 1.27
The release was not created (the list shows only the earlier one). The output of a dry run includes any Secrets in clear text, so use --hide-secret when you capture it in CI logs.
The most useful check is a diff against what is running. The helm-diff plug-in is the usual tool. Plug-ins in Helm 4 are checked for signatures, and a plug-in from GitHub with no signature needs --verify=false, which you should only give for a source you trust:
$ helm plugin install https://github.com/databus23/helm-diff --verify=false 2>&1 | tail -2
Preparing to install into /home/ubuntu/.local/share/helm/plugins/helm-diff
Installed plugin: diff
$ helm diff upgrade demo shop --set message=changed --set replicaCount=3 --no-color 2>&1 | grep -E '^[+-] '
- <html><body><h1>Spring sale now on</h1></body></html>
+ <html><body><h1>changed</h1></body></html>
- replicas: 2
+ replicas: 3
- checksum/config: 14d17a94ac051df8d0dbcd487d5847cda06fee40f07980c507976704a24c7c80
+ checksum/config: 2f0abd86a8d7b80422b4b98294b7aa63e9aae041df996a9268d7fcd0f13f7e01
The lines with a minus are in the cluster now, and the lines with a plus are what the upgrade would make. Notice the checksum line changes with the message, which is the previous section at work. For the same check without a plug-in, render and give the result to kubectl:
$ helm template demo shop --set replicaCount=3 | kubectl diff -f - 2>&1 | grep -E '^[+-] ' | grep -v -E 'generation|checksum' | head -4
- replicas: 2
+ replicas: 3
- <html><body><h1>Spring sale now on</h1></body></html>
+ <html><body><h1>Welcome to the shop</h1></body></html>
(kubectl diff prints the managed-field noise that the plug-in hides, and exits with status 1 when there are differences, which is how to use it in a pipeline.) Diff before every upgrade that matters. The most dangerous upgrade is the one where a values file was edited and the chart version moved at once: the diff tells you what that really does.
Upgrade, history and rollback
helm upgrade applies a new chart version, or new values, to an existing release and records a new revision. The history is the safety net. Start by looking at what the release has now:
$ helm history demo
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Tue Oct 20 00:31:56 2026 superseded shop-0.1.0 1.27 Install complete
2 Tue Oct 20 00:32:04 2026 deployed shop-0.1.0 1.27 Upgrade complete
The first trap catches almost everyone. An upgrade does not remember the --set flags of the last one. Each upgrade computes the values from the chart defaults plus what you give THIS time, so a setting you passed before and leave out now silently goes back to its default. The previous upgrade set the message, and this one only sets the image tag:
$ helm upgrade demo shop --set image.tag=1.28 --wait --timeout 90s | grep -E 'REVISION|STATUS'
STATUS: deployed
REVISION: 3
$ helm get values demo
USER-SUPPLIED VALUES:
image:
tag: "1.28"
$ kubectl get configmap demo-shop -o jsonpath='{.data.index\.html}{"\n"}'
<html><body><h1>Welcome to the shop</h1></body></html>
The message is gone. There are two sound ways to avoid that. The first is to keep every setting in a values file in git and give the full file every time (-f values-prod.yaml), so the file is the truth and no one has to remember anything. The second is --reuse-values, which starts from the values of the last release and merges your new flags into them:
$ helm upgrade demo shop --reuse-values --set message="Spring sale now on" --set replicaCount=3 --wait --timeout 90s | grep -E 'REVISION'
REVISION: 4
$ helm get values demo
USER-SUPPLIED VALUES:
image:
tag: "1.28"
message: Spring sale now on
replicaCount: 3
--reuse-values has its own trap, which shows up when the chart changes. It reuses the old values and ignores any new default the new chart version added, so a template that reads a new value finds nothing. Simulate a chart upgrade: version 0.2.0 adds a footer setting with a default, and the page template uses it.
$ sed -i 's/^version: 0.1.0/version: 0.2.0/' shop/Chart.yaml
$ printf 'footer:\n text: thanks for shopping\n' >> shop/values.yaml
$ sed -i 's|<h1>{{ .Values.message }}</h1>|<h1>{{ .Values.message }}</h1><p>{{ .Values.footer.text }}</p>|' shop/templates/configmap.yaml
$ grep -n footer shop/values.yaml shop/templates/configmap.yaml
shop/values.yaml:19:footer:
shop/templates/configmap.yaml:9: <html><body><h1>{{ .Values.message }}</h1><p>{{ .Values.footer.text }}</p></body></html>
$ helm upgrade demo shop --reuse-values 2>&1 | head -6
Error: UPGRADE FAILED: shop/templates/deployment.yaml:15:28
executing "shop/templates/deployment.yaml" at <include (print $.Template.BasePath "/configmap.yaml") .>:
error calling include:
shop/templates/configmap.yaml:9:59
executing "shop/templates/configmap.yaml" at <.Values.footer.text>:
nil pointer evaluating interface {}.text
The error is a nil pointer on footer.text, because the old values had no footer and --reuse-values refused to look at the new defaults. The flag made for this is --reset-then-reuse-values: take the new chart's defaults, then apply the last release's values, then your flags.
$ helm upgrade demo shop --reset-then-reuse-values --wait --timeout 90s | grep -E 'REVISION'
REVISION: 5
$ helm get values demo
USER-SUPPLIED VALUES:
image:
tag: "1.28"
message: Spring sale now on
replicaCount: 3
$ helm get values demo -a | head -8
COMPUTED VALUES:
footer:
text: thanks for shopping
image:
pullPolicy: IfNotPresent
repository: nginx
tag: "1.28"
ingress:
helm get values -a is the computed result: it shows the new footer default and the old settings together. There is a third flag, --reset-values, which means "forget everything and use only the chart defaults plus my flags", and with nothing else given it leaves nothing supplied at all. Prefer a values file over any of these.
Every upgrade, install and rollback adds a revision, and the history shows each with its status. Add a note to a revision with --description, which helps in a long history:
$ helm upgrade demo shop --reset-then-reuse-values --description "chart 0.2.0, footer added" --wait --timeout 90s | grep -E 'DESCRIPTION'
DESCRIPTION: chart 0.2.0, footer added
$ helm history demo
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Tue Oct 20 00:31:56 2026 superseded shop-0.1.0 1.27 Install complete
2 Tue Oct 20 00:32:04 2026 superseded shop-0.1.0 1.27 Upgrade complete
3 Tue Oct 20 00:32:19 2026 superseded shop-0.1.0 1.27 Upgrade complete
4 Tue Oct 20 00:32:23 2026 superseded shop-0.1.0 1.27 Upgrade complete
5 Tue Oct 20 00:32:28 2026 superseded shop-0.2.0 1.27 Upgrade complete
6 Tue Oct 20 00:32:33 2026 deployed shop-0.2.0 1.27 chart 0.2.0, footer added
To undo, roll back to a revision number. A rollback is itself a new revision (not a rewind): the history keeps growing, and the description says which revision was restored.
$ helm rollback demo 1 --wait --timeout 90s
Rollback was a success! Happy Helming!
$ helm history demo | tail -3
5 Tue Oct 20 00:32:28 2026 superseded shop-0.2.0 1.27 Upgrade complete
6 Tue Oct 20 00:32:33 2026 superseded shop-0.2.0 1.27 chart 0.2.0, footer added
7 Tue Oct 20 00:32:34 2026 deployed shop-0.1.0 1.27 Rollback to 1
$ helm get values demo
USER-SUPPLIED VALUES:
null
$ kubectl get configmap demo-shop -o jsonpath='{.data.index\.html}{"\n"}'
<html><body><h1>Welcome to the shop</h1></body></html>
Helm keeps the last 10 revisions of a release unless you change it (--history-max, or HELM_MAX_HISTORY), and older ones are deleted as new ones arrive, so a revision you want to keep for a year does not live in Helm. A rollback restores the objects Helm manages, so it brings back the old ConfigMap and the old image tag. It does not undo what happened outside Helm: a database migration that ran, a PersistentVolume that was filled, or an object created by hand.
An uninstalled release can leave its history behind with --keep-history, and then be brought back with a rollback. Most people never want this, because the history cannot be seen in a plain helm list, and the name stays taken:
$ helm install keep shop --wait --timeout 90s | grep -E 'STATUS'
STATUS: deployed
$ helm uninstall keep --keep-history
release "keep" uninstalled
$ helm list --uninstalled
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
keep default 1 2026-10-20 00:32:37.811663903 +0000 UTC uninstalled shop-0.2.0 1.27
$ helm rollback keep 1 --wait --timeout 90s
Rollback was a success! Happy Helming!
$ helm list | grep keep
keep default 2 2026-10-20 00:32:40.450949426 +0000 UTC deployed shop-0.2.0 1.27
$ helm uninstall keep | head -1
release "keep" uninstalled
Waiting, failing and rolling back safely
The most important thing about helm install and helm upgrade is that by default they do not wait. They send the objects to the API server and finish. "deployed" means Helm sent everything without an error, and says nothing about whether the pods work. Here is an install with an image tag that does not exist:
$ helm install w1 shop --set image.tag=1.99-nope | grep -E 'STATUS'
STATUS: deployed
$ sleep 8; kubectl get pods -l app.kubernetes.io/instance=w1
NAME READY STATUS RESTARTS AGE
w1-shop-6dff474986-2gfh5 0/1 ErrImagePull 0 9s
w1-shop-6dff474986-vxxvx 0/1 ErrImagePull 0 9s
$ helm status w1 | grep STATUS
STATUS: deployed
NAME READY STATUS RESTARTS AGE
$ helm uninstall w1 | head -1
release "w1" uninstalled
The release says deployed, and both pods are in ErrImagePull. A pipeline that only checks the exit status of helm would call this a success. The flag that fixes it is --wait, with a limit from --timeout (5 minutes unless you change it). Helm 4 has three ways to wait, chosen with --wait=STRATEGY:
watcher watch each object's status until it is ready (--wait with no value means this). It reports why it is not ready.
hookOnly the default when --wait is not given: wait for hooks (a Job in a pre-upgrade hook) but not for the workload.
legacy the older polling method from Helm 3. Not run in the lab.
--wait-for-jobs also waits for Jobs to finish. Now the same install with a wait and a short timeout:
$ helm install w1 shop --set image.tag=1.99-nope --wait --timeout 20s 2>&1 | head -4
Error: INSTALLATION FAILED: resource Deployment/default/w1-shop not ready. status: InProgress, message: Available: 0/2
context deadline exceeded
$ helm list --failed
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
w1 default 1 2026-10-20 00:32:52.100255933 +0000 UTC failed shop-0.2.0 1.27
$ kubectl get pods -l app.kubernetes.io/instance=w1
NAME READY STATUS RESTARTS AGE
w1-shop-6dff474986-99sbq 0/1 ImagePullBackOff 0 20s
w1-shop-6dff474986-zw7xj 0/1 ImagePullBackOff 0 20s
The command fails with a message that names the object and what it was still waiting for, and the release is marked failed. The objects stay in the cluster, which lets you look at them with kubectl describe, but they also keep running (or failing). Remove them with helm uninstall w1 before trying again, because a failed install still owns the release name.
For upgrades, the failed state is no worse: the old pods keep serving while the rollout is stuck, because the Deployment's rolling update does not take down good pods until new ones are ready. The release is marked failed, though, and the next good upgrade replaces it. If you want Helm to put things back by itself, use --rollback-on-failure (in Helm 3 this was called --atomic, which Helm 4 still accepts and warns is deprecated):
$ helm upgrade demo shop --set image.tag=1.99-nope --reset-then-reuse-values --rollback-on-failure --timeout 40s 2>&1 | grep -E '^Error'
Error: UPGRADE FAILED: release demo failed, and has been rolled back due to rollback-on-failure being set: resource Deployment/default/demo-shop not ready. status: InProgress, message: Updated: 1/2
$ helm history demo | tail -3
7 Tue Oct 20 00:32:34 2026 superseded shop-0.1.0 1.27 Rollback to 1
8 Tue Oct 20 00:33:13 2026 failed shop-0.2.0 1.27 Upgrade "demo" failed: resource Deployment/default/demo-shop not ready. status: InProgress, message: Updated: ...
9 Tue Oct 20 00:33:53 2026 deployed shop-0.1.0 1.27 Rollback to 7
$ kubectl get pods -l app.kubernetes.io/instance=demo,app.kubernetes.io/name=shop
NAME READY STATUS RESTARTS AGE
demo-shop-6b78b94bcc-8p8hz 0/1 Terminating 0 41s
demo-shop-8677d4f449-n8r42 1/1 Running 0 80s
demo-shop-8677d4f449-strfm 1/1 Running 0 79s
demo-shop-test 0/1 Completed 0 116s
The history shows the failed revision and, right after it, a new revision that restored the last good one, and the pods are the old, working ones. On an install, --rollback-on-failure removes the release. --wait is turned on for you with the watcher strategy, and the timeout still applies, so give it a number that matches how long your pods really take to start, plus some margin. Without the margin a slow image pull makes a healthy release fail and roll back.
The worst state is a release stuck in pending-upgrade (or pending-install). It happens when the helm process dies in the middle of an upgrade: a CI job that was cancelled, a closed laptop, a lost network. The release record says an operation is running, so the next one is refused. Simulate it by killing helm four seconds into an upgrade:
$ (timeout -s KILL 4 helm upgrade demo shop --reset-then-reuse-values --set replicaCount=4 --wait --timeout 60s; echo "helm was killed")
bash: line 1: 11469 Killed timeout -s KILL 4 helm upgrade demo shop --reset-then-reuse-values --set replicaCount=4 --wait --timeout 60s
helm was killed
$ helm history demo | tail -2
9 Tue Oct 20 00:33:53 2026 deployed shop-0.1.0 1.27 Rollback to 7
10 Tue Oct 20 00:33:54 2026 pending-upgrade shop-0.2.0 1.27 Preparing upgrade
$ helm upgrade demo shop --reset-then-reuse-values --set replicaCount=4 --wait 2>&1 | head -2
Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
Helm will not start another operation, and gives a message that does not say how to fix it. The way out is to roll back to the last good revision, which clears the pending state. Find the number of the deployed revision (the one before the pending one) from the JSON of the history, and roll back to it:
$ helm history demo -o json | jq -r '.[] | select(.status=="deployed") | .revision'
9
$ helm rollback demo $(helm history demo -o json | jq -r '.[] | select(.status=="deployed") | .revision') --wait --timeout 60s
Rollback was a success! Happy Helming!
$ helm history demo | tail -3
9 Tue Oct 20 00:33:53 2026 superseded shop-0.1.0 1.27 Rollback to 7
10 Tue Oct 20 00:33:54 2026 pending-upgrade shop-0.2.0 1.27 Preparing upgrade
11 Tue Oct 20 00:33:59 2026 deployed shop-0.1.0 1.27 Rollback to 9
$ helm get values demo --revision 1
Error: release: not found
(exit status 1)
$ helm get manifest demo --revision 1 | grep image:
Error: release: not found
(exit status 1)
Choose the revision with care. A rollback to a revision that has a broken image waits for the full --timeout (the lab's default of 5 minutes) and fails, which is what happened on the first try of this test, so always check what a revision contains with helm get values demo --revision N, or helm get manifest demo --revision N, before rolling back to it. Running upgrades from one place at a time (a CI job with a concurrency limit) prevents most of these stuck states.
Hooks and tests
A hook is a Kubernetes object that Helm runs at a chosen moment in a release's life, instead of installing it as part of the release. The usual one is a Job that runs a database migration before the new pods start. A hook is a normal template with an annotation:
helm.sh/hook when to run it: pre-install, post-install, pre-upgrade, post-upgrade, pre-delete, post-delete, pre-rollback, post-rollback, or test. Several can be given, separated by commas.
helm.sh/hook-weight an order among hooks of the same kind (a string holding a number, lowest first)
helm.sh/hook-delete-policy when to remove the hook object: before-hook-creation (the default), hook-succeeded, hook-failed
A chart called hooked has a migration Job (before an install and before an upgrade), a ConfigMap, and an announcement Job (after an install and after an upgrade). A value decides whether the migration fails:
$ cat hooked/Chart.yaml
apiVersion: v2
name: hooked
version: 0.1.0
appVersion: "1.0"
$ cat hooked/values.yaml
failMigration: false
$ cat hooked/templates/migrate.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: {{ .Release.Name }}-migrate
annotations:
"helm.sh/hook": pre-install,pre-upgrade
"helm.sh/hook-weight": "0"
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
backoffLimit: 0
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: busybox:1.36
command: ["sh", "-c", "echo running the database migration; sleep 3; {{ if .Values.failMigration }}echo migration failed; exit 1{{ else }}echo migration done{{ end }}"]
$ cat hooked/templates/cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Release.Name }}-app
data:
note: "created after the pre-install hook"
$ cat hooked/templates/keep.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Release.Name }}-kept
annotations:
"helm.sh/resource-policy": keep
data:
note: "this survives helm uninstall"
$ cat hooked/templates/announce.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: {{ .Release.Name }}-announce
annotations:
"helm.sh/hook": post-install,post-upgrade
"helm.sh/hook-delete-policy": before-hook-creation
spec:
backoffLimit: 0
template:
spec:
restartPolicy: Never
containers:
- name: announce
image: busybox:1.36
command: ["sh", "-c", "echo release {{ .Release.Name }} revision {{ .Release.Revision }} is out"]
$ helm install hk hooked --wait --wait-for-jobs --timeout 90s | grep -E 'STATUS|REVISION'
STATUS: deployed
REVISION: 1
$ kubectl get jobs
NAME STATUS COMPLETIONS DURATION AGE
hk-announce Complete 1/1 3s 3s
$ kubectl logs job/hk-announce
release hk revision 1 is out
$ helm get manifest hk | grep -E '^kind|^ name:'
kind: ConfigMap
name: hk-app
kind: ConfigMap
name: hk-kept
$ helm get hooks hk | grep -E 'Source|helm.sh/hook"'
# Source: hooked/templates/announce.yaml
"helm.sh/hook": post-install,post-upgrade
# Source: hooked/templates/migrate.yaml
"helm.sh/hook": pre-install,pre-upgrade
The order was: the migration Job ran and finished (Helm waited for it, and with the hook-succeeded policy removed it), then the ConfigMap was created, then the announcement Job ran. Only the announcement is still around, because its policy removes the old one only when the next one is created. helm get manifest does not list the hooks: they are not part of the release, and Helm does not manage them after they run (they are not deleted by helm uninstall either, unless a delete policy says so).
A hook that fails stops the operation. Make the migration fail on an upgrade:
$ helm upgrade hk hooked --set failMigration=true --timeout 60s 2>&1 | grep -E '^Error'
Error: UPGRADE FAILED: pre-upgrade hooks failed: resource Job/default/hk-migrate not ready. status: Failed, message: Job Failed. failed: 1/1
$ kubectl get jobs
NAME STATUS COMPLETIONS DURATION AGE
hk-announce Complete 1/1 3s 11s
hk-migrate Failed 0/1 7s 7s
$ kubectl logs job/hk-migrate
running the database migration
migration failed
$ helm history hk
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Tue Oct 20 00:34:03 2026 deployed hooked-0.1.0 1.0 Install complete
2 Tue Oct 20 00:34:15 2026 failed hooked-0.1.0 1.0 Upgrade "hk" failed: pre-upgrade hooks failed: resource Job/default/hk-migrate not ready. status: Failed, message: Job Failed. failed: 1/1
The upgrade stopped before the Deployment or the ConfigMap changed, the release is marked failed, and the Job stays so that you can read its log (the hook-succeeded policy removes it only when it succeeds, and adding hook-failed would remove it also on failure, which hides the evidence). Keep these rules in mind for hooks. They run as part of the Helm command, so a slow hook makes the command slow, and --timeout applies to them. They run again on every upgrade, so a migration must be safe to repeat. A Job needs backoffLimit set deliberately, because the default retries six times before it fails. And an ordinary object can opt out of removal with the annotation helm.sh/resource-policy: keep, which the ConfigMap above carries:
$ helm uninstall hk | head -1
These resources were kept due to the resource policy:
$ kubectl get configmap hk-kept hk-app 2>&1 | tail -3
NAME DATA AGE
hk-kept 1 12s
Error from server (NotFound): configmaps "hk-app" not found
The kept ConfigMap is still there, and Helm no longer knows about it. Use that for things that hold data, and clean up by hand on purpose.
Tests are hooks of the kind test. Nothing runs them until you ask with helm test RELEASE, as the shop chart showed earlier. Use them for a check that the release really answers (a request through the Service, a database login), not for unit tests of the templates.
CRDs: installed once, never upgraded, never removed
Charts for operators and add-ons often bring their own CustomResourceDefinitions. A CRD is cluster-wide, and removing it deletes every object of that kind, so Helm treats files in a chart's crds/ folder with great caution. The rules, which are all easy to forget:
- Files in crds/ are installed first, before the templates, so that the custom resources in the templates can be created.
- They are plain YAML, with no templating at all.
- Helm installs them on helm install. It never changes them on helm upgrade, and never deletes them on helm uninstall.
- helm template and a dry run leave them out unless you add --include-crds.
The hooked chart has a CRD in crds/. Install it again (the chart is installed fresh), then change the CRD file in the chart and upgrade:
$ cat hooked/crds/widget.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: widgets.demo.techcommune.org
spec:
group: demo.techcommune.org
names:
kind: Widget
plural: widgets
scope: Namespaced
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
size:
type: integer
$ helm template x hooked | grep -c CustomResourceDefinition; true
0
$ helm template x hooked --include-crds | grep -c CustomResourceDefinition
1
$ helm install hk hooked --wait --timeout 90s | grep -E 'STATUS'
STATUS: deployed
$ kubectl get crd widgets.demo.techcommune.org
NAME CREATED AT
widgets.demo.techcommune.org 2026-10-20T00:34:24Z
Now add a second property to the CRD in the chart and upgrade:
$ cat hooked/crds/widget.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: widgets.demo.techcommune.org
spec:
group: demo.techcommune.org
names:
kind: Widget
plural: widgets
scope: Namespaced
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
size:
type: integer
color:
type: string
$ helm upgrade hk hooked --wait --timeout 90s | grep -E 'STATUS|REVISION'
STATUS: deployed
REVISION: 2
$ kubectl get crd widgets.demo.techcommune.org -o jsonpath='{.spec.versions[0].schema.openAPIV3Schema.properties.spec.properties}{"\n"}'
{"size":{"type":"integer"}}
$ helm uninstall hk | head -1
These resources were kept due to the resource policy:
$ kubectl get crd widgets.demo.techcommune.org
NAME CREATED AT
widgets.demo.techcommune.org 2026-10-20T00:34:24Z
The cluster still has the old schema, with only size, even though the chart has color and the release is at a new revision. And after the uninstall the CRD is still there. Both are deliberate, so a chart upgrade cannot destroy your custom resources. The consequence is that CRDs need their own life cycle. The common ways are: apply the CRDs yourself with kubectl apply --server-side before you upgrade the chart (many projects publish them as a separate manifest or a separate "-crds" chart, which is exactly how k3s ships Traefik, as seen in the first sections), or keep the CRDs in templates/ so Helm upgrades them, at the price that uninstalling the release then deletes the CRD and every object of that kind. Read the chart's own documentation on which one it uses, before the first upgrade, not after.
Remove the lab's CRD when you are done:
$ kubectl delete crd widgets.demo.techcommune.org
customresourcedefinition.apiextensions.k8s.io "widgets.demo.techcommune.org" deleted
$ kubectl delete configmap hk-kept
configmap "hk-kept" deleted from default namespace
Subcharts and dependencies
A chart can depend on other charts, for example an application that needs a cache. The dependencies are listed in Chart.yaml and fetched into the charts/ folder. Here the application chart (app) depends on a chart called cache, found in the folder next to it. A real dependency would name a repository URL or an oci:// address in the same field. The condition makes the dependency optional:
$ cat cache/Chart.yaml
apiVersion: v2
name: cache
version: 0.3.0
$ cat cache/values.yaml
size: 64
$ cat cache/templates/cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Release.Name }}-cache
data:
size: {{ .Values.size | quote }}
region: {{ .Values.global.region | default "unset" | quote }}
$ cat app/Chart.yaml
apiVersion: v2
name: app
version: 1.0.0
dependencies:
- name: cache
version: "0.3.0"
repository: "file://../cache"
condition: cache.enabled
$ cat app/values.yaml
global:
region: eu-west
cache:
enabled: true
size: 256
$ cat app/templates/cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Release.Name }}-app
data:
cacheEnabled: {{ .Values.cache.enabled | quote }}
$ helm template x app 2>&1 | head -2
Error: an error occurred while checking for chart dependencies. You may need to run 'helm dependency build' to fetch missing dependencies: found in Chart.yaml, but missing in charts/ directory: cache
Helm refuses, because the dependency is declared and not downloaded. helm dependency update fetches it and writes Chart.lock with the exact versions found, and helm dependency build installs exactly what Chart.lock says (use build in CI, so the result does not drift):
$ helm dependency update app
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "examples" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 1 charts
Deleting outdated charts
$ ls app app/charts
app:
Chart.lock
Chart.yaml
charts
templates
values.yaml
app/charts:
cache-0.3.0.tgz
$ cat app/Chart.lock
dependencies:
- name: cache
repository: file://../cache
version: 0.3.0
digest: sha256:02f6663901301c2944ef5358067433dceae56e1cc5d6507434a350c66062e4fe
generated: "2026-10-20T00:34:48.417694124Z"
$ helm dependency list app
NAME VERSION REPOSITORY STATUS
cache 0.3.0 file://../cache ok
The subchart is now a packaged archive inside charts/. Values for a subchart go under a key with the subchart's name, so cache.size in the parent's values sets size in the cache chart, which is why the 256 below beats the subchart's own default of 64. The special key global is shared with every subchart, for settings such as a region or an image registry that all of them need:
$ helm template x app | grep -E '^# Source|size:|region:|cacheEnabled'
# Source: app/charts/cache/templates/cm.yaml
size: "256"
region: "eu-west"
# Source: app/templates/cm.yaml
cacheEnabled: "true"
$ helm template x app --set cache.size=1024 | grep size:
size: "1024"
$ helm template x app --set cache.enabled=false | grep '^# Source'
# Source: app/templates/cm.yaml
With the condition false, the subchart is left out altogether. Subcharts can also be given an alias (to use one chart twice under different names), grouped with tags, and have values imported back into the parent with import-values. Those were not run. Three warnings. A subchart is a full chart that you did not write, so everything said about reading charts before installing them applies to each one. Names of named templates are shared across the parent and its subcharts, so a clash overwrites silently. And a subchart's objects are part of the parent release, with the same revision and the same rollback, so they cannot be upgraded separately.
Package and publish: a chart repository
helm package turns a chart folder into a versioned archive, named after the chart name and version, and runs the dependency check on the way. A chart repository is nothing more than a web server that has these archives and an index.yaml, which helm repo index creates. The lab serves it from a folder with Python's built-in web server:
$ helm package app shop
Successfully packaged chart and saved it to: /home/ubuntu/lab/app-1.0.0.tgz
Successfully packaged chart and saved it to: /home/ubuntu/lab/shop-0.2.0.tgz
$ ls *.tgz
app-1.0.0.tgz
shop-0.2.0.tgz
$ tar -tzf app-1.0.0.tgz
app/Chart.yaml
app/Chart.lock
app/values.yaml
app/templates/cm.yaml
app/charts/cache/Chart.yaml
app/charts/cache/values.yaml
app/charts/cache/templates/cm.yaml
$ mkdir -p repo && cp app-1.0.0.tgz shop-0.2.0.tgz repo/ && helm repo index repo --url http://localhost:8879 && cat repo/index.yaml | head -24
apiVersion: v1
entries:
app:
- apiVersion: v2
created: "2026-10-20T00:34:49.90959369Z"
dependencies:
- condition: cache.enabled
name: cache
repository: file://../cache
version: 0.3.0
digest: 1e88d145620873441de5b361f62ed6b2477e5cceed31dfe053c14b89aa7c5b61
name: app
urls:
- http://localhost:8879/app-1.0.0.tgz
version: 1.0.0
shop:
- apiVersion: v2
appVersion: "1.27"
created: "2026-10-20T00:34:49.910104055Z"
description: A small web shop front end
digest: 2ce5562f2339280d60f87525cbceacd596e27713690356209d3a3bfc7bb4179b
name: shop
type: application
urls:
$ helm repo add local http://localhost:8879
"local" has been added to your repositories
$ helm search repo local
NAME CHART VERSION APP VERSION DESCRIPTION
local/app 1.0.0
local/shop 0.2.0 1.27 A small web shop front end
$ helm install fromrepo local/shop --version 0.2.0 --wait --timeout 90s | grep -E 'NAME|STATUS|CHART'
NAME: fromrepo
NAMESPACE: default
STATUS: deployed
$ helm list | grep fromrepo
fromrepo default 1 2026-10-20 00:34:52.614042617 +0000 UTC deployed shop-0.2.0 1.27
The index has an entry for each chart version with the address of the archive and a digest (a checksum), which Helm verifies after the download. For a real repository, serve the folder over HTTPS from any static host (nginx, an S3 bucket behind a CDN, GitHub Pages), and use the real URL when you run helm repo index. To add a new version, package it, copy it next to the others and run helm repo index again with --merge index.yaml, so the old entries stay. Clients pick up new versions after helm repo update.
Treat published versions as immutable. A chart at version 0.2.0 must always mean the same files, and a fix is 0.2.1. Charts that change under the same version break rollbacks and caches, and no one can tell what was deployed. Bump version in Chart.yaml on every change, and keep appVersion for the software inside.
helm package --sign adds a signature file (provenance) that helm install --verify can check against a public key. It needs a GPG key, and was not run in the lab. Without it, the digest in the index protects against a damaged download and does not prove who made the chart.
Tidy up the lab repository before the next section:
$ helm uninstall fromrepo | head -1
release "fromrepo" uninstalled
$ helm repo remove local
"local" has been removed from your repositories
Package and publish: an OCI registry
The same registries that store container images can store charts, and many teams prefer that: one place, one login, the same access control. There is no helm repo add and no index.yaml. A chart is addressed as oci://HOST/PATH, the chart name is the last part of the path, and the version is the image tag. The lab runs the official registry image next to the cluster (on the same container network, so the lab nodes reach it as registry:5000):
$ docker run -d --name registry --network k8net registry:2
$ helm push shop-0.2.0.tgz oci://registry:5000/charts 2>&1 | head -3
Error: failed to perform "Exists" on destination: Head "https://registry:5000/v2/charts/shop/manifests/sha256:aed1fa353762b969e847714bbb833944e487c18251b6dcec1871fe00eaa5ca99": http: server gave HTTP response to HTTPS client
Helm uses HTTPS for registries, and this lab registry speaks plain HTTP, so the first push fails. For a lab, tell Helm to accept HTTP with --plain-http. Never do that for a real registry:
$ helm push shop-0.2.0.tgz oci://registry:5000/charts --plain-http
Pushed: registry:5000/charts/shop:0.2.0
Digest: sha256:aed1fa353762b969e847714bbb833944e487c18251b6dcec1871fe00eaa5ca99
$ curl -s http://registry:5000/v2/_catalog; echo; curl -s http://registry:5000/v2/charts/shop/tags/list; echo
{"repositories":["charts/shop"]}
{"name":"charts/shop","tags":["0.2.0"]}
The registry lists the chart as the repository charts/shop, with the tag 0.2.0 (the chart version). Everything that worked with a repository works from the registry by address:
$ helm show chart oci://registry:5000/charts/shop --version 0.2.0 --plain-http | tail -8
Pulled: registry:5000/charts/shop:0.2.0
Digest: sha256:aed1fa353762b969e847714bbb833944e487c18251b6dcec1871fe00eaa5ca99
apiVersion: v2
appVersion: "1.27"
description: A small web shop front end
name: shop
type: application
version: 0.2.0
$ helm install fromoci oci://registry:5000/charts/shop --version 0.2.0 --plain-http --wait --timeout 90s | grep -E 'NAME|STATUS|CHART'
NAME: fromoci
NAMESPACE: default
STATUS: deployed
$ helm pull oci://registry:5000/charts/shop --version 0.2.0 --plain-http --destination /tmp | tail -1
Digest: sha256:aed1fa353762b969e847714bbb833944e487c18251b6dcec1871fe00eaa5ca99
$ helm uninstall fromoci | head -1
release "fromoci" uninstalled
A registry that needs a login uses helm registry login HOST (and helm registry logout), and Helm reuses the credentials of Docker and Podman. That part, and OCI registries with HTTPS and tokens such as GitHub Container Registry or a cloud registry, were not run, since the lab registry has no authentication. Use a version in every OCI address (--version, or the :tag form), or you take whatever the registry gives as latest, and a CI job that works today fails tomorrow.
Where a release lives: the release Secrets
Helm has no database of its own. Each revision of each release is stored in the cluster, by default as a Secret in the release's namespace, named sh.helm.release.v1.NAME.vN and labelled owner=helm. Look at them:
$ kubectl get secrets -l owner=helm,name=demo
NAME TYPE DATA AGE
sh.helm.release.v1.demo.v10 helm.sh/release.v1 1 67s
sh.helm.release.v1.demo.v11 helm.sh/release.v1 1 63s
sh.helm.release.v1.demo.v2 helm.sh/release.v1 1 2m58s
sh.helm.release.v1.demo.v3 helm.sh/release.v1 1 2m42s
sh.helm.release.v1.demo.v4 helm.sh/release.v1 1 2m39s
sh.helm.release.v1.demo.v5 helm.sh/release.v1 1 2m33s
sh.helm.release.v1.demo.v6 helm.sh/release.v1 1 2m29s
sh.helm.release.v1.demo.v7 helm.sh/release.v1 1 2m28s
sh.helm.release.v1.demo.v8 helm.sh/release.v1 1 109s
sh.helm.release.v1.demo.v9 helm.sh/release.v1 1 69s
$ kubectl get secret sh.helm.release.v1.demo.v1 -o jsonpath='{.type}{" "}{.metadata.labels}{"\n"}'
Error from server (NotFound): secrets "sh.helm.release.v1.demo.v1" not found
(exit status 1)
This is why Helm needs no server, why a release belongs to a namespace, and why helm history and helm rollback work from any machine with the same kubeconfig. It also means three things to know.
First, the Secret holds the whole release: the chart (templates included), the values and the rendered manifest, compressed and encoded twice. Anyone who can read Secrets in that namespace can read every value you passed, including passwords given with --set. Decode one (the data is base64 inside a base64 field, then gzip) to see for yourself:
$ helm install sec shop --set message=topsecret-value --wait --timeout 90s | grep STATUS
STATUS: deployed
$ kubectl get secret sh.helm.release.v1.sec.v1 -o jsonpath='{.data.release}' | base64 -d | base64 -d | gunzip | jq -c '.config'
{"message":"topsecret-value"}
$ helm uninstall sec | head -1
release "sec" uninstalled
The value is in clear text. Kubernetes can encrypt Secrets at rest, which protects the disk and not the people who may read them through the API. The rule that follows: do not pass a secret to a chart as a value. Have the chart take the NAME of a Secret that already exists (a value like existingSecret, which most well-made charts offer), and create that Secret some other way. Tools for that (external secrets operators, sealed secrets, SOPS) were not run.
Second, the number of revisions kept is limited (10 by default, --history-max or HELM_MAX_HISTORY to change), because each one is an object in etcd, and a chart with large templates and many upgrades could fill it. The oldest ones are deleted as new ones are made.
Third, the Secret is not the only place. Helm can store releases in ConfigMaps instead, with the HELM_DRIVER environment variable. The driver must be the same for every command that touches the release, or the other commands will not find it:
$ HELM_DRIVER=configmap helm install cmdrv shop --wait --timeout 90s | grep STATUS
STATUS: deployed
$ kubectl get configmap -l owner=helm
NAME DATA AGE
sh.helm.release.v1.cmdrv.v1 1 1s
$ helm list | grep cmdrv || echo "not listed"
not listed
$ HELM_DRIVER=configmap helm list | grep cmdrv | cut -c1-60
cmdrv default 1 2026-10-20 00:35:05.222205208 +0000
$ HELM_DRIVER=configmap helm uninstall cmdrv | head -1
release "cmdrv" uninstalled
The same release was invisible to plain helm list. If a release "disappears", check that the driver and the namespace are the same as when it was installed, before you suspect anything worse. Do not delete these Secrets by hand: the history goes with them, and a release without its record is back to being unmanaged objects that Helm will refuse to touch (the next section).
Adopt objects that already exist
Helm refuses to overwrite an object it did not create. Before it creates anything, it checks every object in the chart: if one already exists, it must carry the label app.kubernetes.io/managed-by=Helm and the annotations naming this release and namespace. Anything else is treated as someone else's. Create a ConfigMap by hand that has the same name the chart would use, and try to install:
$ kubectl create configmap clash-shop --from-literal=a=b
configmap/clash-shop created
$ helm install clash shop 2>&1 | head -4
Error: INSTALLATION FAILED: unable to continue with install: ConfigMap "clash-shop" in namespace "default" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "clash"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "default"
The message lists exactly what is missing. Helm 4 has a flag for taking over such an object on purpose, --take-ownership, which skips the check and adds the marks. The object is then part of the release, and an uninstall will delete it:
$ helm install clash shop --take-ownership --wait --timeout 90s | grep -E 'STATUS|REVISION'
STATUS: deployed
REVISION: 1
$ kubectl get configmap clash-shop -o jsonpath='{.metadata.annotations}{"\n"}'
{"meta.helm.sh/release-name":"clash","meta.helm.sh/release-namespace":"default"}
$ kubectl get configmap clash-shop -o jsonpath='{.data}{"\n"}'
{"a":"b","index.html":"\u003chtml\u003e\u003cbody\u003e\u003ch1\u003eWelcome to the shop\u003c/h1\u003e\u003cp\u003ethanks for shopping\u003c/p\u003e\u003c/body\u003e\u003c/html\u003e\n"}
The ConfigMap now belongs to the release, and its data is a merge: the key a that was created by hand is still there next to the chart's index.html, because Helm 4 applies objects with server-side apply by default and only changes the fields it manages. If a keep-everything merge is not what you want, remove the object and let Helm create it. The other way to adopt an object, without the flag, is to put the three marks on it yourself with kubectl label and kubectl annotate. That is how objects are moved from one release to another, and was not run.
A second refusal has the same cause in a different place: a release name that is already in use, in the same namespace and under the same driver:
$ helm install clash shop 2>&1 | head -2
level=ERROR msg="release name check failed" error="cannot reuse a name that is still in use"
Error: INSTALLATION FAILED: release name check failed: cannot reuse a name that is still in use
$ helm uninstall clash | head -1
release "clash" uninstalled
The answer is helm upgrade for an existing release, or helm upgrade --install, which does either, depending on whether the release exists (the next section uses it). Names are per namespace, so the same name may be used in another one.
Several environments from one chart
One chart, one values file per environment, one namespace each. The values files hold only what differs:
$ cat values-dev.yaml
replicaCount: 1
message: "Shop (development)"
$ cat values-prod.yaml
replicaCount: 3
message: "Welcome to the shop"
image:
tag: "1.28"
resources:
requests:
cpu: 50m
memory: 32Mi
limits:
memory: 128Mi
ingress:
enabled: true
host: shop.example.com
The command to use in a pipeline is helm upgrade --install, because it does the right thing whether the release exists or not, so the same line works for the first deploy and for the thousandth. Add --create-namespace for the first one, a wait with a timeout, and --rollback-on-failure so that a bad release does not stay behind. The same release name can live in both namespaces:
$ helm upgrade --install web shop -n dev --create-namespace -f values-dev.yaml --wait --rollback-on-failure --timeout 90s | grep -E 'STATUS|NAMESPACE|REVISION'
NAMESPACE: dev
STATUS: deployed
REVISION: 1
$ helm upgrade --install web shop -n prod --create-namespace -f values-prod.yaml --wait --rollback-on-failure --timeout 90s | grep -E 'STATUS|NAMESPACE|REVISION'
NAMESPACE: prod
STATUS: deployed
REVISION: 1
$ helm list -A | grep -E 'NAMESPACE|^web'
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
web dev 1 2026-10-20 00:35:12.085651832 +0000 UTC deployed shop-0.2.0 1.27
web prod 1 2026-10-20 00:35:13.61499678 +0000 UTC deployed shop-0.2.0 1.27
$ kubectl get deployments,ingress -n prod
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/web-shop 3/3 3 3 2s
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/web-shop traefik shop.example.com 172.25.0.2,172.25.0.3 80 2s
$ kubectl get deployments -n dev
NAME READY UP-TO-DATE AVAILABLE AGE
web-shop 1/1 1 1 4s
Run the same line again with no change. It is safe, and it makes a new revision anyway:
$ helm upgrade --install web shop -n prod -f values-prod.yaml --wait --timeout 90s | grep -E 'REVISION'
REVISION: 2
$ helm history web -n prod
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Tue Oct 20 00:35:13 2026 superseded shop-0.2.0 1.27 Install complete
2 Tue Oct 20 00:35:16 2026 deployed shop-0.2.0 1.27 Upgrade complete
An upgrade with nothing changed still increases the revision number (the objects are applied, and Kubernetes sees no difference, so nothing restarts), which is why a long-lived release in a busy pipeline reaches its history limit quickly. Look at the differences between environments before you ship them, by rendering both and comparing:
$ diff <(helm template web shop -f values-dev.yaml) <(helm template web shop -f values-prod.yaml) | grep -E '^[<>] +(replicas|image:|message|host)'
< replicas: 1
> replicas: 3
< image: "nginx:1.27"
> image: "nginx:1.28"
Remove the two environments:
$ helm uninstall web -n dev | head -1; helm uninstall web -n prod | head -1; kubectl delete namespace dev prod | head -2
release "web" uninstalled
release "web" uninstalled
namespace "dev" deleted
namespace "prod" deleted
Two other tools sit on top of this pattern and were not run: Helmfile, which describes many releases and their values in one file, and the GitOps controllers (Argo CD and Flux), which watch a git repository and run the equivalent of these commands for you. If you use one of them, the same rules about values files in git, pinned versions and diffs apply, and the controller takes over the pipeline.
Helm 4 compared with Helm 3
Charts written for Helm 3 (apiVersion: v2 in Chart.yaml, which every chart in this guide uses) work in Helm 4, and the day-to-day commands are the same. What changed is mostly in how a release is applied and waited for, and in a few flags. Everything in this list was seen in the lab, either as behaviour or in the help text of Helm 4.3.1, as marked.
--atomic is now --rollback-on-failure. The old flag still works and prints a deprecation warning (run below). The new one turns on waiting for you.
--force is now --force-replace (from the help text).
--dry-run takes a value: none, client or server. Server sends the objects to the API server for validation (run in the section on looking before you apply).
--wait takes a strategy: watcher, hookOnly or legacy. With no --wait the default is hookOnly, which is why an install can say "deployed" while pods are failing (run earlier).
Objects are applied with server-side apply by default (the help text says --server-side defaults to true). The take-ownership merge above is a result of it. rollback has --server-side auto, which follows the way the release was installed. Behaviour with conflicts (--force-conflicts) was not tested.
Plug-ins are checked for a signature. An unsigned plug-in needs --verify=false to install (run above). Helm 4 also has plug-in types for post-renderers, and --post-renderer takes a plug-in name now instead of a program path (help text only, not run).
--take-ownership on install and upgrade (run in the last section).
helm upgrade has --reset-then-reuse-values (run in the upgrade section).
$ helm install old shop --atomic --dry-run=client 2>&1 | head -2
Flag --atomic has been deprecated, use --rollback-on-failure instead
NAME: old
$ helm install --help | grep -E -- '--rollback-on-failure|--force-replace|--server-side|--wait ' | cut -c1-110
--force-replace force resource updates by replacement
--rollback-on-failure if set, Helm will rollback (uninstall) the installation upo
--server-side object updates run in the server instead of the client (def
--wait WaitStrategy[=watcher] wait until resources are ready (up to --timeout). Use '--wa
--wait-for-jobs if set and --wait enabled, will wait until all Jobs have be
Check the help of the version you use before copying flags from older material, and pin the version of Helm in CI like any other tool, because a flag that was deprecated warns for a while and then goes. Release records written by Helm 3 and read by Helm 4 were not tested in the lab, which only ever ran Helm 4.3.1.
Read before you install: safety and supply chain
A chart is code that makes objects in your cluster with your permissions, and chart repositories are mostly other people's code. Four habits cover most of the risk.
Look at what a chart will make, before the first install. Download it and render it, then list the kinds, the images and anything that widens privileges:
$ helm pull examples/hello-world --untar --destination /tmp/inspect && helm template hw /tmp/inspect/hello-world | grep -E '^kind:|image:'
kind: ServiceAccount
kind: Service
kind: Deployment
image: "nginx:1.16.0"
$ helm template hw /tmp/inspect/hello-world | grep -ciE 'privileged|hostPath|hostNetwork|ClusterRole|hostPID'; true
0
A count of 0 means none of those words appear in this chart. A chart that adds ClusterRoles, mounts the host's files or runs privileged is not wrong by itself (a monitoring agent may need it), but you should know it before, not after. Also read the CRDs and the hooks, which helm template hides unless you add --include-crds, and which can run arbitrary Jobs.
Pin what you install. Give --version for every chart, and in the chart's own values pin the image tag, or even better its digest, because a tag can be moved. A chart installed without a version takes the newest in the index, and the same command gives different results on different days. When you keep your own copy of a chart in git, helm pull it by version and commit the folder.
Check provenance where it exists. Chart signing (helm package --sign, helm install --verify) lets you check that a chart was signed by a key you trust. It needs GPG keys, and was not run in the lab, and most public charts are not signed, so in practice the trust is in the repository and the people behind it. Prefer charts from the project that makes the software, read the changelog before moving to a new version, and use helm diff for the upgrade.
Keep secrets out of values. The release record contains every value in clear text, as shown above, and --set puts them in the shell history and the CI logs as well. Use the name of a pre-created Secret instead. And remember that a plug-in is a program that runs on your machine with your credentials: install them from sources you trust, and be wary of --verify=false on anything you have not read.
Helm's permissions are those of the kubeconfig it uses. For a pipeline, make a ServiceAccount that may only work in the namespace it deploys to, with a Role that lists the verbs and kinds the chart needs, and not cluster-admin. That is the main limit on how much damage a bad chart or a leaked CI token can do. (RBAC is explained in the Kubernetes guide; a Helm-specific Role was not run.)
Troubleshooting by symptom
Most Helm failures come in a small number of shapes. These are the real messages the lab produced, the cause, and the fix. First the ones that need no cluster:
$ helm install x examples/nope 2>&1 | head -2
Error: INSTALLATION FAILED: chart "nope" matching not found in examples index. (try 'helm repo update'): no chart name found
$ helm status nothere 2>&1 | head -2
Error: release: not found
$ helm upgrade nothere shop 2>&1 | head -2
Error: UPGRADE FAILED: "nothere" has no deployed releases
$ KUBECONFIG=/tmp/none helm list 2>&1 | head -2
Error: kubernetes cluster unreachable: Get "http://localhost:8080/version": dial tcp [::1]:8080: connect: connection refused
$ helm template x shop --set replicaCount=3 -f nofile.yaml 2>&1 | head -2
Error: open nofile.yaml: no such file or directory
chart "nope" matching not found in examples index the repository name or the chart name is wrong, or you forgot helm repo update. Run helm search repo to see what you have.
release: not found wrong release name, or the wrong namespace (-n). helm list -A finds it. Remember HELM_DRIVER.
has no deployed releases an upgrade of something that was never installed, or a first install that failed. Use helm upgrade --install, or uninstall the failed one.
kubernetes cluster unreachable the kubeconfig is wrong or missing (here it points at localhost:8080, the default when no file is found). Helm reads the same file as kubectl, so test kubectl get nodes first.
open nofile.yaml: no such file or directory a -f path that does not exist, or the working directory is not what you think.
The ones met in the sections above, by what you see:
cannot reuse a name that is still in use
a release by that name exists (or a failed one still holds it). helm list -a, then helm upgrade --install, or helm uninstall.
another operation (install/upgrade/rollback) is in progress
a pending-upgrade or pending-install record, because an earlier helm died. Roll back to the last deployed revision (the upgrade section).
UPGRADE FAILED: ... resource Deployment/default/NAME not ready ... context deadline exceeded
the new pods did not become ready within --timeout. Look with kubectl describe pod and kubectl logs; the usual causes are a wrong image tag, a failing probe, no room. Add --rollback-on-failure so that it is put back by itself.
exists and cannot be imported into the current release: invalid ownership metadata
an object with that name exists and is not Helm's. Adopt it (--take-ownership) or delete it. A leftover from a failed install of the same chart is the usual cause.
YAML parse error on ...
the rendered text is not YAML. Use helm template --debug to see it, and check nindent and the whitespace dashes.
nil pointer evaluating interface {}.NAME
a value in the path is missing. Check the values file and the --set path, and remember --reuse-values hides new defaults.
values don't meet the specifications of the schema
a value has the wrong type or range. The message names the path.
found in Chart.yaml, but missing in charts/ directory
run helm dependency update (or build).
http: server gave HTTP response to HTTPS client
the registry speaks plain HTTP. Fix the registry to use HTTPS. For a lab only, --plain-http.
For anything else, in this order: helm status RELEASE (is it failed, pending, deployed?), helm get manifest RELEASE (what was sent), kubectl get events and kubectl describe on the object that is wrong, and helm get values RELEASE -a (the values it really used). If the release is stuck and you must start again, helm uninstall RELEASE removes everything the release owns, and a leftover object that Helm does not own has to be deleted by hand.
Safe habits
keep every value in a file in git, one per environment no one has to remember which --set flags were used last time
give --version for every chart, and pin the image tags the same command means the same thing next month
run helm lint and helm template in CI, and a server dry run before a real apply mistakes are caught before the cluster sees them
read helm diff upgrade before every upgrade that matters you see what really changes, including the checksum that restarts pods
use helm upgrade --install --wait --timeout N --rollback-on-failure one command for both cases, and a bad release does not stay behind
set the timeout to the real start-up time plus a margin too short rolls back healthy releases, too long wastes a pipeline
use --reset-then-reuse-values or a full values file, never --reuse-values new chart defaults are not silently ignored
bump version in Chart.yaml for every chart change, and never re-publish a version rollbacks, caches and audits can trust the number
put the config checksum in the pod template annotations a changed ConfigMap or Secret restarts the pods
write hooks that are safe to run twice, and set backoffLimit they run on every upgrade
make a plan for CRDs before the first upgrade Helm does not upgrade or delete the ones in crds/
keep secrets out of values; refer to a Secret by name the release record holds every value in clear text
run one helm operation at a time per release a killed run leaves pending-upgrade
give pipelines a namespaced service account, not cluster-admin it limits what a bad chart can do
read a chart (helm template, helm pull --untar) before the first install from a source you do not know it runs with your permissions
What was not run in the lab
The lab was two Ubuntu 24.04 containers running k3s v1.35.9, with Helm 4.3.1. These were not run, so treat what this guide says about them as general knowledge and not as tested:
Charts from the big public repositories (Bitnami, ingress-nginx, kube-prometheus-stack and so on). Only the Helm project's example repository, the lab's own repository and the lab's own registry were used.
OCI registries with HTTPS and a login (helm registry login, GitHub Container Registry, cloud registries), and the lab registry was reached over plain HTTP.
Signed charts and plug-ins: helm package --sign, helm install --verify, plug-in signature checks. The diff plug-in was installed with --verify=false.
The lookup function against a live cluster, post-renderers (and Kustomize as one), library charts, subchart alias, tags and import-values, Helm test hooks that fail.
The legacy wait strategy, --force-conflicts, --force-replace, server-side apply conflicts, and --server-side=false.
Release storage in SQL (HELM_DRIVER=sql), and release records written by Helm 3 and read by Helm 4.
Helmfile, Argo CD and Flux, and GitOps pipelines in general.
Secret management for values (SOPS, sealed secrets, external secrets), and a Helm-specific RBAC Role for a pipeline.
Charts with StatefulSets and persistent volumes, upgrades of databases, and anything that needs a cloud load balancer.
Helm on Windows and macOS (only the Linux binary was used), and Helm versions other than 4.3.1.
Cheat sheet
Install and set up
helm version versions of the client
helm env where Helm keeps its files, and the defaults it uses
source <(helm completion bash) tab completion (zsh and fish also exist)
helm plugin install URL --verify=false install an unsigned plug-in (helm plugin list shows them)
Repositories and charts
helm repo add NAME URL add a repository
helm repo update refresh your copy of every index
helm repo list the repositories you added
helm repo remove NAME
helm search repo WORD search the repositories you added (--versions lists every version)
helm search hub WORD search Artifact Hub (needs internet)
helm show chart CHART the chart's metadata
helm show values CHART every setting and its default
helm pull CHART --untar download and unpack, to read it
helm template NAME CHART render the YAML without a cluster
Install, change, undo
helm install NAME CHART -f values.yaml --wait --timeout 5m
helm upgrade --install NAME CHART -n NS --create-namespace -f values.yaml --wait --rollback-on-failure --timeout 5m
helm upgrade NAME CHART --reset-then-reuse-values --set KEY=VALUE keep the last values, take the new defaults
helm rollback NAME REVISION --wait --timeout 5m restore a revision (makes a new one)
helm uninstall NAME remove the release (--keep-history keeps its record)
helm test NAME run the chart's test pods
Look at releases
helm list -A every release, in every namespace (-a includes failed, --failed, --uninstalled)
helm status NAME state and notes
helm history NAME the revisions (-o json for scripts)
helm get values NAME only what you supplied (-a for all computed values, --revision N for an old one)
helm get manifest NAME the YAML that was applied
helm get hooks NAME / helm get notes NAME
Check before applying
helm lint CHART --strict
helm install NAME CHART --dry-run=server validate against the API server, saves nothing (--hide-secret)
helm diff upgrade NAME CHART -f values.yaml what an upgrade would change (the helm-diff plug-in)
helm template NAME CHART | kubectl diff -f - the same without a plug-in
Values
-f file.yaml a values file (repeatable; later wins)
--set a.b=c one value (quote lists: --set 'tags={a,b}')
--set-string a=1234 force text
--set-json 'a={"b":1}' a structure as JSON
--set-file a=./file the content of a file (no trailing newline!)
Write and publish a chart
helm create NAME a starter chart
helm dependency update CHART / helm dependency build CHART fetch subcharts (build follows Chart.lock)
helm package CHART make NAME-VERSION.tgz
helm repo index DIR --url URL write index.yaml for a folder of packages
helm push NAME-VERSION.tgz oci://HOST/PATH publish to an OCI registry (--plain-http for a plain-HTTP lab one)
helm install NAME oci://HOST/PATH/CHART --version V
In a template
{{ .Values.x }} {{ .Release.Name }} {{ .Chart.AppVersion }} {{ include "name" . | nindent 4 }}
{{ .Values.x | default "y" | quote }} {{ toYaml .Values.res | nindent 8 }} {{ required "msg" .Values.x }}
{{- if .Values.on }} ... {{- end }} {{- range .Values.list }} {{ . }} {{- end }} {{- with .Values.m }} ... {{- end }}
checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
Hooks, CRDs, ownership
"helm.sh/hook": pre-upgrade "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
"helm.sh/resource-policy": keep leave the object when the release is removed
crds/ installed once; never upgraded, never removed
--take-ownership adopt objects that already exist
Common problems and tips
A release says deployed and the pods are failing. Without --wait Helm does not look at the pods. Add --wait with a --timeout, and --rollback-on-failure for upgrades.
An upgrade changed a setting I did not touch. Each upgrade starts from the chart defaults plus THIS command's values. Use a full values file with -f, or --reset-then-reuse-values. Check what is set with helm get values NAME, and with -a for the computed values.
A ConfigMap or Secret changed and the pods did not restart. Kubernetes does not restart pods for a changed ConfigMap. Put a checksum of the file in the pod template annotations, as the shop chart does, or run kubectl rollout restart after the upgrade.
The upgrade says another operation is in progress. A helm process died in the middle. helm history NAME shows pending-upgrade. Roll back to the last deployed revision, and avoid running two pipelines on one release.
--reuse-values gives a nil pointer after a chart upgrade. It ignores the new chart's defaults. Use --reset-then-reuse-values.
helm template works and the install fails. Template renders on your machine only. Use helm install --dry-run=server to validate against the cluster, and read the error, which names the object.
An object exists and Helm will not touch it. It was made by hand or by another release. Delete it, or use --take-ownership to adopt it.
A CRD in my chart was not updated. Files in crds/ are installed once and never upgraded or removed. Apply CRD changes yourself, before the chart upgrade.
I deleted a release and the volume is still there. A uninstall does not remove claims made by a StatefulSet's volume template. Look with kubectl get pvc.
I cannot find a release. Wrong namespace (try helm list -A), or a different HELM_DRIVER, or it is uninstalled with --keep-history (try helm list --uninstalled).
The rollback took five minutes. The revision you rolled back to was itself broken, and a rollback with --wait waits for the whole timeout. Check helm get values NAME --revision N before you choose it.
A chart gives different results on different days. The command had no --version, or the image tag is a moving one. Pin both.
Tips. Keep the chart and the values files in git next to the application, and keep version and appVersion honest. Use helm get manifest to see what was really applied. Use the same Helm version on your machine and in CI. Read helm show values before you set anything. A small chart that you wrote and understand is often better than a large one with a hundred values you do not use. And remember that helm uninstall is quick and removes everything the release made, so check the namespace first.
