🧂 Salt Guide

Salt: states, grains, pillar and minions (and salt-ssh)
salt '*' state.apply test=True

Salt (the Salt Project) is a configuration management and remote execution tool. A server called the master keeps the configuration; agents called minions connect to it and obey. You can run a command on a thousand machines at once (salt '*' cmd.run 'uptime') or describe the state a machine should be in, in YAML files, and have Salt make it so, again and again. It can also run with no master at all, or with no agent at all (salt-ssh). This page builds a small lab and tests every one of those modes: a master and two minions, a masterless machine, and an agentless one. It shows how the pieces fit (states, grains, pillar, top file), how to preview and apply a change, and the places where Salt quietly does something other than what you meant: a fingerprint check that fails on a perfectly good key, pillar that reads back empty or masked, a grain that is invisible until it is refreshed, a stray command that registers a key on the wrong master.
Every output below ran for real, on 9 October 2026: Salt 3008.3 (with the Python 3.14.7 that comes bundled in its package) from the Salt Project's own apt repository, on Ubuntu 24.04 Docker containers. Containers are not servers: there is no systemd inside, so I started the master and the minions by hand (salt-master -d, salt-minion -d) and Salt used the init-script service provider; systemd service management was not run. Where something was not run, this page says "not run in the lab".

Docker containers on a private network, all built from one image (ubuntu:24.04 plus the Salt packages), started with docker run --init:
  master    the Salt master, reachable under the name "salt" (a network alias): that is the default name a minion looks for
  minion1   a minion
  minion2   a minion
  solo      a machine with the packages but no daemon, for masterless runs
  web3      a machine with only an SSH server and no Salt at all, for salt-ssh
$ salt-call --versions-report | head -4
Salt Version:
          Salt: 3008.3

Python Version:
        Python: 3.14.7 (main, Sep 30 2026, 03:35:11) [GCC 11.2.0]
Two lines that appear on many Salt commands in this lab and are left out of the outputs below, because they are noise: a Python "DeprecationWarning: This process is multi-threaded, use of fork() may lead to deadlocks" from the bundled Python, and "unclosed MasterEvent ... call destroy() or use as a context manager" from the command-line tools.

  master    the server. It listens on two ports, 4505 (it publishes jobs to minions) and 4506 (minions return results and fetch files)
  minion    the agent. It connects OUT to the master, so a minion needs no open port of its own: on minion1, ss showed no listener from Salt (only nginx, which the state installed, and Docker's DNS), and two established connections to the master's ports:
              ESTAB 172.21.0.3:58356  172.21.0.2:4505
              ESTAB 172.21.0.3:46322  172.21.0.2:4506
  key       each minion has a key pair; the master must accept the minion's public key before it will talk to it
  grain     a fact about a minion (operating system, host name, IP addresses, your own labels). Minions report them, and you can target by them
  pillar    data the master hands to a minion, selected per minion. This is where values and (in principle) secrets belong
  state     one declaration: "this package is installed", "this file looks like that". States are written in YAML files with the extension .sls (SaLt State)
  top file  /srv/salt/top.sls says which states apply to which minions
  module    remote execution functions (cmd.run, pkg.version, network.ip_addrs): things you RUN. State modules (pkg.installed, file.managed) describe what should BE: they are the ones used in .sls files
Like the other tools, Salt compares the machine with the declaration and changes only what differs. Three ways to run it:
  master + minions       the full model, shown first
  masterless             salt-call --local on one machine, reading the files from its own disk
  salt-ssh               the master logs in over SSH and runs Salt on the fly; nothing to install on the target except Python

Ubuntu's own repositories carry no Salt packages (apt-cache policy for salt-master, salt-minion and salt-ssh printed nothing on 24.04). The project's install guide for Debian and Ubuntu, read on 9 October 2026, says: download the repository key, download the salt.sources file, update, and install. What I ran:
$ mkdir -m 755 -p /etc/apt/keyrings
$ curl -fsSL https://packages.broadcom.com/artifactory/api/security/keypair/SaltProjectKey/public | gpg --dearmor > /etc/apt/keyrings/salt-archive-keyring.pgp
$ curl -fsSL https://github.com/saltstack/salt-install-guide/releases/latest/download/salt.sources > /etc/apt/sources.list.d/salt.sources
$ apt-get update
$ apt-cache policy salt-master
salt-master:
  Installed: (none)
  Candidate: 3008.3
$ apt-get install salt-master salt-minion salt-ssh
The file it created is in the DEB822 format and shows exactly where the packages come from:
$ cat /etc/apt/sources.list.d/salt.sources
X-Repolib-Name: Salt Project
...
Enabled: yes
Types: deb
URIs: https://packages.broadcom.com/artifactory/saltproject-deb
Signed-By: /etc/apt/keyrings/salt-archive-keyring.pgp
Suites: stable
Components: main
(the Description and website lines in the middle are left out.) The key is bound to this one repository with Signed-By, not trusted for everything. The packages are "onedir": the whole Salt, with its own Python, lives in /opt/saltstack (218 MB here), so Salt does not use the system Python. The guide recommends the LTS releases for production and gives a way to pin one with an apt preferences file (Package: salt-* / Pin: version 3008.* / Pin-Priority: 1001); the same guide's example calls 3008 an LTS release. I installed the candidate (3008.3) and did not test the pin. Without a pin, an apt upgrade can move the master and every minion to a new major version, and a master must not be older than its minions (I did not test mixed versions).
Start the daemons (in a real system: systemctl enable --now salt-master; the minion likewise, which was not run in the lab). On a minion, the master's name goes in a config snippet:
$ cat /etc/salt/minion.d/master.conf
master: salt
$ salt-master -d
$ ss -ltn | grep -E ":(4505|4506) "
LISTEN 0      1000         0.0.0.0:4505       0.0.0.0:*
LISTEN 0      1000         0.0.0.0:4506       0.0.0.0:*
$ salt-minion -d

A minion that starts for the first time sends its public key and waits. On the master:
$ salt-key -L
Accepted Keys:
Denied Keys:
Unaccepted Keys:
minion1.lab
minion2.lab
Rejected Keys:
Until a key is accepted the minion is invisible to commands:
$ salt '*' test.ping; echo exit=$?
ERROR: No return received
No minions matched the target. No command was sent, no jid was assigned.
exit=2
The step that matters is the one before accepting: is this key really the one from the machine you think it is? The documented check is to compare the fingerprint the master shows with the one the minion shows. In this version the two commands DO NOT AGREE for the same key:
$ salt-key -f minion1.lab                                 (on the master)
Unaccepted Keys:
minion1.lab:  0c:bd:dd:bb:74:af:0e:44:e1:1a:a1:ac:71:91:9a:12:5e:f3:7c:fc:72:17:c1:b0:4c:47:80:eb:e1:da:12:68
$ salt-call --local key.finger                            (on minion1)
local:
    5d:11:cc:df:22:15:fa:9b:14:d9:d8:a7:c9:61:a1:bf:98:2d:67:55:55:79:cd:e3:74:40:15:09:0b:76:7f:72
It is not a different key. I copied the key file out of the minion and compared it with the master's copy: the same text, except that the minion's file ends with a newline and the master's does not. The difference is in what the two commands hash. The function that computes the fingerprint (salt.utils.crypt.pem_finger) hashes only the base64 lines BETWEEN the "BEGIN" and "END" lines when it is given a file path (that is what key.finger does), and hashes EVERYTHING when it is given the text of the key (that is what salt-key does, the key coming from the master's key store). I proved it on the master by reproducing both numbers:
sha256 of whole file, stripped   0c:bd:dd:bb:74:af:0e:44:e1:1a:a1:ac:71:91:9a:12     <- what salt-key -f printed
pem_finger(path=...)             5d:11:cc:df:22:15:fa:9b:14:d9:d8:a7:c9:61:a1:bf     <- what key.finger printed
So the comparison the documentation describes cannot succeed here, and a careful administrator would conclude that the key had been swapped. A check that compares like with like: hash the whole key text, without its final newline, on the minion, and compare it with the master's number with the colons removed:
$ printf '%s' "$(cat /etc/salt/pki/minion/minion.pub)" | sha256sum | cut -c1-64          (on the minion)
$ salt-key -f minion1.lab | tail -1 | awk '{print $2}' | tr -d ':'                          (on the master)
master says:  0cbdddbb74af0e44e11aa1ac71919a125ef37cfc7217c1b04c4780ebe1da1268
minion1 computes: 0cbdddbb74af0e44e11aa1ac71919a125ef37cfc7217c1b04c4780ebe1da1268
master says:  a76e728c88a371be1fa8ce272adfda8d5c3e28131899b9c22a97b9d5c8906de6
minion2 computes: a76e728c88a371be1fa8ce272adfda8d5c3e28131899b9c22a97b9d5c8906de6
Both minions match. (The same mismatch very likely applies to the master's own key, salt-key -F master against key.finger_master on the minion; I did not test that one.) Then accept, one by one (-a NAME) or all at once (-A), with -y to skip the question:
$ salt-key -a minion1.lab -y
Key for minion minion1.lab accepted.
$ salt-key -A -y
Key for minion minion2.lab accepted.
$ salt-key -L
Accepted Keys:
minion1.lab
minion2.lab
Denied Keys:
Unaccepted Keys:
Rejected Keys:
$ salt '*' test.ping
minion1.lab:
    True
minion2.lab:
    True
The master config has an option to accept every key automatically (auto_accept). It was not used, and should not be on a network you do not control: anybody who can reach port 4506 would become one of your minions, and minions are told what to run.

The first word after salt is the target: which minions. The second is the function to run.
$ salt '*' test.version
minion2.lab:
    3008.3
minion1.lab:
    3008.3
$ salt '*' grains.item os osrelease kernel id
minion1.lab:
    ----------
    id:
        minion1.lab
    kernel:
        Linux
    os:
        Ubuntu
    osrelease:
        24.04
(shown for one of the two minions.) Target types, all run:
$ salt 'minion1*' test.ping                              a glob on the minion ID
$ salt -L 'minion1.lab,minion2.lab' test.ping            an explicit list
$ salt -E 'minion[12]\.lab' test.ping                    a regular expression
$ salt -G 'os:Ubuntu' test.ping                          a grain (both answered)
$ salt -C 'G@kernel:Linux and not minion2*' test.ping    a compound: grains and globs with and/or/not
minion1.lab:
    True
(that is the output of the last one.) A target that matches nothing is not an error from the minion's point of view, but the command exits with 2:
$ salt -G 'os:Rocky' test.ping; echo exit=$?
ERROR: No return received
No minions matched the target. No command was sent, no jid was assigned.
exit=2
Run commands and read answers (modules, not states: these do things, they do not describe them):
$ salt '*' cmd.run 'hostname; uptime -p'
minion2.lab:
    minion2.lab
    up 1 week, 4 hours, 24 minutes
$ salt '*' pkg.version curl
minion1.lab:
    8.5.0-2ubuntu10.15
minion2.lab:
    8.5.0-2ubuntu10.15
$ salt '*' network.ip_addrs
minion1.lab:
    - 172.21.0.3
minion2.lab:
    - 172.21.0.4
(the uptime is the Docker host's, because a container shares its kernel.) --batch-size 1 runs one minion at a time, which is how you avoid restarting every web server at once:
$ salt '*' test.ping --batch-size 1
Executing run on ['minion2.lab']
Executing run on ['minion1.lab']
Every command is a job with an ID (a jid), kept by the master, and you can look it up later:
$ salt-run jobs.list_jobs search_function='cmd.run' | head -12
20261009141650725441:
    ----------
    Arguments:
        - hostname; uptime -p
    Function:
        cmd.run
    StartTime:
        2026, Oct 09 14:16:50.725441
    Target:
        *
    Target-type:
        glob
$ salt-run manage.up
- minion1.lab
- minion2.lab

I stopped the salt-minion process on minion2 (by its PID) and asked everybody:
$ salt '*' test.ping; echo exit=$?
ERROR: Minions returned with non-zero exit code
minion1.lab:
    True
minion2.lab:
    Minion did not return. [Not connected]
exit=1
$ salt-run manage.down
- minion2.lab
$ salt-run manage.up
- minion1.lab
$ salt '*' test.ping -t 3; echo exit=$?
ERROR: Minions returned with non-zero exit code
minion1.lab:
    True
minion2.lab:
    Minion did not return. [Not connected]
exit=1
(-t 3 shortens the wait to three seconds.) "Not connected" means the master knows the minion's connection is gone; a minion that is connected but slow gives a different message, "Minion did not return. [No response]" (not produced in the lab). Mind the timing: my first test came a moment too early, while the minion's process had not finished stopping, and the ping still got True. A salt-minion that is stopped at the master's door looks fine until its connection times out.

Grains come from the minion itself. Besides the built-in ones you can add your own, in /etc/salt/grains on the minion. A trap: written by hand, a new grain is invisible until the minion reloads its grains:
$ salt 'minion1*' cmd.run 'printf "role: web\ntier: frontend\n" > /etc/salt/grains'
$ salt -G 'role:web' test.ping; echo exit=$?
ERROR: No return received
No minions matched the target. No command was sent, no jid was assigned.
exit=2
$ salt 'minion1*' saltutil.refresh_grains
minion1.lab:
    True
$ salt -G 'role:web' test.ping
minion1.lab:
    True
$ salt 'minion1*' grains.item role tier
minion1.lab:
    ----------
    role:
        web
    tier:
        frontend
The function grains.setval writes the file AND refreshes in one go:
$ salt 'minion2*' grains.setval role web
minion2.lab:
    ----------
    role:
        web
$ salt -G 'role:web' test.ping
minion2.lab:
    True
minion1.lab:
    True
Use grains for what the machine IS (and for targeting); use pillar for what it should be GIVEN.

Pillar is defined on the master, in /srv/pillar by default (the master config says so: pillar_roots is /srv/pillar and /srv/spm/pillar), with its own top file that says who gets what:
$ cat /srv/pillar/top.sls
base:
  '*':
    - webserver
$ cat /srv/pillar/webserver.sls
webserver:
  port: 80
  sites:
    shop: Welcome to the shop
    blog: Read the blog
Three things I saw that will otherwise cost you time.
First, a minion keeps the pillar it loaded earlier, and reading it can say "nothing" while states already use the new data. I had just created the files, and:
$ salt '*' pillar.get webserver:sites
minion2.lab:
minion1.lab:
$ salt 'minion1*' saltutil.refresh_pillar
minion1.lab:
    True
Empty, although state.apply (which compiles the pillar fresh) had used the sites a moment before. After saltutil.refresh_pillar the value is there.
Second, in this version (3008) the pillar is MASKED in the output of the command line:
$ salt 'minion1*' pillar.get webserver:port
minion1.lab:
    **********
$ salt 'minion1*' pillar.get webserver:port unmask=True
minion1.lab:
    80
$ salt 'minion1*' pillar.items
minion1.lab:
    ----------
    webserver:
        ----------
        port:
            **********
        sites:
            ----------
The source of the module marks the unmask option as versionadded 3008.0. The data itself is not hidden from states and templates; only what the commands print is. It is a sensible protection for a terminal that someone is watching, and it surprises you the first time you debug.
Third, data can be overridden per minion, and merges: I added a second pillar file for minion2 with only the port, and listed it in the top file:
$ cat /srv/pillar/minion2.sls
webserver:
  port: 8081
$ cat /srv/pillar/top.sls
base:
  '*':
    - webserver
  'minion2.lab':
    - minion2
$ salt '*' pillar.get webserver:port unmask=True
minion2.lab:
    8081
minion1.lab:
    8080
$ salt 'minion2*' pillar.get webserver:sites unmask=True
minion2.lab:
    ----------
    blog:
        Read the blog
    shop:
        Welcome to the shop
(minion1 shows 8080 because the shared file was changed to that value in the next section.) The port is overridden for minion2, and the list of sites from the shared file is still there: the dictionaries were merged, not replaced.

A state file (.sls) is YAML, with Jinja templating on top (the Jinja runs first, then the YAML is read). The top file ties minions to state files:
$ cat /srv/salt/top.sls
base:
  'minion*':
    - webserver
$ cat /srv/salt/webserver/init.sls
{% set port = salt['pillar.get']('webserver:port', 80) %}
{% set sites = salt['pillar.get']('webserver:sites', {}) %}

nginx:
  pkg.installed: []
  service.running:
    - enable: True
    - reload: True
    - require:
      - pkg: nginx

/etc/nginx/sites-enabled/default:
  file.absent:
    - require:
      - pkg: nginx
    - watch_in:
      - service: nginx

{% for site, message in sites.items() %}
/etc/nginx/conf.d/{{ site }}.conf:
  file.managed:
    - source: salt://webserver/files/site.conf.jinja
    - template: jinja
    - mode: '0644'
    - context:
        site: {{ site }}
        port: {{ port }}
    - require:
      - pkg: nginx
    - watch_in:
      - service: nginx

/var/www/{{ site }}/index.html:
  file.managed:
    - makedirs: True
    - contents: |
        <h1>{{ message }}</h1>
        <p>Site {{ site }} on {{ grains['host'] }}, {{ grains['os'] }} {{ grains['osrelease'] }}.</p>
{% endfor %}
$ cat /srv/salt/webserver/files/site.conf.jinja
# Managed by Salt (webserver state)
server {
    listen {{ port }};
    server_name {{ site }}.example.org;
    root /var/www/{{ site }};
}
Read it from the top. The two lines with {% set %} read the pillar (with defaults). A state declaration is: an ID (nginx, a path), a function (pkg.installed, file.managed) and its arguments as a YAML list. When the ID is a path, it is also the name of the file. The loop makes one config file and one page for every site in the pillar, so the data lives in pillar and the logic in the state. salt://webserver/files/... means "from the master's file server, under /srv/salt" (the default file_roots is /srv/salt and /srv/spm/salt).
What Salt will apply, and what the compiled state looks like:
$ salt 'minion1*' state.show_top
minion1.lab:
    ----------
    base:
        - webserver
$ salt 'minion1*' state.show_sls webserver | head -14
minion1.lab:
    ----------
    /etc/nginx/conf.d/blog.conf:
        ----------
        __env__:
            base
        __sls__:
            webserver
        file:
            |_
              ----------
              source:
                  salt://webserver/files/site.conf.jinja
(that is the compiled form of one state, after the Jinja loop: one such entry per site, with the template context (port 80, site blog) filled in further down.) Both of these are functions you run against a minion. salt-run state.show_top does not exist: it answers "'state.show_top' is not available."

test=True is the preview: it says what WOULD change and changes nothing.
$ salt 'minion1*' state.apply test=True | tail -22
          ID: /var/www/shop/index.html
    Function: file.managed
      Result: None
     Comment: The file /var/www/shop/index.html is set to be changed
              Note: No changes made, actual changes may
              be different due to other states.
     Started: 14:19:13.738386
    Duration: 1.168 ms
     Changes:
              ----------
              newfile:
                  /var/www/shop/index.html
...
Summary for minion1.lab
------------
Succeeded: 7 (unchanged=6, changed=5)
Failed:    0
------------
Total states run:     7
Total run time:   3.260 s
"Result: None" is Salt's word for "would change". (The note is honest: on a machine where nginx is not installed yet, the preview cannot know what the package would bring.) The real run:
$ salt 'minion1*' state.apply
minion1.lab:
----------
          ID: nginx
    Function: pkg.installed
      Result: True
     Comment: The following packages were installed/updated: nginx
     Started: 14:19:53.477883
    Duration: 6987.297 ms
     Changes:
              ----------
              nginx:
                  ----------
                  new:
                      1.24.0-2ubuntu7.18
                  old:
              nginx-common:
                  ----------
                  new:
                      1.24.0-2ubuntu7.18
                  old:
----------
          ID: /etc/nginx/sites-enabled/default
    Function: file.absent
      Result: True
     Comment: Removed file /etc/nginx/sites-enabled/default
     Started: 14:20:00.468262
    Duration: 3.506 ms
     Changes:
              ----------
              removed:
                  /etc/nginx/sites-enabled/default
...
Summary for minion1.lab
------------
Succeeded: 7 (changed=7)
Failed:    0
------------
Total states run:     7
Total run time:   7.084 s
The full output is long; --state-output=terse prints one line per state. Applied again, every state is "Clean" and the whole run takes about 70 milliseconds:
$ salt 'minion1*' state.apply --state-output=terse | tail -14
  Name: /etc/nginx/sites-enabled/default - Function: file.absent - Result: Clean - Started: 14:20:03.163180 - Duration: 0.287 ms
  Name: /etc/nginx/conf.d/shop.conf - Function: file.managed - Result: Clean - Started: 14:20:03.163564 - Duration: 19.611 ms
  Name: /etc/nginx/conf.d/blog.conf - Function: file.managed - Result: Clean - Started: 14:20:03.183343 - Duration: 12.423 ms
  Name: nginx - Function: service.running - Result: Clean - Started: 14:20:03.195927 - Duration: 18.64 ms
  Name: /var/www/shop/index.html - Function: file.managed - Result: Clean - Started: 14:20:03.214735 - Duration: 2.592 ms
  Name: /var/www/blog/index.html - Function: file.managed - Result: Clean - Started: 14:20:03.217459 - Duration: 1.947 ms
Summary for minion1.lab
------------
Succeeded: 7
Failed:    0
------------
Total states run:     7
Total run time:   72.877 ms
$ salt 'minion1*' cmd.run 'for s in shop blog; do curl -s -H "Host: $s.example.org" localhost/; done'
minion1.lab:
    <h1>Welcome to the shop</h1>
    <p>Site shop on minion1, Ubuntu 24.04.</p>
    <h1>Read the blog</h1>
    <p>Site blog on minion1, Ubuntu 24.04.</p>
state.apply with no argument applies whatever the top file gives this minion. With a name (state.apply webserver) it applies that state file and ignores the top file. If the top file matches nobody, the answer is a failure:
$ salt 'minion1*' state.apply
minion1.lab:
----------
          ID: states
    Function: no.None
      Result: False
     Comment: No Top file or master_tops data matches found. Please see master log for details.
(I got this by changing the top file to match 'web*'.)

A requisite is a rule between two states:
  require / require_in     this one needs that one first
  watch / watch_in         like require, and when that one CHANGES, this one reacts (a service reloads or restarts)
  onchanges, onfail        run only when something changed, or failed (not run in the lab)
Without a requisite, Salt runs the states in the order they are written in the file. With watch_in on the config files, the service had to wait for the files: in the second apply above the service line comes AFTER the two conf files and BEFORE the two index pages, although the service is written second in the file. Here is what the watch does when the data changes. I set port 8080 in the pillar on the master and applied; the nginx worker processes, counted before and after, are the objective check that the reload really happened:
$ salt 'minion1*' cmd.run "pgrep -f 'nginx: worker' | sort -n | tr '\n' ' '"
minion1.lab:
    1284 1285 1286 1287 1288 1289 1291 1292 1530
$ salt 'minion1*' state.apply | grep -E 'Function: service|Comment: Service|Succeeded'
    Function: service.running
     Comment: Service reloaded
Succeeded: 7 (changed=3)
$ salt 'minion1*' cmd.run "pgrep -f 'nginx: worker' | sort -n | tr '\n' ' '"
minion1.lab:
    1628 1629 1630 1631 1632 1633 1634 1635 1640
$ salt 'minion1*' cmd.run 'curl -s -o /dev/null -w "8080: %{http_code}\n" -H "Host: shop.example.org" localhost:8080/; curl -s -o /dev/null -w "80: %{http_code}\n" -H "Host: shop.example.org" localhost:80/'
ERROR: Minions returned with non-zero exit code
minion1.lab:
    8080: 200
    80: 000
(the last number of each list is the shell that ran pgrep itself. The error line is only curl returning non-zero for the port that is now closed.) A different set of worker IDs, port 8080 open and port 80 closed: the reload with reload: True did what it says. Three states changed: the two conf files and the service.

When one state fails, the ones that depend on it fail ("requisite failed"), and the ones that do not depend on it still run:
$ cat err/fail.sls
boom:
  cmd.run:
    - name: /bin/false

after-boom:
  file.managed:
    - name: /tmp/after-boom
    - contents: x
    - require:
      - cmd: boom

independent:
  file.managed:
    - name: /tmp/independent
    - contents: x
$ salt 'minion1*' state.apply err.fail --state-output=terse; echo exit=$?
ERROR: Minions returned with non-zero exit code
minion1.lab:
  Name: /bin/false - Function: cmd.run - Result: Failed - Started: 14:21:07.485631 - Duration: 6.651 ms
  Name: /tmp/after-boom - Function: file.managed - Result: Failed - Started: 14:21:07.495481 - Duration: 0.015 ms
  Name: /tmp/independent - Function: file.managed - Result: Changed - Started: 14:21:07.495588 - Duration: 6.648 ms
Summary for minion1.lab
------------
Succeeded: 1 (changed=2)
Failed:    2
------------
Total states run:     3
Total run time:  13.314 ms
exit=1
$ salt 'minion1*' cmd.run 'ls /tmp/after-boom /tmp/independent 2>&1'
    ls: cannot access '/tmp/after-boom': No such file or directory
    /tmp/independent
The dependent file was never created; the independent one was. (Note "Succeeded: 1 (changed=2)": the count of changes includes the failed command's.) The exit codes seen: 0 when everything succeeded, 1 from the master's salt command when any minion returned a failure, and 2 when the target matched nothing. On the minion, salt-call --local state.apply --retcode-passthrough exited 2 for a failed run and 0 for a successful one, including a run that changed things (see the masterless section), so as with Chef you cannot tell "changed" from "nothing to do" by the exit code; read the summary.

cmd.run, the state, is like the module but declarative, and it has the same problem as exec in Puppet and execute in Chef: with no guard it runs every time, and every run reports a change.
$ cat err/guards.sls
no-guard:
  cmd.run:
    - name: echo ran > /tmp/g-noguard

with-creates:
  cmd.run:
    - name: echo ran > /tmp/g-creates
    - creates: /tmp/g-creates

with-unless:
  cmd.run:
    - name: echo ran > /tmp/g-unless
    - unless: test -f /tmp/g-unless
$ salt 'minion1*' state.apply err.guards --state-output=terse | grep -E 'Name:|Succeeded'     (second run)
  Name: echo ran > /tmp/g-noguard - Function: cmd.run - Result: Changed - Started: 14:21:12.217763 - Duration: 4.83 ms
  Name: echo ran > /tmp/g-creates - Function: cmd.run - Result: Clean - Started: 14:21:12.222761 - Duration: 714.921 ms
  Name: echo ran > /tmp/g-unless - Function: cmd.run - Result: Clean - Started: 14:21:12.937800 - Duration: 5.979 ms
Succeeded: 3 (changed=1)
creates (the file exists) and unless (this command succeeds) make the command run once. In the first run of the same file all three ran and reported "Changed".
File modes: I wrote the same mode three ways, an unquoted 644, the quoted string '0644' and the unquoted 0644 (which YAML reads as an octal number), and all three produced the same result:
$ salt 'minion1*' cmd.run 'stat -c "%n %a" /tmp/m-int /tmp/m-str /tmp/m-zero'
    /tmp/m-int 644
    /tmp/m-str 644
    /tmp/m-zero 644
Salt is forgiving here (Puppet 8 refuses an unquoted mode, as its page on this site shows). I would still write the quoted form, because it reads the same in every tool.

Orchestrate runs states in a fixed order across several minions, with requisites between steps, from the master:
$ cat /srv/salt/orch/deploy.sls
step-one:
  salt.function:
    - name: cmd.run
    - tgt: minion2.lab
    - arg:
      - echo "step one ran on $(hostname)" > /tmp/orch.log

step-two:
  salt.state:
    - tgt: minion1.lab
    - sls: webserver
    - require:
      - salt: step-one
$ salt-run state.orchestrate orch.deploy
          ID: step-one
    Function: salt.function
      Result: True
     Comment: Function ran successfully. Function cmd.run ran on minion2.lab.
          ID: step-two
    Function: salt.state
      Result: True
     Comment: States ran successfully. No changes made to minion1.lab.
Succeeded: 2 (changed=1)
Failed:    0
(the output is shortened to the ID, function, result and comment lines.) salt.function runs a module, salt.state applies a state file, and the require makes step two wait for step one. This is how you say "the database first, then the web servers" in one place.

The default is push: the master tells the minions what to do, when you run a command. A minion can also be told to run something on its own, regularly. I gave minion2 a schedule that runs state.apply every 20 seconds:
$ salt 'minion2*' schedule.add highstate function=state.apply seconds=20
minion2.lab:
    ----------
    changes:
        ----------
        highstate:
            added
    comment:
        Added job: highstate to schedule.
    result:
        True
$ salt 'minion2*' schedule.list
minion2.lab:
    schedule:
      highstate:
        enabled: true
        function: state.apply
        jid_include: true
        maxrunning: 1
        name: highstate
        saved: true
        seconds: 20
The proof that it ran by itself is the master's job cache, which saw three state.apply jobs from minion2.lab, twenty seconds apart, none of them started by me:
20261009142202460979 state.apply minion2.lab root 2026, Oct 09 14:22:02.460979
20261009142222466782 state.apply minion2.lab root 2026, Oct 09 14:22:22.466782
20261009142242666006 state.apply minion2.lab root 2026, Oct 09 14:22:42.666006
$ salt 'minion2*' schedule.delete highstate
        highstate:
            removed
In real life the interval is 30 minutes or an hour, not 20 seconds. (The same can be set in the minion's configuration or in pillar; not run in the lab.)

What happens when a machine is reinstalled and its Salt keys are gone? I stopped minion2, deleted its key files and started it again. The minion says:
CRITICAL The Salt Master has rejected this minion's public key!
To repair this issue, delete the public key for this minion on the Salt Master and restart this minion.
Or restart the Salt Master in open mode to clean out the keys. The Salt Minion will now exit.
and the master logs the reason, and now lists the ID twice:
ERROR Authentication attempt from minion2.lab failed, the public keys did not match. This may be an attempt to compromise the Salt cluster.
$ salt-key -L
Accepted Keys:
minion1.lab
minion2.lab
Denied Keys:
minion2.lab
Unaccepted Keys:
Rejected Keys:
$ salt "minion2*" test.ping
minion2.lab:
    Minion did not return. [Not connected]
This is the protection working: a machine that claims an existing name with a different key is refused, which is what you want when it is an intruder and what you must fix when it is your own reinstalled server. The repair is to delete the old key on the master, restart the minion, check the new fingerprint (with the method from the keys section), and accept:
$ salt-key -d minion2.lab -y
Key for minion minion2.lab deleted.
Key for minion minion2.lab deleted.
$ salt-key -L
Accepted Keys:
minion1.lab
Denied Keys:
Unaccepted Keys:
Rejected Keys:
(then, after the minion was started again:)
$ salt-key -L
Accepted Keys:
minion1.lab
Unaccepted Keys:
minion2.lab
master says:  ac1ee6da422902208fb222d3cbd7e0b56a95363587e91608f8c4a95010846a82
minion2 computes: ac1ee6da422902208fb222d3cbd7e0b56a95363587e91608f8c4a95010846a82
$ salt-key -a minion2.lab -y
Key for minion minion2.lab accepted.
$ salt '*' test.ping
minion2.lab:
    True
minion1.lab:
    True
A line you will also see in a minion's log while its key is still waiting to be accepted: "The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate".

salt-call --local runs Salt on the machine you are on, reading the states and pillar from its own /srv/salt and /srv/pillar. There is no daemon, no key and no network:
$ salt-call --local test.ping
local:
    True
$ salt-call --local grains.item os host
local:
    ----------
    host:
        solo
    os:
        Ubuntu
$ salt-call --local pillar.get webserver:port unmask=True
local:
    80
The first apply of the top file failed, as expected: this machine is named "solo", and the top file only matches 'minion*':
$ salt-call --local state.apply --retcode-passthrough --state-output=terse | tail -9; echo exit=${PIPESTATUS[0]}
  Name: states - Function: no.None - Result: Failed
Summary for local
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   0.000 ms
exit=2
(--retcode-passthrough makes the exit code follow the result of the run: 2 for failure.) Naming the state skips the top file:
$ salt-call --local state.apply webserver test=True --state-output=terse | tail -8
Summary for local
------------
Succeeded: 7 (unchanged=6, changed=5)
Failed:    0
$ salt-call --local state.apply webserver --retcode-passthrough --state-output=terse | tail -9; echo exit=${PIPESTATUS[0]}
Succeeded: 7 (changed=7)
Failed:    0
exit=0
$ salt-call --local state.apply webserver --retcode-passthrough --state-output=terse | tail -5; echo exit=${PIPESTATUS[0]}
Succeeded: 7
Failed:    0
exit=0
$ for s in shop blog; do curl -s -H "Host: $s.example.org" localhost/; done
<h1>Welcome to the shop</h1>
<p>Site shop on solo, Ubuntu 24.04.</p>
<h1>Read the blog</h1>
<p>Site blog on solo, Ubuntu 24.04.</p>
Note the exit code 0 for a run that changed seven things.
The trap, which I fell into: I then ran salt-call state.apply WITHOUT --local on this machine. With no master configured it uses the default master name, "salt", and in my lab that name belongs to the master container:
$ salt-call state.apply
Minion failed to authenticate with the master, has the minion key been accepted?
...a Python traceback ending in SystemExit: 2, exit=2
It had also registered a key request on that master:
$ salt-key -L          (on the master)
Accepted Keys:
minion1.lab
minion2.lab
Denied Keys:
Unaccepted Keys:
solo.lab
Rejected Keys:
$ salt-key -d solo.lab -y
Key for minion solo.lab deleted.
On a real network the default name salt may resolve to something, and a stray salt-call without --local asks it to be a minion. Always type --local on a masterless machine, or set file_client: local in /etc/salt/minion.d so that the default is safe (not run in the lab).

salt-ssh needs no Salt on the target, only SSH access and Python. The master has a roster, the list of machines and how to log in:
$ cat /etc/salt/roster
web3:
  host: slab-ssh
  user: root
  priv: /root/.ssh/id_lab
(a key pair made for this lab, with the public key in the target's authorized_keys; the target in this lab is a bare Ubuntu with sshd.) The first attempt:
$ salt-ssh -i web3 test.ping; echo exit=$?
web3:
    ----------
    _error:
        The command resulted in a non-zero exit code
    parsed:
        None
    retcode:
        10
    stderr:
        Warning: Permanently added 'slab-ssh' (ED25519) to the list of known hosts.
        ERROR: Unable to locate appropriate python command
    stdout:
        ERROR: Python version error. Recommendation(s) follow:
        - Install Python 3 on the target machine(s)
        - You can use ssh_pre_flight or raw shell (-r) to install Python 3
exit=10
(-i means "ignore host keys": convenient in a lab, not in production, because it accepts any host key.) The base Ubuntu image has no Python 3, and the message says what to do. The raw shell (-r) needs no Python:
$ salt-ssh -i web3 -r 'apt-get update -qq ...; apt-get install -y -qq python3 ...; python3 --version'
    stdout:
        Python 3.12.3
$ salt-ssh -i web3 test.ping
web3:
    True
$ salt-ssh -i web3 grains.item os host
web3:
    ----------
    host:
        web3
    os:
        Ubuntu
$ salt-ssh -i web3 state.apply webserver --state-output=terse | tail -12
  Name: nginx - Function: service.running - Result: Changed - Started: 14:25:55.655112 - Duration: 63.886 ms
  ...
Summary for web3
------------
Succeeded: 7 (changed=7)
Failed:    0
------------
Total states run:     7
Total run time:   7.923 s
$ salt-ssh -i web3 state.apply webserver --state-output=terse | tail -6
Succeeded: 7
Failed:    0
Total run time:  58.022 ms
The same state file configured the machine, with the master's pillar. The port was 8080 on this machine, because the master's pillar file had been changed to 8080 earlier in this lab, and salt-ssh used it:
$ for s in shop blog; do curl -s -H "Host: $s.example.org" localhost:8080/; done          (on web3)
<h1>Welcome to the shop</h1>
<p>Site shop on web3, Ubuntu 24.04.</p>
<h1>Read the blog</h1>
<p>Site blog on web3, Ubuntu 24.04.</p>
and afterwards the target had no Salt process running at all (a count of processes with salt in their command line: 0). It fits a few machines, first boots, or places where an agent is not allowed. I did not measure how its speed compares with a minion's, and roster options other than the key (passwords, sudo), and salt-ssh at scale, were not run in the lab.

Each of these ran in the lab, from a small state file applied to minion1.
  Rendering SLS 'base:err.yaml' failed: while parsing a block mapping ... did not find expected key       bad YAML indentation. The message gives the line and column of the block and of the line that does not fit:
      in "<unicode string>", line 2, column 3 / in "<unicode string>", line 4, column 4
  Rendering SLS 'base:err.jinja' failed: Jinja variable '... DictProxy object' has no attribute 'nosuch'; line 4       a Jinja error: it prints the failing line with an arrow (<======), here pillar["nosuch"]["deeper"] for a pillar key that does not exist
  Rendering SLS 'base:err.dup' failed: while constructing a mapping ... found conflicting ID 'a'       two states with the same ID in one file
  Referenced state does not exist for requisite [require: (file: nosuch)] in state [/tmp/r] in SLS [err.req]       a require that points at nothing
  State 'file.nosuch' was not found in SLS 'err.nofunc' / Reason: 'file.nosuch' is not available.       a state function that does not exist (this one is a normal failed state: Result: False)
  No matching sls found for 'nosuch' in env 'base'       the state name does not exist in file_roots
  No Top file or master_tops data matches found.       the top file matches nobody
  Minion did not return. [Not connected]       the minion's connection is gone (stopped, or a rejected key)
  ERROR: No return received / No minions matched the target.       a target that matches nothing, exit 2
  The Salt Master has rejected this minion's public key!       the key changed (see "A minion with a new key")
  Authentication attempt from minion2.lab failed, the public keys did not match.       the same, on the master's side
  Minion failed to authenticate with the master, has the minion key been accepted?       a salt-call (without --local) that the master has not accepted
  ERROR: Unable to locate appropriate python command       salt-ssh: no Python 3 on the target (exit 10)
  'state.show_top' is not available.       state.show_top is not a runner: run it with salt 'target' state.show_top

  - Real machines, systemd (the daemons were started by hand), and the packaged services
  - Windows, macOS and other Linux families as minions; salt-cloud; proxy minions for network devices
  - The reactor, beacons, returners, the event system beyond the job cache, salt-api, syndic and multi-master setups, high availability
  - Pillar encryption (GPG renderer), git as the file server, external pillars, SDB, the salt-cp command, the minion's startup_states
  - Version mixes (a master older than a minion, upgrading across majors), the apt pin
  - The master's own fingerprint check (salt-key -F master against key.finger_master)
  - requisites onchanges and onfail, event-driven runs, test=True combined with watch chains in depth
  - Roster options for salt-ssh other than a private key, and salt-ssh with sudo
  - Large numbers of minions: everything here was two minions, one masterless machine and one SSH target

Is everybody there:                              salt-run manage.up        salt-run manage.down
Run a command everywhere (or on some):           salt '*' cmd.run 'uptime'       salt -G 'os:Ubuntu' cmd.run 'uptime'
Preview, then apply, the whole top file:         salt '*' state.apply test=True       salt '*' state.apply
One state, one minion:                           salt 'minion1*' state.apply webserver
One machine at a time (rolling):                 salt 'web*' state.apply --batch-size 1
One line per state:                              salt '*' state.apply --state-output=terse
What does the master think a minion gets:       salt 'minion1*' state.show_top       salt 'minion1*' state.show_sls webserver
Reload grains / pillar after a change:           salt '*' saltutil.refresh_grains       salt '*' saltutil.refresh_pillar
See pillar values:                               salt 'minion1*' pillar.items unmask=True
Accept, delete and list keys:                    salt-key -a NAME -y       salt-key -d NAME -y       salt-key -L
Check a fingerprint (like with like):            salt-key -f NAME  (master)   printf '%s' "$(cat /etc/salt/pki/minion/minion.pub)" | sha256sum  (minion)
Jobs that ran:                                   salt-run jobs.list_jobs
Masterless, one state:                           salt-call --local state.apply webserver --retcode-passthrough
Agentless:                                       salt-ssh -i web3 state.apply webserver     (needs Python 3 on the target)

Symptom                                          Usually
Fingerprints on master and minion differ         in 3008.3 salt-key -f and key.finger hash different input for the SAME key. Compare with the sha256sum method above
No minions matched the target (exit 2)           the key is not accepted, the glob or grain is wrong, or the grain was written by hand and not refreshed (saltutil.refresh_grains)
Minion did not return. [Not connected]           the minion process is stopped, or its key was rejected: check the minion log and salt-key -L
pillar.get shows nothing, or **********          nothing: refresh it (saltutil.refresh_pillar); the stars: pillar masking since 3008, add unmask=True
State applies nothing: "No Top file ... matches"  the top file does not match the minion's ID: name the state (state.apply NAME), or fix the glob
Unaccepted key from a machine you did not set up   a stray salt-call without --local, or a minion started with the default master name "salt". Delete it with salt-key -d
Reinstalled server cannot connect                the master still has its old key: delete it (salt-key -d), restart the minion, verify, accept
Every run reports cmd.run as changed             cmd.run without creates / unless / onlyif
A service does not reload after a config change  the config states need watch_in (or the service a watch), and the service needs reload: True to reload instead of restart
salt-ssh: Unable to locate appropriate python    install Python 3 on the target (salt-ssh -r, or ssh_pre_flight)
A YAML line fails in a way that makes no sense   indentation (spaces only) or the Jinja that was run before the YAML was read: show the rendered text with state.show_sls

Preview with test=True, apply, run it again and expect "Clean". Keep exit codes in mind: 0 is "no failure", not "nothing changed".
Keep data in pillar and logic in states: a loop over a pillar dictionary is a whole fleet's worth of config in one short file.
Verify a key before you accept it, and verify it with a method that compares like with like.
Refresh grains and pillar after changing them, and use unmask=True when debugging pillar.
Write down every dependency: require for order, watch/watch_in for reactions.
Guard every cmd.run, and use a real state (pkg, file, service, user) whenever there is one.
Pin the Salt version, upgrade the master first, and test the upgrade on a copy.
Prefer the push-and-schedule combination you can reason about: an interval on the minion for steady state, a manual state.apply for changes.
Use --local on masterless machines, always.
Keep the SLS files in version control, and treat the master as the most sensitive machine you have: it can run commands on every minion.
Only manage machines you are authorized to manage.