🐝 vm-bhyve: Virtual Machines with Short Commands

The manager that turns the long bhyve command line into vm create, vm install and vm start, with a tested walkthrough: install, boot, network, snapshot and clone

Video: vm-bhyve (FreeBSD video)
Prefer to watch? This short video runs this page's main commands on a real FreeBSD 15.0 machine, with on-screen captions and no voiceover. Click to play, or watch it on YouTube.

Why a manager?

On the bhyve page we started a virtual machine with one very long command: slots, devices, a tap, a bridge, a disk path, a console. It teaches you how bhyve works, and it is miserable to repeat. vm-bhyve is the helper most people use on top of it. It is a set of shell scripts (the package puts them in /usr/local/lib/vm-bhyve) that keeps each virtual machine's settings in a small file, creates the tap devices, bridges and ZFS volumes for you, and gives you a single command, vm, with sub-commands: vm create, vm install, vm start, vm stop, vm snapshot, vm clone and so on.

Everything below was run for real on FreeBSD 15.0-RELEASE, in the same nested lab as the bhyve page: a virtual machine (2 processors, 2 GB) running guests inside it. It is slower than real hardware, and it works.

Step 1: install and initialise

root@practice:~ # pkg install -y vm-bhyve 2>&1 | grep -E "vm-bhyve|Number"
	vm-bhyve: 1.7.5 [FreeBSD-ports]
Number of packages to be installed: 1
[1/1] Fetching vm-bhyve-1.7.5: ......... done
[1/1] Installing vm-bhyve-1.7.5...
[1/1] Extracting vm-bhyve-1.7.5: .......... done

(A note from our own mistakes: the first pkg install said No packages available to install matching 'vm-bhyve'. On the poudriere page we had switched the official repositories off on this machine, so pkg could only see our own small repository. We removed our repository file and ran pkg update -f, and it was found.) Two lines in /etc/rc.conf switch the manager on:

root@practice:~ # sysrc vm_enable=YES
vm_enable:  -> YES
root@practice:~ # sysrc vm_dir="zfs:zroot/vm"
vm_dir:  -> zfs:zroot/vm
root@practice:~ # vm init
root@practice:~ # ls -a /zroot/vm
.		..		.config		.img		.iso		.templates
root@practice:~ # ls /usr/local/share/examples/vm-bhyve
config.sample		freebsd-zvol.conf	legacy-arch.conf	legacy-ubuntu.conf	linux-zvol.conf		openbsd.conf
default.conf		freepbx.conf		legacy-debian.conf	linux-grub-zvol.conf	linux.conf		resflash.conf
dragonfly.conf		legacy-alpine.conf	legacy-gentoo.conf	linux-grub.conf		netbsd.conf		windows.conf
root@practice:~ # cp /usr/local/share/examples/vm-bhyve/* /zroot/vm/.templates/
root@practice:~ # ls /zroot/vm/.templates
config.sample		freebsd-zvol.conf	legacy-arch.conf	legacy-ubuntu.conf	linux-zvol.conf		openbsd.conf
default.conf		freepbx.conf		legacy-debian.conf	linux-grub-zvol.conf	linux.conf		resflash.conf
dragonfly.conf		legacy-alpine.conf	legacy-gentoo.conf	linux-grub.conf		netbsd.conf		windows.conf

Step 2: a virtual switch

root@practice:~ # vm switch create -a 10.88.0.1/24 private
root@practice:~ # vm switch list
NAME     TYPE      IFACE       ADDRESS       PRIVATE  MTU  VLAN  PORTS
private  standard  vm-private  10.88.0.1/24  no       -    -     -
root@practice:~ # ifconfig vm-private | grep -E "^vm-private|inet |member"
vm-private: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	inet 10.88.0.1 netmask 0xffffff00 broadcast 10.88.0.255
root@practice:~ # grep 'skip' /etc/pf.conf
set skip on { lo0 bridge0 }
root@practice:~ # sed -i '' 's/set skip on { lo0 bridge0 }/set skip on { lo0 bridge0 vm-private }/' /etc/pf.conf
root@practice:~ # pfctl -nf /etc/pf.conf
root@practice:~ # pfctl -f /etc/pf.conf
root@practice:~ # grep 'skip' /etc/pf.conf
set skip on { lo0 bridge0 vm-private }

vm switch create -a 10.88.0.1/24 private made a virtual switch named private. Behind it vm-bhyve built a bridge, vm-private, with the address 10.88.0.1/24 for the host. Every guest connected to this switch will be on that network, and nothing else: it is a private network inside the machine. (A switch that is joined to a real network card, so that guests appear on your LAN, is made with vm switch add: from the documentation, not run here.)

Then the firewall: the host runs pf from the pf page, whose default rule blocks every interface. We added the new bridge to the interfaces that pf skips, as we did for bridge0 on the bhyve page, and the problem never appeared. Any bridge that carries your guests' traffic needs this.

Step 3: install media

root@practice:~ # time dd if=/dev/cd0 of=/var/tmp/FreeBSD-15.0-RELEASE-dvd1.iso bs=1m
4201+1 records in
4201+1 records out
4405243904 bytes transferred in 8.345128 secs (527882148 bytes/sec)
        8.34 real         0.05 user         1.62 sys
root@practice:~ # vm iso /var/tmp/FreeBSD-15.0-RELEASE-dvd1.iso
/zroot/vm/.iso/FreeBSD-15.0-RELEASE-dvd1.iso    1% of 4201 MB   [... progress shown while it copies ...]
root@practice:~ # vm iso
DATASTORE           FILENAME
default             FreeBSD-15.0-RELEASE-dvd1.iso

We copied the installation disc into a file (8 seconds), because, as on the bhyve page, the loader needs a regular file. vm iso FILE imports it into the .iso folder, and plain vm iso lists the available images. (Given a web address, vm iso URL downloads an image for you: from the documentation, not run here.)

Step 4: create a guest

root@practice:~ # vm create -t freebsd-zvol -s 4G -c 1 -m 768M guest1
root@practice:~ # vm list
NAME    DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO  STATE
guest1  default    bhyveload  1    768M    -    No    Stopped
root@practice:~ # cat /usr/local/share/examples/vm-bhyve/freebsd-zvol.conf
loader="bhyveload"
cpu=1
memory=256M
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"
root@practice:~ # cat /zroot/vm/guest1/guest1.conf
loader="bhyveload"
cpu="1"
memory="768M"
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"
uuid="e7fc4fa2-c18c-11f1-ae02-589cfc10bd50"
network0_mac="58:9c:fc:03:60:8b"
root@practice:~ # sysrc -f /zroot/vm/guest1/guest1.conf network0_switch="private"
network0_switch: public -> private
root@practice:~ # grep network0 /zroot/vm/guest1/guest1.conf
network0_type="virtio-net"
network0_switch="private"
network0_mac="58:9c:fc:03:60:8b"
root@practice:~ # zfs list -r -o name,used,volsize zroot/vm
NAME                    USED  VOLSIZE
zroot/vm               3.59G        -
zroot/vm/guest1         156K        -
zroot/vm/guest1/disk0    56K       4G
root@practice:~ # ls /zroot/vm/guest1
guest1.conf	vm-bhyve.log

One command made the guest: vm create -t freebsd-zvol -s 4G -c 1 -m 768M guest1 (template, 4 GB disk, 1 CPU, 768 MB of memory, name). vm list shows it, Stopped. Now read the guest's settings file, which is the whole configuration of the machine:

A small trap: the missing device file

Our first attempt to start the guest failed, with no message on screen. The manager's log said why, or at least how:

root@practice:~ # tail -4 /zroot/vm/guest1/vm-bhyve.log
2026-10-06T13:51:04+00:00:  [bhyve iso device: -s 3:0,ahci-cd,/zroot/vm/.iso/FreeBSD-15.0-RELEASE-dvd1.iso,ro,bootindex=1]
2026-10-06T13:51:04+00:00: starting bhyve (run 1)
2026-10-06T13:51:04+00:00: bhyve exited with status 4
2026-10-06T13:51:04+00:00: destroying network device tap0
2026-10-06T13:51:04+00:00: stopped
root@practice:~ # [the bhyve command from the log, run by hand to see its error message]
bhyve: Could not open backing file: /dev/zvol/zroot/vm/guest1/disk0: No such file or directory
Could not open backing file: No such file or directory
Device emulation initialization error: No such file or directory
root@practice:~ # ls /dev/zvol
ls: /dev/zvol: No such file or directory

bhyve exited with status 4 is all that vm-bhyve knows. Running the same bhyve command by hand showed the real message: the disk's device file, /dev/zvol/zroot/vm/guest1/disk0, did not exist, and there was no /dev/zvol folder at all. The volume was there and healthy. The device file for it was not.

We tried to find out when this happens:

root@practice:~ # zfs create -s -V 8M -o volmode=dev zroot/vm/t3
root@practice:~ # ls /dev/zvol/zroot/vm/
guest1
root@practice:~ # zfs create -V 8M -o volmode=dev zroot/vm/t4
root@practice:~ # ls /dev/zvol/zroot/vm/
guest1	t4

A sparse volume (-s) created with volmode=dev got no device file, while the same volume without -s did. vm-bhyve's template uses exactly the sparse kind. When it happened, setting the volume's volmode again (zfs set volmode=geom and then back to dev) made the device file appear, and the guest started. We could not make it happen every time: later runs, with brand-new guests, started without any trick. In a later test on a fresh machine it came back (the first guest after vm init had no device file, and vm install stopped at once), and the same zfs set volmode trick fixed it. So treat it as a quirk to know about, not as something you will certainly meet. If a guest fails with exited with status 4 and a "backing file" message, check for the device file. A way around it is a non-sparse volume: change disk0_dev to zvol in the guest's file (from the documentation, not run here), at the cost of reserving the whole 4 GB.

Step 5: install the guest

root@practice:~ # vm install guest1 FreeBSD-15.0-RELEASE-dvd1.iso
Starting guest1
  * found guest in /zroot/vm/guest1
  * booting...
root@practice:~ # vm list
NAME    DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO  STATE
guest1  default    bhyveload  1    768M    -    No    Bootloader (11541)
root@practice:~ # vm console guest1
[... the guest boots from the ISO: the same boot messages as on the bhyve page ...]
Console type [vt100]: 
[... in the guest console: the installer shell, the unattended settings typed in with a heredoc, then bsdinstall script (it took 68 seconds in this nested setup), and halt -p ...]
root@practice:~ # vm list
NAME    DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO  STATE
guest1  default    bhyveload  1    768M    -    No    Stopped

The ordinary way is to click through the installer's screens on the same console. We only scripted it so that the page could be tested from start to finish.

Step 6: boot it from its own disk

root@practice:~ # vm start guest1
Starting guest1
  * found guest in /zroot/vm/guest1
  * booting...
root@practice:~ # vm list
NAME    DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO  STATE
guest1  default    bhyveload  1    768M    -    No    Bootloader (12175)
root@practice:~ # vm switch list
NAME     TYPE      IFACE       ADDRESS       PRIVATE  MTU  VLAN  PORTS
private  standard  vm-private  10.88.0.1/24  no       -    -     -
root@practice:~ # ping -c 2 -t 3 10.88.0.2
PING 10.88.0.2 (10.88.0.2): 56 data bytes
64 bytes from 10.88.0.2: icmp_seq=0 ttl=64 time=2.722 ms
64 bytes from 10.88.0.2: icmp_seq=1 ttl=64 time=0.987 ms

--- 10.88.0.2 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.987/1.854/2.722/0.867 ms
root@practice:~ # ssh root@10.88.0.2 'hostname; uname -srm; zpool list -H -o name,size,alloc; df -h / | tail -1'
(root@10.88.0.2) Password for root@guest1: 
guest1
FreeBSD 15.0-RELEASE amd64
zroot	3.75G	694M
zroot/ROOT/default    3.6G    686M    2.9G    19%    /
root@practice:~ # vm list
NAME    DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO  STATE
guest1  default    bhyveload  1    768M    -    No    Running (12247)
root@practice:~ # vm switch list
NAME     TYPE      IFACE       ADDRESS       PRIVATE  MTU  VLAN  PORTS
private  standard  vm-private  10.88.0.1/24  no       -    -     -

vm start guest1 boots the guest from its disk, without the install disc (the disc was only attached by vm install). The state goes through Bootloader to Running. About a minute later it answers ping on 10.88.0.2 in a millisecond or two, and we can log in with SSH from the host: the guest says it is guest1, FreeBSD 15.0-RELEASE amd64, with its own ZFS pool, zroot, of 3.75 GB with 694 MB used, and a root file system that is 19 percent full. (SSH asked for a password this time. For real guests you will use keys.) That is a complete, separate FreeBSD machine, created, installed and booted with a handful of short commands.

Looking at a guest

root@practice:~ # vm info guest1 | head -24
------------------------
Virtual Machine: guest1
------------------------
  state: running (12247)
  datastore: default
  loader: bhyveload
  uuid: 89e8fb3f-c190-11f1-ae02-589cfc10bd50
  cpu: 1
  memory: 768M
  memory-resident: 187420672 (178.738M)

  console-ports
    com1: /dev/nmdm-guest1.1B

  network-interface
    number: 0
    emulation: virtio-net
    virtual-switch: private
    fixed-mac-address: 58:9c:fc:00:4a:aa
    fixed-device: -
    active-device: tap0
    desc: vmnet/guest1/0/private
    mtu: 1500
    bridge: vm-private

vm info guest1 prints everything about a running guest: its state and process number, the CPU and memory, memory-resident: how much of its 768 MB it is actually using on the host (178 MB), the console device, and the network card: virtio-net, plugged into the switch private through the bridge vm-private, using the tap device tap0. The other everyday commands:

Snapshots and clones, the ZFS way

root@practice:~ # vm snapshot guest1@clean
/usr/local/sbin/vm: WARNING: guest1 appears to be running on practice (locked)
/usr/local/sbin/vm: ERROR: guest1 must be powered off first (use -f to override)
root@practice:~ # vm stop guest1
Sending ACPI shutdown to guest1
root@practice:~ # vm list
NAME    DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO  STATE
guest1  default    bhyveload  1    768M    -    No    Stopped

We tried to take a snapshot of the running guest, and vm refused: guest1 must be powered off first (use -f to override). A snapshot of a running machine's disk is like a snapshot of a computer whose power was pulled: it will usually be fine, but the manager makes you decide on purpose with -f. The clean way is to stop the guest first. vm stop sent the shutdown request, and after a few seconds the state was Stopped.

root@practice:~ # vm list
NAME    DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO     STATE
guest1  default    bhyveload  1    768M    -    Yes [1]  Stopped
root@practice:~ # vm snapshot guest1@clean
root@practice:~ # zfs list -r -t snapshot -o name,used zroot/vm/guest1
NAME                          USED
zroot/vm/guest1@clean           0B
zroot/vm/guest1/disk0@clean     0B
root@practice:~ # vm clone guest1@clean guest2
root@practice:~ # vm list
NAME    DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO     STATE
guest1  default    bhyveload  1    768M    -    Yes [1]  Stopped
guest2  default    bhyveload  1    768M    -    No       Stopped
root@practice:~ # zfs list -r -o name,used,refer,origin zroot/vm/guest2
NAME                    USED  REFER  ORIGIN
zroot/vm/guest2           8K   112K  zroot/vm/guest1@clean
zroot/vm/guest2/disk0     8K   481M  zroot/vm/guest1/disk0@clean
root@practice:~ # vm start guest2
Starting guest2
  * found guest in /zroot/vm/guest2
  * booting...
root@practice:~ # ping -c 2 -t 3 10.88.0.2
PING 10.88.0.2 (10.88.0.2): 56 data bytes
64 bytes from 10.88.0.2: icmp_seq=0 ttl=64 time=2.037 ms
64 bytes from 10.88.0.2: icmp_seq=1 ttl=64 time=0.894 ms

--- 10.88.0.2 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.894/1.465/2.037/0.571 ms
root@practice:~ # ssh root@10.88.0.2 'hostname; uname -sr'
(root@10.88.0.2) Password for root@guest1: 
guest1
FreeBSD 15.0-RELEASE
root@practice:~ # vm list
NAME    DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO     STATE
guest1  default    bhyveload  1    768M    -    Yes [1]  Stopped
guest2  default    bhyveload  1    768M    -    No       Running (14283)
root@practice:~ # zfs list -r -o name,used,refer zroot/vm/guest1 zroot/vm/guest2
NAME                    USED  REFER
zroot/vm/guest1         481M   112K
zroot/vm/guest1/disk0   481M   481M
zroot/vm/guest2        2.65M   120K
zroot/vm/guest2/disk0  2.56M   482M

We started guest2, and it booted like any other guest. Notice what a clone inherits: it has the same host name (guest1, as the SSH login shows) and the same fixed address 10.88.0.2. Never run a clone and its original at the same time without changing those, or you will have two machines with the same IP address on one network. (We started only the clone: the original was stopped.) The sizes in the last zfs list: the original disk 481M, the clone's disk 2.56M: only what has changed since the snapshot. Clones are perfect for testing a risky change on a copy of a real machine.

root@practice:~ # vm stop guest2
Sending ACPI shutdown to guest2
root@practice:~ # vm list
NAME    DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO     STATE
guest1  default    bhyveload  1    768M    -    Yes [1]  Stopped
guest2  default    bhyveload  1    768M    -    No       Stopped
root@practice:~ # vm destroy -f guest2
root@practice:~ # vm list
NAME    DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO     STATE
guest1  default    bhyveload  1    768M    -    Yes [1]  Stopped
root@practice:~ # zfs list -r -o name,used zroot/vm
NAME                    USED
zroot/vm               4.06G
zroot/vm/guest1         481M
zroot/vm/guest1/disk0   481M

vm stop guest2 and vm destroy -f guest2 removed the clone completely, and zfs list shows only guest1 left.

Autostart

root@practice:~ # sysrc vm_list="guest1"
vm_list:  -> guest1
root@practice:~ # sysrc vm_delay=5
vm_delay:  -> 5
root@practice:~ # grep "^vm_" /etc/rc.conf
vm_enable="YES"
vm_dir="zfs:zroot/vm"
vm_list="guest1"
vm_delay="5"

vm_list="guest1" in /etc/rc.conf makes the guests in the list start when the host boots, and vm_delay="5" waits five seconds between them. (After setting it, vm list shows Yes [1] in the AUTO column: guest number 1 in the start order. You can see it in the listing before the snapshot.) The guests are also shut down gracefully when the host shuts down (from the documentation, not run here).

What we did not do

All from the documentation, not run here:

Which tool for which job

⚠️ A clone is a twin. It has the same host name, the same fixed IP address and the same SSH keys as the original. Change them before you put a clone on a network next to the original.

Why did the virtual machine go to therapy?<br>
Too many clones,<br>
and nobody knew which one was the original.

🧪 Try it yourself

On a FreeBSD machine with ZFS and CPU virtualization (a nested VM works, slowly):

1. Install vm-bhyve, set vm_enable and vm_dir in /etc/rc.conf, run vm init and copy the example templates. 2. Make a switch with an address, and (if you use pf) let pf skip its bridge. 3. Import an install ISO with vm iso. 4. Create a guest from the freebsd-zvol template, set its switch, and read its settings file. 5. Install it with vm install and vm console (use the installer's screens, or an unattended script). 6. Start it from its disk, ping it and log in with SSH. 7. Stop it, take a snapshot, clone it, start the clone and look at the sizes with zfs list. Whatever you do, check the clone's host name and address. 8. Destroy the clone, and set up the original for autostart.

The cheat sheet

← poudriere: Build Your Own Packages, Properly · ↑ FreeBSD Corner · ipfw: The Other FreeBSD Firewall →