💾 Persistent Volumes on OpenStack: Cinder, Snapshots, Resize and Boot from Volume

A volume is a disk that outlives any server. Create one, attach it, format and fill it, move it to another server, snapshot it, grow it, boot from one, and see what is really behind it: a thin LVM volume exported over iSCSI. With every error and trap, run for real

💡 How this page was made. Everything shown as output was run for real on 8 October 2026 on the DevStack cloud from the earlier pages (stable/2026.1, one machine, 4 CPUs, 8 GB of RAM). Commands run as the ordinary demo user, except lines marked devstack-admin (an administrator, for things a normal user may not see). Guests are tiny CirrOS instances; inside them the commands run as the cirros user with sudo. The ssh commands are shortened (the key and a few options are left out), and some long tables are trimmed, which the transcripts say. Backups and object storage are not installed in this cloud, and the page shows what that looks like instead of pretending.

Contents
   Why volumes exist
   What Cinder is, and what is behind it
   Step 1: create a volume
   Step 2: attach it, then make it usable
   What is really behind /dev/vdb
   The point of it all: data that outlives the server
   Thin provisioning: promising more than exists
   Snapshots, and volumes made from snapshots
   Growing a volume
   Boot from a volume
   A volume you did not make: the image cache
   What is not installed: backups and object storage
   Errors, in their real words
   Cleaning up
   The pitfalls, in one list
   What this page did not cover
   The cheat sheet

Why volumes exist

When you launch a server (see the previous page) its disk is part of the server. On the compute host it is just a file, and it lives and dies with the instance. A volume is a separate disk managed by its own service, Cinder, with its own life:

you create it on its own, any size you like (within your quota),
you attach it to a running server (and detach it again),
it keeps its data when the server is stopped, rebuilt or deleted,
you can snapshot it, copy it from a snapshot, grow it, and boot a server from it.

It is the OpenStack counterpart of an EBS volume on AWS or a managed disk on Azure. It is a block device: the guest sees an empty disk and you put a filesystem on it. The other storage kinds are different services. Object storage (Swift: files accessed over HTTP) is not installed in this cloud:

openstack container list
public endpoint for object-store service in RegionOne region not found

and shared file systems (Manila) are not covered at all.

What Cinder is, and what is behind it

Cinder is, like Nova, a set of cooperating processes: an API (c-api), a scheduler (c-sch, which decides which storage backend gets a new volume, like Nova's scheduler for hosts) and a volume service (c-vol, which talks to the real storage). The real storage is a backend with a driver. Here is what the demo user sees, and what is behind it on the DevStack machine:

$ openstack volume type list
+--------------------------------------+-------------+-----------+
| ID                                   | Name        | Is Public |
+--------------------------------------+-------------+-----------+
| 097633d1-8682-48e3-be0f-2fc0eb1e1c6c | lvmdriver-1 | True      |
| 32a4bcf0-f40a-4470-8875-6724bf909d74 | __DEFAULT__ | True      |
+--------------------------------------+-------------+-----------+
$ openstack volume list; openstack volume snapshot list
$ openstack limits show --absolute -c Name -c Value | grep -iE "volume|gigabyte|snapshot|backup"
| max_total_backup_gigabytes  |  1000 |
| max_total_backups           |    10 |
| max_total_snapshots         |    10 |
| max_total_volume_gigabytes  |  1000 |
| max_total_volumes           |    10 |
| total_backup_gigabytes_used |     0 |
| total_backups_used          |     0 |
| total_gigabytes_used        |     0 |
| total_snapshots_used        |     0 |
| total_volumes_used          |     0 |
$ openstack volume service list            (as the demo user)
ForbiddenException: 403: Client Error for url: http://10.0.2.15/volume/v3/os-services, Policy doesn't allow volume_extension:services:index to be performed.
$ openstack container list                 (Swift, the object storage)
public endpoint for object-store service in RegionOne region not found

Two volume types. A type chooses a backend and its properties (fast, slow, replicated...). Here both go to the same place, and lvmdriver-1 is the default.
Quotas: 10 volumes, 10 snapshots, 1000 GB, 10 backups for this project.
Services are admin territory. The demo user gets 403 ... Policy doesn't allow volume_extension:services:index: only an administrator may list the Cinder services.

The storage itself, on the machine that runs cinder-volume:

$ sudo vgs --units g -o vg_name,vg_size,vg_free
  VG                        VSize  VFree
  stack-volumes-lvmdriver-1 10.00g 0.47g
$ sudo lvs -o lv_name,lv_size,lv_attr,data_percent
  LV                             LSize Attr       Data%
  stack-volumes-lvmdriver-1-pool 9.50g twi-a-tz-- 0.00
$ grep -A8 "^\[lvmdriver-1\]" /etc/cinder/cinder.conf
[lvmdriver-1]
image_volume_cache_enabled = True
volume_clear = zero
lvm_type = auto
target_prefix = iqn.2010-10.org.openstack:
target_port = 3260
target_protocol = iscsi
target_helper = lioadm
volume_group = stack-volumes-lvmdriver-1
$ grep -E "^(enabled_backends|default_volume_type)" /etc/cinder/cinder.conf
default_volume_type = lvmdriver-1
enabled_backends = lvmdriver-1
$ systemctl is-active devstack@c-api devstack@c-sch devstack@c-vol
active
active
active

DevStack's backend is the simplest one that works: LVM. A 10 GB file (the one named in local.conf on the install page) is attached as a loop device and made into a volume group; inside it, a thin pool of 9.5 GB. Every Cinder volume becomes a thin logical volume in that pool. The volume service then makes each volume reachable over the network with iSCSI, using the Linux kernel's own target (target_helper = lioadm, port 3260). Production clouds typically use real storage arrays or Ceph for the same job, with the same OpenStack commands on top (not shown here).

Step 1: create a volume

openstack volume create --size 1 data1

$ openstack volume create --size 1 data1           (the table is trimmed)
| id                | edb223a8-3e74-4be2-af0f-1abb102a885f |
| name              | data1                                |
| size              | 1                                    |
| status            | creating                             |
| type              | lvmdriver-1                          |
| bootable          | False                                |
| encrypted         | False                                |
| multiattach       | False                                |
| availability_zone | nova                                 |
| attachments       | []                                   |
$ openstack volume list
+--------------------------------------+-------+-----------+------+-------------+
| ID                                   | Name  | Status    | Size | Attached to |
+--------------------------------------+-------+-----------+------+-------------+
| edb223a8-3e74-4be2-af0f-1abb102a885f | data1 | available |    1 |             |
+--------------------------------------+-------+-----------+------+-------------+
$ sudo lvs -o lv_name,lv_size,lv_attr,data_percent,pool_lv
  LV                                          LSize Attr       Data%  Pool
  stack-volumes-lvmdriver-1-pool              9.50g twi-aotz-- 0.00
  volume-edb223a8-3e74-4be2-af0f-1abb102a885f 1.00g Vwi-a-tz-- 0.00   stack-volumes-lvmdriver-1-pool
$ sudo vgs --units g -o vg_name,vg_size,vg_free
  VG                        VSize  VFree
  stack-volumes-lvmdriver-1 10.00g 0.47g
$ openstack limits show --absolute -c Name -c Value | grep -E "total_(volumes|gigabytes)_used"
| total_gigabytes_used        |     1 |
| total_volumes_used          |     1 |

The volume goes creating, then available (a couple of seconds): it exists, and it is attached to nothing. Look at the storage machine: a new thin logical volume named volume-<the volume's ID> of 1.00 GB with 0.00% used, and the volume group's free space did not change (0.47 GB before and after). That is thin provisioning: the pool hands out space only as it is written. The project's quota usage went up by one volume and one gigabyte.

Step 2: attach it, then make it usable

Attach it to a running server (here vm1, a CirrOS server with a floating IP, as on the previous page):

openstack server add volume vm1 data1

$ openstack server add volume vm1 data1
+-----------------------+--------------------------------------+
| Field                 | Value                                |
+-----------------------+--------------------------------------+
| ID                    | edb223a8-3e74-4be2-af0f-1abb102a885f |
| Server ID             | 6cbc940d-46db-4ed8-812e-dc3f67c41966 |
| Volume ID             | edb223a8-3e74-4be2-af0f-1abb102a885f |
| Device                | /dev/vdb                             |
| Tag                   | None                                 |
| Delete On Termination | False                                |
+-----------------------+--------------------------------------+
$ openstack volume list
+--------------------------------------+-------+--------+------+------------------------------+
| ID                                   | Name  | Status | Size | Attached to                  |
+--------------------------------------+-------+--------+------+------------------------------+
| edb223a8-3e74-4be2-af0f-1abb102a885f | data1 | in-use |    1 | Attached to vm1 on /dev/vdb  |
+--------------------------------------+-------+--------+------+------------------------------+
$ (inside vm1) cat /proc/partitions
major minor  #blocks  name

 252        0    1048576 vda
 252        1    1039343 vda1
 252       15       8192 vda15
 252       16    1048576 vdb
$ (inside vm1) sudo blkid /dev/vdb; echo "blkid rc=$?"
blkid rc=2
$ (inside vm1) sudo mkfs.ext4 -q -L data1 /dev/vdb; sudo blkid /dev/vdb
/dev/vdb: LABEL="data1" UUID="e73fb15d-ea50-4c98-bc70-3c9b0328ef92" BLOCK_SIZE="4096" TYPE="ext4"
$ (inside vm1) sudo mkdir -p /mnt/data1 && sudo mount /dev/vdb /mnt/data1 && df -h /mnt/data1 | tail -2
Filesystem                Size      Used Available Use% Mounted on
/dev/vdb                973.4M     24.0K    906.2M   0% /mnt/data1
$ (inside vm1) echo "written on a volume at $(date -u +%T)" | sudo tee /mnt/data1/hello.txt; sudo sync; ls -la /mnt/data1
written on a volume at 23:39:06
total 28
drwxr-xr-x    3 root     root          4096 Oct  9 00:39 .
drwxr-xr-x    3 root     root          4096 Oct  9 00:39 ..
-rw-------    1 root     root            32 Oct  9 00:39 hello.txt
drwx------    2 root     root         16384 Oct  9 00:39 lost+found

What to read:

The attachment is hot: the server keeps running. Nova names the device /dev/vdb (vda is the root disk). The volume's status becomes in-use, and volume list has the Attached to column.
Inside the guest, the new disk simply appears in /proc/partitions, 1 GB, with no partition table and no filesystem: blkid finds nothing (return code 2).
So you format it: mkfs.ext4 -L data1 /dev/vdb, mount, and use it. Everything that follows (a file written, then synced) is plain Linux. The file hello.txt was written by sudo tee, so it is owned by root with mode 600: permissions live inside the volume, and travel with it.
/dev/vdb is a suggestion, not a promise. A second volume attached to the same server later showed up as /dev/vdc, and after a detach and a reattach the letters can change. In /etc/fstab or scripts, refer to the volume by label or UUID (LABEL=data1), as blkid shows.

What is really behind /dev/vdb

For the curious, here is the whole chain, from the storage side up to the guest. It is worth seeing once, because it explains why attaching a volume involves two services (Cinder prepares the volume, Nova connects the host to it):

$ openstack volume show data1 -c status -c attachments -f yaml
attachments:
- attached_at: '2026-10-08T23:38:44.000000'
  attachment_id: 3e27de87-cde6-4b73-b4e0-4f170c1f63eb
  device: /dev/vdb
  host_name: null
  id: edb223a8-3e74-4be2-af0f-1abb102a885f
  server_id: 6cbc940d-46db-4ed8-812e-dc3f67c41966
  volume_id: edb223a8-3e74-4be2-af0f-1abb102a885f
status: in-use
$ sudo lvs -o lv_name,lv_size,lv_attr,data_percent
  LV                                          LSize Attr       Data%
  stack-volumes-lvmdriver-1-pool              9.50g twi-aotz-- 0.36
  volume-edb223a8-3e74-4be2-af0f-1abb102a885f 1.00g Vwi-aotz-- 3.42
$ sudo targetcli ls            (trimmed to the lines that matter)
o- / .................................................................................. [...]
  o- backstores ....................................................................... [...]
  | o- block ........................................................... [Storage Objects: 1]
  | | o- iqn.2010-10.org.openstack:volume-edb223a8-3e74-4be2-af0f-1abb102a885f  [/dev/stack-volumes-lvmdriver-1/volume-edb223a8-3e74-4be2-af0f-1abb102a885f (1.0GiB) write-thru activated]
  o- iscsi ......................................................................... [Targets: 1]
  |   o- iqn.2010-10.org.openstack:volume-edb223a8-3e74-4be2-af0f-1abb102a885f ...... [TPGs: 1]
  |     |     o- portals ................................................................ [Portals: 1]
  |       o- 10.0.2.15:3260 ............................................................... [OK]
$ sudo iscsiadm -m session
tcp: [1] 10.0.2.15:3260,1 iqn.2010-10.org.openstack:volume-edb223a8-3e74-4be2-af0f-1abb102a885f (non-flash)
$ sudo virsh domblklist <the instance>
 Target   Source
------------------------------------------------------------------------------------
 vda      /opt/stack/data/nova/instances/6cbc940d-46db-4ed8-812e-dc3f67c41966/disk
 vdb      /dev/sda

Reading it from the bottom:

A thin LV on the storage machine (volume-edb223a8-..., the data percentage grew from 0 to 3.42 after mkfs wrote its metadata).
Cinder exports it as an iSCSI target (iqn.2010-10.org.openstack:volume-edb223a8-...) with the Linux target targetcli shows, and only the compute host's initiator name is allowed in (acls).
The compute service logs in as an iSCSI initiator: iscsiadm -m session shows the session, and the volume appears on the compute host as /dev/sda.
libvirt hands that host device to the guest as a virtio disk: domblklist shows vdb /dev/sda, next to vda, which is a plain file under /opt/stack/data/nova/instances/<server id>/disk.

On this one-machine cloud the storage server and the compute host are the same box, but the same chain works across the network: that is the point of iSCSI. The attachment record (the attachments list in volume show) is Cinder's note that "this server has this volume". If you ever delete things behind OpenStack's back, that is where the two sides disagree.

The point of it all: data that outlives the server

Detach the volume from vm1 (unmount it first), and attach it to a different server:

$ (inside vm1) sudo umount /mnt/data1
$ openstack server remove volume vm1 data1
$ openstack volume list -c Name -c Status -c "Attached to"
+-------+-----------+-------------+
| Name  | Status    | Attached to |
+-------+-----------+-------------+
| data1 | available |             |
+-------+-----------+-------------+
$ openstack server create ... --wait vm2        (a second server, with its own floating IP)
$ openstack server add volume vm2 data1 -c Device -f value
/dev/vdb
$ (inside vm2) sudo mkdir -p /mnt/data1 && sudo mount /dev/vdb /mnt/data1 && ls /mnt/data1
hello.txt
lost+found
$ openstack server add volume vm1 data1        (while vm2 still has it)
BadRequestException: 400: Client Error for url: http://10.0.2.15/compute/v2.1/servers/6cbc940d-46db-4ed8-812e-dc3f67c41966/os-volume_attachments, Invalid volume: volume edb223a8-3e74-4be2-af0f-1abb102a885f is already attached to instances: 1d6eb8e6-08a2-4482-b...
$ openstack server delete --wait vm2        (with the volume still mounted in it)
$ openstack volume list -c Name -c Status -c "Attached to"
+-------+-----------+-------------+
| Name  | Status    | Attached to |
+-------+-----------+-------------+
| data1 | available |             |
+-------+-----------+-------------+
$ openstack server add volume vm1 data1 -c Device -f value
/dev/vdb
$ (inside vm1) sudo mount /dev/vdb /mnt/data1 && sudo cat /mnt/data1/hello.txt
written on a volume at 23:39:06
$ (inside vm1) sudo dmesg | grep -iE "ext4|recover" | tail -3
[   89.320960] EXT4-fs (vdb): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[  246.715323] EXT4-fs (vdb): recovery complete
[  246.715337] EXT4-fs (vdb): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.

After server remove volume, the volume is available and disappears from the guest (the partition list is back to vda only).
Attached to a second server, vm2, the file is there: hello.txt and lost+found. The data lives in the volume, not in any server.
One volume cannot be attached to two servers at once (by default): 400 ... is already attached to instances: .... A volume type with multi-attach exists for special setups with cluster filesystems (not run here).
Deleting a server does not delete its attached volumes. vm2 was deleted with the volume still mounted inside it, and the volume came back available with its data. But look at what the next server's kernel said when it mounted it: EXT4-fs (vdb): recovery complete. The filesystem had been cut off in the middle, so ext4 had to replay its journal. The data was fine this time; the lesson is: unmount before you detach or delete.

Thin provisioning: promising more than exists

The volume is 1 GB, but only what you write takes space in the pool. Writing 200 MB in the guest:

$ sudo lvs -o lv_name,lv_size,data_percent
  LV                                          LSize Data%
  stack-volumes-lvmdriver-1-pool              9.50g 0.50
  volume-edb223a8-3e74-4be2-af0f-1abb102a885f 1.00g 4.79
$ (inside vm1) sudo dd if=/dev/zero of=/mnt/data1/big.bin bs=1M count=200; sudo sync; df -h /mnt/data1 | tail -1
200+0 records out
/dev/vdb                973.4M    200.0M    706.2M  22% /mnt/data1
$ sudo lvs -o lv_name,lv_size,data_percent
  LV                                          LSize Data%
  stack-volumes-lvmdriver-1-pool              9.50g 2.56
  volume-edb223a8-3e74-4be2-af0f-1abb102a885f 1.00g 24.32

The volume's usage went from 4.79% to 24.32% (200 MB of 1 GB, plus the filesystem), and the pool from 0.50% to 2.56%. That is how a 9.5 GB pool can hold many volumes. And it can promise far more than it has: here a 20 GB volume was created on a pool of 9.5 GB, and it is available, while a 300 GB volume went to error. The administrator's view of the pool explains both:

$ OS_CLOUD=devstack-admin openstack volume backend pool list --long -f json      (a few of the capabilities)
total_capacity_gb            9.5
free_capacity_gb             9.01
allocated_capacity_gb        6
provisioned_capacity_gb      7.0
max_over_subscription_ratio  20.0
thin_provisioning_support    True
thick_provisioning_support   False
location_info                LVMVolumeDriver:cloud1:stack-volumes-lvmdriver-1:thin:0
$ openstack volume create --size 20 huge        (more than the whole 9.5 GB pool)
$ openstack volume show huge -c status -c size -f yaml
size: 20
status: available
$ sudo lvs -o lv_name,lv_size,data_percent | grep -E "LV|volume-495e"
  LV                                             LSize  Data%
  volume-495e556d-819f-4ebb-893a-76a96e7e6cd9    20.00g 0.00
$ sudo vgs --units g -o vg_name,vg_size,vg_free
  VG                        VSize  VFree
  stack-volumes-lvmdriver-1 10.00g 0.47g
$ openstack volume create --size 300 enormous
$ openstack volume show enormous -c status -c size -f yaml
size: 300
status: error
$ journalctl -u devstack@c-sch | grep -E "Filter .* returned|Filtering removed"
DEBUG cinder.scheduler.base_filter [None req-8ccd665f-... demo None] Filter AvailabilityZoneFilter returned 1 host(s) {{...}}
DEBUG cinder.scheduler.base_filter [None req-8ccd665f-... demo None] Filter CapacityFilter returned 0 host(s) {{...}}
DEBUG cinder.scheduler.base_filter [None req-8ccd665f-... demo None] Filter CapabilitiesFilter returned 0 host(s) {{...}}
INFO cinder.scheduler.base_filter [None req-8ccd665f-... demo None] Filtering removed all hosts for the request with volume ID 'a7fc1d70-b9cf-4990-ae9d-1fa4ec82c8fe'. Filter results: AvailabilityZoneFilter: (start: 1, end: 1), ...

The pool has 9.5 GB real, of which 6 GB are already allocated to volumes, and it allows over-subscription up to a ratio of 20. By that arithmetic the scheduler is willing to promise about 190 GB in total (9.5 times 20; the exact edge was not tested): 20 GB fits, 300 GB does not, and the scheduler's log says why: Filter CapacityFilter returned 0 host(s), then "Filtering removed all hosts for the request". This is Cinder's version of Nova's "No valid host was found", and you read it the same way: the volume is in error, the request was refused by the scheduler's filters, and the reason is in c-sch's log (which you can only read on the cloud's own machines).

The danger in thin provisioning is the day the promises are called in. If guests actually write more than the pool physically holds, the pool fills up and writes to the volumes in it start to fail, for everybody, at the same moment. (That was not provoked here.) So with thin pools: watch the pool's real usage (lvs shows it, a monitoring system should), not the sum of the volume sizes.

Snapshots, and volumes made from snapshots

A snapshot freezes a volume's contents at one moment. On an attached volume it needs --force, because the filesystem may be half-written:

$ openstack volume snapshot create --volume data1 snap1
BadRequestException: 400: Client Error for url: http://10.0.2.15/volume/v3/snapshots, Invalid volume: Volume edb223a8-3e74-4be2-af0f-1abb102a885f status must be available, but current status is: in-use.
$ openstack volume snapshot create --volume data1 --force snap1 -c name -c status -c size -c volume_id -f yaml
name: snap1
size: 1
status: creating
volume_id: edb223a8-3e74-4be2-af0f-1abb102a885f
$ openstack volume snapshot list
+--------------------------------------+-------+-------------+-----------+------+
| ID                                   | Name  | Description | Status    | Size |
+--------------------------------------+-------+-------------+-----------+------+
| 3e43a0b9-cf23-441c-ab78-83dc7ba57d6e | snap1 | None        | available |    1 |
+--------------------------------------+-------+-------------+-----------+------+
$ sudo lvs -o lv_name,lv_size,lv_attr,data_percent,origin
  LV                                             LSize Attr       Data%  Origin
  _snapshot-3e43a0b9-cf23-441c-ab78-83dc7ba57d6e 1.00g Vwi---tz-k        volume-edb223a8-3e74-4be2-af0f-1abb102a885f
  stack-volumes-lvmdriver-1-pool                 9.50g twi-aotz-- 2.56
  volume-edb223a8-3e74-4be2-af0f-1abb102a885f    1.00g Vwi-aotz-- 24.32
$ (inside vm1) echo "written AFTER the snapshot" | sudo tee /mnt/data1/later.txt; sudo sync; ls /mnt/data1
written AFTER the snapshot
big.bin
hello.txt
later.txt
lost+found
$ openstack volume create --snapshot snap1 --size 1 data1-copy -c name -c size -c snapshot_id -c status -f yaml
name: data1-copy
size: 1
snapshot_id: 3e43a0b9-cf23-441c-ab78-83dc7ba57d6e
status: creating
$ openstack server add volume vm1 data1-copy -c Device -f value
/dev/vdc
$ (inside vm1) sudo mkdir -p /mnt/copy && sudo mount /dev/vdc /mnt/copy && ls -la /mnt/copy; sudo cat /mnt/copy/hello.txt; ls /mnt/copy/later.txt
total 204828
drwxr-xr-x    3 root     root          4096 Oct  9 00:42 .
drwxr-xr-x    4 root     root          4096 Oct  9 00:42 ..
-rw-------    1 root     root     209715200 Oct  9 00:42 big.bin
-rw-------    1 root     root            32 Oct  9 00:39 hello.txt
drwx------    2 root     root         16384 Oct  9 00:39 lost+found
written on a volume at 23:39:06
ls: /mnt/copy/later.txt: No such file or directory
$ sudo lvs -o lv_name,lv_size,lv_attr,data_percent,origin
  LV                                             LSize Attr       Data%  Origin
  _snapshot-3e43a0b9-cf23-441c-ab78-83dc7ba57d6e 1.00g Vwi-a-tz-k 24.32  volume-edb223a8-3e74-4be2-af0f-1abb102a885f
  stack-volumes-lvmdriver-1-pool                 9.50g twi-aotz-- 2.57
  volume-8b9dd6d4-7cd8-41a4-a0d6-1c27133e95f4    1.00g Vwi-aotz-- 24.32  _snapshot-3e43a0b9-cf23-441c-ab78-83dc7ba57d6e
  volume-edb223a8-3e74-4be2-af0f-1abb102a885f    1.00g Vwi-aotz-- 24.32

Without --force: 400 ... status must be available, but current status is: in-use. With it, the snapshot goes creating, then available, within a few seconds.
On the storage side the snapshot is another thin logical volume (_snapshot-<id>), whose origin is the volume. It costs almost nothing at first, and grows as the original changes.
A new volume created from the snapshot (volume create --snapshot snap1 --size 1 data1-copy) is a writable copy, attached like any other (it showed up as /dev/vdc). It has hello.txt and big.bin as they were, and does not have later.txt, which was written after the snapshot. The copy's own LV shows the snapshot as its origin.
--force gives you a crash-consistent snapshot, as if the power had failed at that moment: good enough for a filesystem with a journal, not good enough for a busy database. For those, stop the writer or flush first (not tested here).

⚠️ A snapshot is not a backup. It lives in the same pool, on the same storage, so if the pool dies, the snapshots die with the volume. The lvs lines above show it plainly: both are inside stack-volumes-lvmdriver-1. A backup goes somewhere else (see the backups page), and the Cinder backup service is the next section's story.

Growing a volume

Volumes grow, they do not shrink. The first attempt on an attached volume:

$ openstack volume set --size 2 data1
Failed to set volume size: --os-volume-api-version 3.42 or greater is required to extend in-use volumes.
One or more of the set operations failed
$ openstack volume set --size 1 data1            (a size that is not bigger than the current 1 GB)
Failed to set volume size: New size must be greater than 1 GB
One or more of the set operations failed
$ openstack --os-volume-api-version 3.42 volume set --size 2 data1
$ openstack volume list -c Name -c Status -c Size
+------------+--------+------+
| Name       | Status | Size |
+------------+--------+------+
| data1-copy | in-use |    1 |
| data1      | in-use |    2 |
+------------+--------+------+
$ sudo lvs -o lv_name,lv_size,data_percent | grep -E "LV|volume-edb"
  LV                                             LSize Data%
  volume-edb223a8-3e74-4be2-af0f-1abb102a885f    2.00g 12.16
$ (inside vm1) cat /proc/partitions | grep vdb; df -h /mnt/data1 | tail -1
 252       16    2097152 vdb
/dev/vdb                973.4M    200.0M    706.2M  22% /mnt/data1
$ (inside vm1) sudo resize2fs /dev/vdb; df -h /mnt/data1 | tail -1
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/vdb is now 524288 (4k) blocks long.

/dev/vdb                  1.9G    200.0M      1.6G  11% /mnt/data1

By default the client refuses: "--os-volume-api-version 3.42 or greater is required to extend in-use volumes." The message names the cure: ask for the newer API version for that command (--os-volume-api-version 3.42), and it works while the volume is attached and mounted.
Inside the running guest, /proc/partitions showed the new size at once (2097152 blocks, 2 GB) without a reboot or a rescan. The filesystem did not grow by itself, so resize2fs /dev/vdb did it, online: 1.9 GB available. Two steps: the cloud grows the disk, you grow the filesystem.
Shrinking is not possible: a size that is not bigger than the current one is refused ("New size must be greater than 1 GB"). To make a volume smaller, create a smaller one and copy the data.

Boot from a volume

So far the root disk was a file on the compute host, tied to the server. You can instead make the root disk a volume, which can be snapshotted, grown and moved like the others:

openstack server create --flavor cirros256 --image cirros-0.6.3-x86_64-disk \
    --boot-from-volume 2 --network private --key-name mykey --security-group ssh-ping --wait bfv

$ openstack server create --flavor cirros256 --image cirros-0.6.3-x86_64-disk --boot-from-volume 2 --network private --key-name mykey --security-group ssh-ping --wait bfv -c status -c image -c volumes_attached -f yaml
image: N/A (booted from volume)
status: ACTIVE
volumes_attached:
- delete_on_termination: false
  id: 62155e13-9581-4d2d-aac9-238c17815511
$ openstack volume list
+--------------------------------------+------------+--------+------+------------------------------+
| ID                                   | Name       | Status | Size | Attached to                  |
+--------------------------------------+------------+--------+------+------------------------------+
| 62155e13-9581-4d2d-aac9-238c17815511 |            | in-use |    2 | Attached to bfv on /dev/vda  |
| 8b9dd6d4-7cd8-41a4-a0d6-1c27133e95f4 | data1-copy | in-use |    1 | Attached to vm1 on /dev/vdc  |
| edb223a8-3e74-4be2-af0f-1abb102a885f | data1      | in-use |    1 | Attached to vm1 on /dev/vdb  |
+--------------------------------------+------------+--------+------+------------------------------+
$ openstack volume show 62155e13-9581-4d2d-aac9-238c17815511 -c bootable -c status -c size -c volume_image_metadata -f yaml | head -9
bootable: true
size: 2
status: in-use
volume_image_metadata:
  checksum: 87617e24a5e30cb3b87fda8c0764838f
  container_format: bare
  disk_format: qcow2
  hw_rng_model: virtio
  image_id: 0a909d81-1440-4430-a266-930399e6e8b5
$ (inside bfv) cat /proc/partitions; df -h / | tail -1
major minor  #blocks  name

 252        0    2097152 vda
 252        1    2087919 vda1
 252       15       8192 vda15
/dev/vda1                 1.9G     43.9M      1.8G   2% /
$ sudo virsh domblklist <the instance>
 Target   Source
--------------------
 vda      /dev/sdc
$ openstack server delete --wait bfv
$ openstack volume list
+--------------------------------------+------------+-----------+------+------------------------------+
| ID                                   | Name       | Status    | Size | Attached to                  |
+--------------------------------------+------------+-----------+------+------------------------------+
| 62155e13-9581-4d2d-aac9-238c17815511 |            | available |    2 |                              |
| 8b9dd6d4-7cd8-41a4-a0d6-1c27133e95f4 | data1-copy | in-use    |    1 | Attached to vm1 on /dev/vdc  |
| edb223a8-3e74-4be2-af0f-1abb102a885f | data1      | in-use    |    1 | Attached to vm1 on /dev/vdb  |
+--------------------------------------+------------+-----------+------+------------------------------+
$ openstack volume show 62155e13-9581-4d2d-aac9-238c17815511 -c status -c bootable -c attachments -f yaml
attachments: []
bootable: true
status: available

Cinder made a 2 GB volume from the image and attached it as /dev/vda. The server's image field reads N/A (booted from volume). The volume is bootable: true, and remembers which image it came from (volume_image_metadata).
Inside the guest the root disk is vda, 2 GB (the volume's size: the root partition grew to fill it, as it did for a flavor on the previous page), and on the compute host the libvirt disk is /dev/sdc, the iSCSI device, not a file.
The trap: delete_on_termination is false. Delete the server, and the volume stays (status available, still bootable, 2 GB). It is not an error: you may want to boot from it again. But it is a leftover that nobody reminds you of, and it counts against your quota. Check openstack volume list after you delete servers. (There are options to delete the boot volume together with the server; they were not tried here.)
The benefit: a server that can be rebuilt from its volume on another host, because the disk is not tied to one machine's files. That is why people use boot from volume for servers that matter.

A volume you did not make: the image cache

After deleting every volume of my own, lvs on the storage machine still showed one logical volume, 1 GB, 10.94% used, that openstack volume list as the demo user did not list. The administrator can see it:

$ openstack volume list                          (as the demo user, after deleting every volume of mine)
$ sudo lvs -o lv_name,lv_size,data_percent
  LV                                          LSize Data%
  stack-volumes-lvmdriver-1-pool              9.50g 1.15
  volume-05782988-3e56-49ec-98dc-06734d218de4 1.00g 10.94
$ OS_CLOUD=devstack-admin openstack volume list --all-projects -c ID -c Name -c Status -c Size
+--------------------------------------+--------------------------------------------+-----------+------+
| ID                                   | Name                                       | Status    | Size |
+--------------------------------------+--------------------------------------------+-----------+------+
| 05782988-3e56-49ec-98dc-06734d218de4 | image-0a909d81-1440-4430-a266-930399e6e8b5 | available |    1 |
+--------------------------------------+--------------------------------------------+-----------+------+
$ OS_CLOUD=devstack-admin openstack volume show 05782988-3e56-49ec-98dc-06734d218de4 -c name -c os-vol-tenant-attr:tenant_id -f yaml
name: image-0a909d81-1440-4430-a266-930399e6e8b5
os-vol-tenant-attr:tenant_id: f9316cdb10da402a96f49f35809b3b74
$ grep -E "^(image_volume_cache|cinder_internal_tenant)" /etc/cinder/cinder.conf
cinder_internal_tenant_user_id = 071242dd73e5490a880efdf25ad69efc
cinder_internal_tenant_project_id = f9316cdb10da402a96f49f35809b3b74
image_volume_cache_enabled = True

It is the image-volume cache: with image_volume_cache_enabled = True, Cinder keeps a ready-made volume of a popular image (image-<image ID>) so the next boot-from-volume can clone it instead of copying the image again. It belongs to an internal project (service, the project ID in cinder_internal_tenant_project_id), so ordinary users never see it, and it is not a leak. Just remember that the storage can hold more than the sum of what you can list.

What is not installed: backups and object storage

A cloud has the services its administrator installed. This DevStack has no backup service (cinder-backup) and no Swift. The first one is worth a close look, because the API accepted the request anyway:

$ openstack volume backup create --name bk1 data1-copy
+-----------+--------------------------------------+
| Field     | Value                                |
+-----------+--------------------------------------+
| id        | 782ff2a2-f825-4aa9-9b61-e13f2f236e44 |
| name      | bk1                                  |
| volume_id | 8b9dd6d4-7cd8-41a4-a0d6-1c27133e95f4 |
+-----------+--------------------------------------+
$ openstack volume backup list -c Name -c Status -c Size
+------+--------+------+
| Name | Status | Size |
+------+--------+------+
| bk1  | error  |    1 |
+------+--------+------+
$ openstack volume backup show bk1 -c status -c fail_reason -f yaml
fail_reason: Service not found for creating backup.
status: error
$ openstack volume backup delete bk1
Failed to delete backup with name or ID 'bk1': NotFoundException: 404: Client Error for url: http://10.0.2.15/volume/v3/backups/782ff2a2-f825-4aa9-9b61-e13f2f236e44, Service cinder-backup could not be...
$ OS_CLOUD=devstack-admin openstack volume backup delete --force 782ff2a2-f825-4aa9-9b61-e13f2f236e44
Failed to delete backup with name or ID '782ff2a2-...': HttpException: 500: Server Error for url: http://10.0.2.15/volume/v3/backups/782ff2a2-.../action, The server has eit...
$ systemctl is-active devstack@c-bak
inactive
$ sudo systemctl start devstack@c-bak
Failed to start devstack@c-bak.service: Unit devstack@c-bak.service not found.
$ OS_CLOUD=devstack-admin openstack volume service list -c Binary -c Host -c State
+------------------+--------------------+-------+
| Binary           | Host               | State |
+------------------+--------------------+-------+
| cinder-scheduler | cloud1             | up    |
| cinder-volume    | cloud1@lvmdriver-1 | up    |
+------------------+--------------------+-------+

volume backup create returned an ID and a table, as if it had worked. The record then went to error with Service not found for creating backup.
You cannot delete it. The user's delete says the backup service could not be found; the administrator's --force fails with a 500, and backup set --state available with a 404. Every road leads through the service that does not exist.
The administrator's service list shows why: only cinder-scheduler and cinder-volume. And there is no devstack@c-bak unit to start, because the backup service was never installed (going by DevStack's service naming, it would take enable_service c-bak in local.conf and a new stack.sh; not tried here).
To keep the lab tidy, the orphaned record was removed directly in the database, and the usage counter that still said "1 backup" was fixed with cinder-manage quota sync (an administrator command: it printed backups: invalid usage saved=1 actual=0 - fixed). Do not edit the database of a real cloud for this; start the backup service and delete the record through the API.
The lesson: before you rely on a feature, ask the cloud whether it has it. As an administrator, openstack volume service list shows which Cinder services run; as a user, openstack catalog list shows which services exist at all.

Errors, in their real words

$ openstack volume delete data1             (while attached)
Failed to delete volume with name or ID 'data1': BadRequestException: 400: Client Error for url: http://10.0.2.15/volume/v3/volumes/edb223a8-3e74-4be2-af0f-1abb102a885f?cascade=False&force=False, Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots, awaiting a transfer, or be disassociated from snapshots after volume transfer.
$ openstack volume delete data1             (detached, but it still has a snapshot: the same message)
Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots, awaiting a transfer, or be disassociated from snapshots after volume transfer.
1 of 1 volumes failed to delete.
$ openstack volume delete --purge data1       (delete the snapshots with it)
$ openstack volume list -c Name -c Status -c Size
+------------+-----------+------+
| Name       | Status    | Size |
+------------+-----------+------+
|            | available |    2 |
| data1-copy | available |    1 |
+------------+-----------+------+
$ openstack volume snapshot list
$ openstack server add volume vm1 nosuchvol
No Volume found for nosuchvol

One message, many causes. The delete error lists every condition at once: the volume must be available or in an error state, and must not be migrating, attached, in a group, have snapshots, awaiting a transfer... Whichever applies to you, you get the same sentence. Check status and attachments, then openstack volume snapshot list.
--purge deletes the volume together with its snapshots. Careful: volumes made from those snapshots stay (here data1-copy survived), which was the reason to test it.
No Volume found for nosuchvol is a typo: names must match, and two volumes may even have the same name. Use the ID when in doubt.

Quota:

$ openstack limits show --absolute -c Name -c Value | grep -E "total_volumes_used|max_total_volumes |total_gigabytes_used"
| max_total_volumes           |    10 |
| total_gigabytes_used        |     6 |
| total_volumes_used          |     3 |
$ OS_CLOUD=devstack-admin openstack quota set --volumes 3 demo
$ openstack volume create --size 1 one-too-many
HttpException: 413: Client Error for url: http://10.0.2.15/volume/v3/volumes, VolumeLimitExceeded: Maximum number of volumes allowed (3) exceeded for quota 'volumes'.
$ OS_CLOUD=devstack-admin openstack quota set --volumes 10 demo

413 VolumeLimitExceeded: Maximum number of volumes allowed (3) exceeded for quota 'volumes': the project is at its limit (here an administrator lowered it for the test and restored it). Note the status code is 413 where Nova's quota error was 403: the services are written by different teams.

Cleaning up

Delete in this order: servers (they free the attachments), then volumes (after their snapshots, or with --purge), then the rest (floating IPs, groups, key pairs). Then check both sides:

$ sudo cinder-manage quota sync                 (a stuck usage counter, after the backup record was removed)
	backups: invalid usage saved=1 actual=0 - fixed
	backup_gigabytes: invalid usage saved=1 actual=0 - fixed
Processing quota usage for project f9316cdb10da402a96f49f35809b3b74
Action successfully completed
$ openstack limits show --absolute -c Name -c Value | grep -E "total_(volumes|gigabytes|snapshots|backups)_used"
| total_backups_used          |     0 |
| total_gigabytes_used        |     0 |
| total_snapshots_used        |     0 |
| total_volumes_used          |     0 |
$ sudo iscsiadm -m session
iscsiadm: No active sessions.
$ sudo targetcli ls /iscsi
o- iscsi ......................................................... [Targets: 0]

All the quota counters are back to 0, no iSCSI session is left on the compute host, and the target list is empty. The one logical volume that remains is the image cache, by design.

The pitfalls, in one list

A new volume is an empty, unformatted disk. mkfs it, once: formatting an attached volume that already holds data destroys it.
Unmount before detaching or deleting the server. The data may survive, but you will see recovery complete, and with some filesystems you will not be so lucky.
Device names change. Mount by label or UUID.
A snapshot is in the same pool as its volume. It is not a backup.
Thin provisioning promises more than exists. Watch the pool's real usage.
Boot volumes outlive their servers unless told otherwise. List your volumes after deleting servers.
Check that the service exists (backups, object storage) before you build on it.
Mind the quota counters and the leftovers (volume list, snapshot list, backup list): they all count.

What this page did not cover

Real storage backends (Ceph, arrays, NFS), volume types with different properties (and retype), encrypted volumes, multi-attach, QoS.
Backups that work (the service was not installed), volume transfer between projects, migration between backends.
Object storage (Swift) and shared file systems (Manila), not installed here.
Making a pool run out of space, and what it looks like (not provoked).
Delete-on-termination options for boot volumes, and cinder in Horizon.

The cheat sheet

openstack volume type list                                 the kinds of storage on offer
openstack volume create --size 10 NAME                     a new, empty volume
openstack volume list / show NAME                          status, size, attachments
openstack server add volume SERVER VOLUME                  attach (hot): gives a /dev/vdX
openstack server remove volume SERVER VOLUME               detach (unmount it first!)
blkid /dev/vdb ; mkfs.ext4 -L LABEL /dev/vdb ; mount ...   inside the guest: format ONCE, mount
openstack volume snapshot create --volume V [--force] S    snapshot (force = attached)
openstack volume create --snapshot S --size 10 NAME        a writable copy from a snapshot
openstack --os-volume-api-version 3.42 volume set --size 20 NAME     grow (even attached)
resize2fs /dev/vdb                                         inside the guest: grow the filesystem
openstack server create ... --boot-from-volume 20 NAME     root disk on a volume
openstack volume delete [--purge] NAME                     --purge also deletes its snapshots
openstack volume snapshot list / backup list               what else is counting against you
openstack limits show --absolute | grep total_             quotas and current usage
openstack volume service list                              (admin) which Cinder services run
openstack volume backend pool list --long                  (admin) pool capacity and the over-subscription ratio
sudo lvs -o lv_name,lv_size,lv_attr,data_percent           (DevStack machine) the thin volumes and the pool
journalctl -u devstack@c-sch                               why a volume went to error

← Your First OpenStack Instance: Key Pair, Security Group, Floating IP and the Mistakes Everybody Makes · ↑ All tutorials · Your Own Networks on OpenStack: Subnets, Routers, Ports and Floating IPs (Neutron) →