Nova-Compute + LVM, an IOPS Love Story

The OpenStack Nova project has long had the ability to provision instances into logical volumes however in my travels I've found most deployments don't use it as the go-to instance storage. However recently I've been asked by a couple folks how it can be done with OpenStack-Ansible.

This post will will simply show how I would configure OpenStack-Ansible to deploy Nova-Compute using logical volumes for instances storage and a little bit about why this might be desirable.

The setup

This post assumes a suitable volume group is available on the host. For me, this will be vg01.

compute1:~# vgdisplay vg01
  --- Volume group ---
  VG Name               vg01
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  19
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                6
  Open LV               6
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               1.82 TiB
  PE Size               4.00 MiB
  Total PE              476931
  Alloc PE / Size       98306 / 384.01 GiB
  Free  PE / Size       378625 / 1.44 TiB
  VG UUID               B2zktP-Sw96-TUlJ-H2vx-0Vuf-cu3d-p3mpsj

The hardware overview I have running within this environment can be seen here.

Configuring the Cloud

Add a configuration override for nova into the user_variables.yml file.

nova_nova_conf_overrides:
  ephemeral_storage_encryption:
    enabled: True
  libvirt:
    images_type: lvm
    images_volume_group: vg01
    volume_clear: zero
    disk_cachemodes: none

In this example I'm using the OpenStack-Ansible configuration template capabilities to inject additional config into the nova.conf. This set of overrides will enable encrypted ephemeral disks, set the instance image type to lvm, define the volume group to deploy into and how logical volume slices, define the cleaned method when an instance is destroyed and set the disk cache mode so that we're able to take advantage of native IO. More of the possible Nova configuration options can be seen here.

Deploying the configuration

  • If the deployment is a new one, simply run the playbooks normally once the override is in place.

  • If this is an existing deployment we can reconfigure nova accordingly using the playbooks limits and tags.

openstack-ansible os-nova-install.yml --tags nova-config --limit nova_compute

After the playbooks have completed the required configuration will be in place and the nova-compute service will have been restarted.

Making sure it all works

Provision an instance to the reconfigured compute node(s). Once the instance is active, you can log into the compute host and run lvs which will show logical volumes using an instance UUID as it's base name.

Use the lvdisplay command to inspect the online volumes.

compute1:~# lvdisplay /dev/vg01/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX*
  --- Logical volume ---                                                                                                                                               
  LV Path                /dev/vg01/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX_disk                                                                                           
  LV Name                XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX_disk
  VG Name                vg01
  LV UUID                XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
  LV Write Access        read/write
  LV Creation host, time compute1, 2017-12-15 00:58:51 -0600
  LV Status              available
  # open                 2
  LV Size                128.00 GiB
  Current LE             32768
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:3
   
  --- Logical volume ---
  LV Path                /dev/vg01/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX_disk.eph0
  LV Name                XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX_disk.eph0
  VG Name                vg01
  LV UUID                XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
  LV Write Access        read/write
  LV Creation host, time compute1, 2017-12-15 00:59:06 -0600
  LV Status              available
  # open                 2
  LV Size                64.00 GiB
  Current LE             16384
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:4
   
  --- Logical volume ---
  LV Path                /dev/vg01/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX_disk.swap
  LV Name                XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX_disk.swap
  VG Name                vg01
  LV UUID                XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
  LV Write Access        read/write
  LV Creation host, time compute1, 2017-12-15 00:59:07 -0600
  LV Status              available
  # open                 2
  LV Size                4.00 MiB
  Current LE             1
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:6

As seen in the output, I have a swap disk, an ephemeral disk, and a root disk. Which corresponds to my running instance.

compute1:~# nova show XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX 
+--------------------------------------+----------------------------------------------------------+                                                                    
| Property                             | Value                                                    |                                                                    
+--------------------------------------+----------------------------------------------------------+                                                                    
| LOCAL_NET network                    | 172.17.24.105                                            |                                                                    
| OS-DCF:diskConfig                    | AUTO                                                     |                                                                    
| OS-EXT-AZ:availability_zone          | nova                                                     |                                                                    
| OS-EXT-SRV-ATTR:host                 | compute1                                                 |                                                                    
| OS-EXT-SRV-ATTR:hostname             | rpc-test-1                                               |                                                                    
| OS-EXT-SRV-ATTR:hypervisor_hostname  | compute1.rk-home                                         |
| OS-EXT-SRV-ATTR:instance_name        | instance-0000000b                                        |
| OS-EXT-SRV-ATTR:kernel_id            |                                                          |
| OS-EXT-SRV-ATTR:launch_index         | 0                                                        |
| OS-EXT-SRV-ATTR:ramdisk_id           |                                                          |
| OS-EXT-SRV-ATTR:reservation_id       | r-cwvc02ta                                               |
| OS-EXT-SRV-ATTR:root_device_name     | /dev/sda                                                 |
| OS-EXT-SRV-ATTR:user_data            | -                                                        |
| OS-EXT-STS:power_state               | 1                                                        |
| OS-EXT-STS:task_state                | -                                                        |
| OS-EXT-STS:vm_state                  | active                                                   |
| OS-SRV-USG:launched_at               | 2017-12-15T06:59:19.000000                               |
| OS-SRV-USG:terminated_at             | -                                                        |
| accessIPv4                           |                                                          |
| accessIPv6                           |                                                          |
| config_drive                         |                                                          |
| created                              | 2017-12-15T06:58:44Z                                     |
| description                          | -                                                        |
| flavor:disk                          | 128                                                      |
| flavor:ephemeral                     | 64                                                       |
| flavor:extra_specs                   | {}                                                       |
| flavor:original_name                 | m1.medium                                                |
| flavor:ram                           | 12288                                                    |
| flavor:swap                          | 4                                                        |
| flavor:vcpus                         | 8                                                        |
| hostId                               | fd8e91a39068e6eef6f902c1f325ed56dbcbf492b97d08c4d469dcec |
| host_status                          | UP                                                       |
| id                                   | XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX                     |
| image                                | Ubuntu 16.04 (YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY)      |
| key_name                             | cloudnull                                                |
| locked                               | False                                                    |
| metadata                             | {}                                                       |
| name                                 | rpc-test-1                                               |
| os-extended-volumes:volumes_attached | []                                                       |
| progress                             | 0                                                        |
| security_groups                      | default                                                  |
| status                               | ACTIVE                                                   |
| tags                                 | []                                                       |
| tenant_id                            | 7ae65c813bb049eaab926a87a569d299                         |
| updated                              | 2017-12-15T06:59:19Z                                     |
| user_id                              | 35354f3c389a475fb8a3517c7c260c97                         |
+--------------------------------------+----------------------------------------------------------+

Why you might care

While this is not a revolutionary post or an amazing revelation, it does show how easy it is to update a cloud using OpenStack-Ansible and articulates some of the flexibility & maturity OpenStack provides. As for why you should care, running instances in logical volumes can improve performance in terms of both IOPS and Latency, is very easy to manage, and is damn stable.


Great Success!

I hope this shows how simple it can be to make seemingly complex changes to an OpenStack-Ansible powered cloud. If you've applied this solution to your environment, for test or production, let me know how it works and I hope you enjoy the native IO :).