Cloudy NixOS (OpenStack)

Recently, I decided to go down the journey of running things with NixOS. While I started with my laptop, I naturally decided to make my workspace problems part of my cloud adventure. So I figured I'd share my adventure along with my initial process for getting NixOS into the cloud. In this post, I've provided some of the commands I ran and a link to the repository I'm using to setup my OpenStack images on a Genestack cloud.

Why?

There were a couple of places on the internet that highlighted how to generate NixOS images using a generator via nixpkg. While OpenStack was an option, the generator didn't result in an image I could use as a general-purpose, standalone base image and didn't seem to incorporate many of the "features" I've come to expect in the cloud. So, to solve my cloudy wants with my new NixOS addiction, I set off to do a thing.

Image Requirements

After poking around, I decided I needed my NixOS-OpenStack to have the following attributes.

  • UEFI boot
  • Q35 machine type (compatible)
  • QEMU aware
  • Virtio support
  • Latest stable NixOS 23.11
  • Latest Kernel Image (6.8+)
  • Cloud-Init enabled
  • Standalone system

The Repository

To do what I needed, I put together a repository, which I've iterated on to craft the image I feel is well suited for the cloud and fits my immediate needs.

GitHub - cloudnull/nixos-openstack: Flake for OpenStack compatible NixOS images
Flake for OpenStack compatible NixOS images. Contribute to cloudnull/nixos-openstack development by creating an account on GitHub.

Git Repository for my OpenStack NixOS image

🚸
To make the above repo work, it assumes you have flakes and nix-command on your system.

Running the build within the Repository

To get started, clone the repository locally.

git clone https://github.com/cloudnull/nixos-openstack

Git clone for the flake repo

Create an OpenStack-compatible image for NixOS

nix build .#nixosConfigurations.build-qcow2.config.system.build.qcow2

Nix build command resulting in a QCOW2

Running the build with a remote repository

There's no need to clone the above repo to run the build; the following command can be used to run the build.

nix build git+https://github.com/cloudnull/nixos-openstack#nixosConfigurations.build-qcow2.config.system.build.qcow2

Verifying the Image

After the build, you will have a new qcow2 file in the results directory.

ll result/
total 630988
-r--r--r-- 1 root root 649986048 Dec 31  1969 nixos.qcow2
dr-xr-xr-x 2 root root      4096 Dec 31  1969 nix-support

Image result example

Upload the image and store the result in Glance

openstack --os-cloud default image create \
          --progress \
          --disk-format qcow2 \
          --container-format bare \
          --public \
          --file result/nixos.qcow2 \
          --property hw_scsi_model=virtio-scsi \
          --property hw_disk_bus=scsi \
          --property hw_vif_multiqueue_enabled=true \
          --property hw_qemu_guest_agent=yes \
          --property hypervisor_type=kvm \
          --property img_config_drive=optional \
          --property hw_machine_type=q35 \
          --property hw_firmware_type=uefi \
          --property os_require_quiesce=yes \
          --property os_type=linux \
          --property os_admin_user=nixos \
          --property os_distro=nixos \
          --property os_version=23.11 \
          nixos-23.11

Glance image upload command

The image output will look similar to this.

+——————+-————————————————————————————————————————————————————————————————————————+
| Field            | Value                                                                                                                                                                                                                   |
+——————+-————————————————————————————————————————————————————————————————————————+
| container_format | bare                                                                                                                                                                                                                    |
| created_at       | 2024-04-06T18:31:55Z                                                                                                                                                                                                    |
| disk_format      | qcow2                                                                                                                                                                                                                   |
| file             | /v2/images/f73588c5-468e-4678-9974-33300dce7c39/file                                                                                                                                                                    |
| id               | f73588c5-468e-4678-9974-33300dce7c39                                                                                                                                                                                    |
| min_disk         | 0                                                                                                                                                                                                                       |
| min_ram          | 0                                                                                                                                                                                                                       |
| name             | nixos-23.11                                                                                                                                                                                                             |
| owner            | 4d04429679c44b9ab3cafd523b9f86fd                                                                                                                                                                                        |
| properties       | hw_disk_bus='scsi', hw_firmware_type='uefi', hw_machine_type='q35', hw_qemu_guest_agent='yes', hw_scsi_model='virtio-scsi', hw_vif_multiqueue_enabled='True', hypervisor_type='kvm', img_config_drive='optional',       |
|                  | os_admin_user='nixos', os_distro='nixos', os_hidden='False', os_require_quiesce='True', os_type='linux', os_version='23.11', owner_specified.openstack.md5='', owner_specified.openstack.object='images/nixos-23.11',   |
|                  | owner_specified.openstack.sha256=''                                                                                                                                                                                     |
| protected        | False                                                                                                                                                                                                                   |
| schema           | /v2/schemas/image                                                                                                                                                                                                       |
| status           | queued                                                                                                                                                                                                                  |
| tags             |                                                                                                                                                                                                                         |
| updated_at       | 2024-04-06T18:31:55Z                                                                                                                                                                                                    |
| visibility       | public                                                                                                                                                                                                                  |
+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Glance Image output

The image we created with all the flags will enable the OpenStack features I need for the cloud: EFI boot, virtio, newer machine type, multi-queue networking, and image quiescing. This is made better by cloud-init, which will handle the systemd-networkd setup, hostnames, host keys, default users, SSH keys, etc. From my Nix noob point of view, the setup is allowing me to succeed in the cloud with NixOS.

💡
The metadata provided will ensure that OpenStack boots our image with the appropriate drivers and optimizations.

Boot your server

💡
Note the nic, flavor, and key-name will all be unique for your environment.

Creating the instance is simple and follows the standard OpenStack server create process.

openstack --os-cloud default server create \
          --image nixos-23.11 \
          --nic net-id=flat \
          --flavor m1.small \
          --key-name controller-0 \
          nixos-23.11

OpenStack Server create command example

The server creation will look similar to the following:

+-————————————+-———————————————————+
| Field                               | Value                                                    |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                   |
| OS-EXT-AZ:availability_zone         | nova                                                     |
| OS-EXT-SRV-ATTR:host                | compute-0.cloud.local                                    |
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute-0.cloud.local                                    |
| OS-EXT-SRV-ATTR:instance_name       | instance-0000004d                                        |
| OS-EXT-STS:power_state              | Running                                                  |
| OS-EXT-STS:task_state               | None                                                     |
| OS-EXT-STS:vm_state                 | active                                                   |
| OS-SRV-USG:launched_at              | 2024-04-06T18:32:18.000000                               |
| OS-SRV-USG:terminated_at            | None                                                     |
| accessIPv4                          |                                                          |
| accessIPv6                          |                                                          |
| addresses                           | flat=172.16.25.163                                       |
| adminPass                           | ************                                             |
| config_drive                        |                                                          |
| created                             | 2024-04-06T18:32:09Z                                     |
| flavor                              | m1.small (4ef01fb8-6afa-46f8-b20f-86cf60388791)          |
| hostId                              | afb64fd7a445a42c4dee560085e4dc4db3751f923ecd76f98b21c36e |
| id                                  | d212be0d-3ed0-4096-b763-b08a96fd575e                     |
| image                               | nixos-23.11 (f73588c5-468e-4678-9974-33300dce7c39)       |
| key_name                            | controller-0                                             |
| name                                | nixos-23.11                                              |
| progress                            | 0                                                        |
| project_id                          | 4d04429679c44b9ab3cafd523b9f86fd                         |
| properties                          |                                                          |
| security_groups                     | name='default'                                           |
| status                              | ACTIVE                                                   |
| updated                             | 2024-04-06T18:32:18Z                                     |
| user_id                             | 236cbdbe0eb545d68a21c58cb782c924                         |
| volumes_attached                    |                                                          |
+-------------------------------------+----------------------------------------------------------+

OpenStack CLI Output example

Creating and viewing the instance information in the OpenStack web UI is also possible.

OpenStack Skyline Web UI

After booting, all of the cloud-init bits will run, and we can log in via SSH with our defined keypair. The instance can also be accessed via the built-in serial console.

Console Output

Server and system logs are also available for the instance.

OpenStack Skyline Console Log

Hope this helps someone

While I'm sure I've made some mistakes, and there are ways this could be much better, it works! The following configuration generates a simple, fully featured NixOS virtual machine image (which could probably be used with Baremetal via Ironic, too... soon-tm).

The best part of this setup is that it works inside and outside of OpenStack. The image is general purpose enough to be used on most any KVM hypervisor effectively, which is what I was going after, even if my system of choice happens to be OpenStack 😉

More NixOS to come

Happy Nix'ing! More soon.