Rolling Restart using OpenStack-Ansible

Sometimes it's useful to do a rolling restart of OpenStack services throughout the cloud. In OpenStack-Ansible we've the means to make this easy, you just need a little BASH to tie it all together. In this little aside, I'm restarting the entire cloud using the OpenStack-Ansible inventory.

Listing possible services

To get a list of all of the services I could simply take all of the install playbooks as a list using something like this:

ls -1 *.yml | awk -F'-' '/os-/ {print $2}' | uniq

However, I don't want to simply take the order in which the files are found as a means to restart the environment. I want to run this operation in a specific order to lessen the impact seen within the environment.

Bashing the array

To make this a more controlled restart I will be building a simple bash variable with the service name. I will then iterate over the list and restart all discovered processes on the target nodes.

RESTART_ORDER=$(awk '/include:/ {print $3}' setup-openstack.yml | uniq | awk -F'-' '{print $2}')

For completeness, I added all services found in the openstack-setup.yml playbook, however, this list would likely be shorter and could have other, non-standard, services. Customize this list to suit your needs.

Iterating my way to success

Tthe loop I used to iterate over the list will restart all discovered services on the target nodes within the /etc/init and /etc/init.d directories. After the services are restarted the command will wait 3 seconds before continuing the rest of the loop. The anisble command will also limit the forks to 2 which will ensure we're not restarting everything at the same time within a given group. Customize the sleep time and forks to meet your needs and level of comfort.

for item in ${RESTART_ORDER}; do
  ansible -m shell -a "(systemctl list-unit-files --state=enabled --type=service || ls -1 /etc/init/ /etc/init.d/) | grep -e \"${item}\" -e apache -e nginx  | awk -F'.' '{print \$1}' | xargs -I{} -n 1 service {} restart && sleep 3" "${service_group}_all" --forks 2
done

With this simple loop, we're able to do a rolling restart across an entire OpenStack cloud without having to run a whole host of ad-hoc commands all using the OpenStack-Ansible inventory.

That was easy!