@portfast

News & views, serious business.

Friday 2 February 2018 - Reboots and OOB console functionality

We recently undertook a reboot of quite a few of our longest lived virtual machines, as a change in the way the kernel handles certain structures internal to our chosen virtualisation technology meant that we couldn't live migrate these to a new host to perform maintenance.

As a bit of background to this, we make extensive use of live migration to peform maintenance on our virtualisation platform without having any customer impact. When we want to do something to a host that involves downtime, we can move all the running virtual machines off it seamlessly with almost no downtime, only a few seconds while the layer 3 route update propagates in our core.

Unfortunately what sometimes happens when one has, and we are going to blow our own trumpets here, virtual machines capable of several year uptimes is that things like boot loaders can be broken by software updates and this can go unnoticed for long periods. With this in mind, we have introduced a feature to the out of band (OOB) console which allows you to boot the VM from a kernel file held on the host, bypassing the inbuilt boot loader on the machine and allowing you to get it started so that the boot loader can be reinstated properly.

We have put together some documentation for this feature here.