We just learned that our VM provider, Linode, had to perform some emergency reboots. Three of our servers have already been taken care of, but more are still to come. This led to our site being unavailable for a period of approximately an hour. Here is the reboot schedule:
Identified - Linode has received a Xen Security Advisory (XSA) that requires us to perform updates to our legacy Xen host servers. In order to apply the updates, hosts and the Linodes running on them must be rebooted. The XSAs will be publicly released by the Xen project team on July 26th. We must complete this maintenance before then.
Here's the schedule and status:
|Server||Purpose||Maintenance Schedule (UTC)|
We apologize for any inconvenience.
[Update: It appears the second round of reboots has completed successfully, and, thanks to the advance notice, the site stayed up throughout. We anticipate that the site will still continue to operate normally through the last-scheduled reboot. Many thanks for your understanding and patience.]
[Update #2: We are taking advantage of a free offer from Linode, our hosting provider, to convert our VPSs (Virtual Private Servers) from Xen to KVM. The rebooting was required to repair a Xen vulnerability. As a bonus, the Xen to KVM conversion gives us a free upgrade to twice as much memory. The additional memory will provide much-needed additional headroom on our servers and possibly provide a performance improvement. Thanks to our redundancy, the changes should not be noticeable when we reboot/upgrade, except for the IRC and e-mail servers as they are single-hosted.]
[Update #3: Thanks to the tireless efforts of paulej72 well into the wee hours of this morning with able assistance by audioguy in straightening out some IP issues as well as Deucalion and TheMightyBuzzard providing guidance and support, all but two of our Xen servers have been upgraded to KVM. This free upgrade doubled the amount of memory available to our VMs, giving us some much-needed headroom. That leaves beryllium (IRC and email) and boron (DNS, Hesiod name service) as the two servers that have not been upgraded yet. Date/time is TBD.]
[Update #4: Boron will be reconfigured shortly, and then Bberyllium after that. Plan on an hour or two, though, obviously, we'll try to keep the downtime to a minimum!]
[Update #5: Boron's second upgrade for the ram sat in the queue for several hours, so Beryllium had to wait until paulej72 got up and finished it this morning (0830 EDT)]