posted by
paulej72
on Sunday December 13 2015, @03:57PM
from the if-at-first-you-don't-sucseede dept.
from the if-at-first-you-don't-sucseede dept.
Hello fellow Soylentils!
We were informed by Linode (our hosting provider) that they needed to perform Xen updates on all of our servers. This forces a reboot of our virtual servers which may cause the site (and other services) to be temporarily unavailable.
Here is the two-day reboot schedule along with what runs on each server:
| Status | Day | Date | Time | Server | Affects |
|---|---|---|---|---|---|
| Done | Mon | 2015-12-14 | 0300 UTC | lithium | Development Server, slashd, Varnish, MySQL, Apache |
| Done | Mon | 2015-12-14 | 0400 UTC | helium | Production Back End, MySQL NDB, DNS, Hesoid, Kerberos |
| Done | Mon | 2015-12-14 | 0400 UTC | beryllium | IRC, MySQL, Postfix, Mailman, Yourls |
| Done | Mon | 2015-12-14 | 0400 UTC | sodium | Primary Load Balancer |
| Done | Mon | 2015-12-14 | 0400 UTC | magnesium | Backup Load Balancer |
| Done | Mon | 2015-12-14 | 0700 UTC | boron | DNS, Hesoid, Kerberos, Staff Slash |
| Done | Tue | 2015-12-15 | 0300 UTC | fluorine | Production Front End, slashd, Varnish, MySQL, Apache, ipnd |
| Done | Tue | 2015-12-15 | 0300 UTC | neon | Production Back End, MySQL NDB cluster |
| Done | Tue | 2015-12-15 | 0400 UTC | hydrogen | Production Front End, Varnish, MySQL, Apache, Sphinx |
As we found some errors with our MySql setup during the last Xen updates, The Main Site will be down starting at 2015-12-14 at 0300 UTC while we reconfigure MySQL prior to the shutdown of the main MySQL server helium. The site should be up by 0500 UTC.
We apologize in advance for any inconvenience and appreciate your understanding as we try and get things up and running following each reboot.
This discussion has been archived.
No new comments can be posted.
Server Reboots Again for Xen Updates
|
Log In/Create an Account
| Top
| 16 comments
| Search Discussion
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
(Score: 2) by Runaway1956 on Sunday December 13 2015, @04:02PM
http://www.amazon.com/Zen-Art-Systems-Analysis-Meditations/dp/0595256791 [amazon.com]
Hail to the Nibbler in Chief.
(Score: 2) by Knowledge Troll on Sunday December 13 2015, @04:48PM
I looked a Hesiod the last time I needed central auth but stuck with LDAP. I'm glad to see it in use somewhere and I'm wondering how it has worked out for you. Looked like a great idea if it would do what was needed and DNSSEC was enabled.
Having Kerberos on the naming server though seems weird. The kind of weird where a bug in BIND (or dns server of choice) followed up with a bug in the OS Kernel causes the entire auth system to be untrustable.
I am really curious though how this architecture has worked out.
(Score: 0) by Anonymous Coward on Sunday December 13 2015, @11:25PM
No, they just use their NGO and banks and various organisations to undermine western societies.
(Score: 0) by Anonymous Coward on Sunday December 13 2015, @04:58PM
Got a similar email from Linode. They wrote that I could avoid the reboot by switching to KVM. Thoughts? Does KVM need as many reboots as Xen?
(Score: 2) by isostatic on Sunday December 13 2015, @07:33PM
Fewer security problems perhaps?
A modern resiliant installation spreads out across multiple nodes on multiple continents and can survive many of them vanishing without notice. It can even self heal as the number of nodes for a given task reduces, so new ones are created.
The stickler tends to be DNS over multiple networks, you could get an outage of a few minutes if you have say a ttl of 300 and you lose the IP address DNS points to.
(Score: 0) by Anonymous Coward on Sunday December 13 2015, @08:47PM
Probably. I'll be getting the linode email at work on Monday. I'm closing down the Linode VM's and moving to hardware virtualization with another provider precisely so I don't have to use xen kernels - even if performance is a little worse than under xen.
(Score: 2) by paulej72 on Monday December 14 2015, @02:13AM
We need to test if our setup is KVM compatible. As they did not give us much time to prepare, I thought it best to hold off on that migration.
Team Leader for SN Development
(Score: 1, Informative) by Anonymous Coward on Monday December 14 2015, @10:13AM
I converted our 5 remaining Xen Linode VPSs to KVM this weekend. No issues whatsoever. Plus, the downtime was much shorter than the Xen maintenance window.
The response time on KVM seems to be improved as well. It's certainly something SN should consider.
(Score: -1, Offtopic) by Anonymous Coward on Sunday December 13 2015, @05:03PM
I LOVE TO RUB MY BALLS ON MY GRANDMA'S FOREHEAD!
(Score: -1, Troll) by Anonymous Coward on Sunday December 13 2015, @05:12PM
RIP in pieces Grandma
1927-2016
died of testicle overdose
(Score: 2) by boltronics on Monday December 14 2015, @02:07AM
I don't know about Linode, but AWS (and surely other provides) gives you the option to launch instances onto Xen hosts that already have the update applied. That means you can migrate your instances to new hosts in advance of the reboot, and avoid downtime - assuming you are using a configuration management system to seamlessly deploy new instances (which is a generally a really good idea). That's how we manage our websites, the bulk of which is spread across 21 instances.
It's GNU/Linux dammit!
(Score: 2) by paulej72 on Monday December 14 2015, @02:15AM
Our setup is a bit too small to have configuration management. It would be more work than what it would get us.
Team Leader for SN Development
(Score: 2) by zeigerpuppy on Thursday December 17 2015, @04:31AM
My setup is a bit small for this too,
But am interested in what you suggest for configuration management.
I assume that the primary issues would transitioning DNS redirections, http proxy configs and ensuring database continuity.
On my Xen server I currently manage things by having a management VM that acts as a salt master but wondering what tools you recommend for more complex setups?
(Score: 3, Informative) by boltronics on Thursday December 17 2015, @06:47AM
We're using Salt Stack as well.
I create clean AMI base images using Debian Image Builder [github.com] (currently without systemd), and share those AMIs from our production account to our separate AWS staging accounts for testing. All instances everywhere are then launched from the same AMI which we fully trust since we built it.
We have salt-cloud configured to provision instances as required, as we do not (as of yet) auto-scale. We do have a Orchestrate setup in place, but haven't had much of a need to use it - instead preferring to replace just part of our infrastructure at a time (such as the backend app nodes and memcached servers, and that's just a matter of spinning up new replacement instances. In the case of app nodes, the latest builds are deployed from git directly or (depending on the application) from our CI server which builds debs and pushes them to an apt repo when all tests pass. When we're satisfied all the new app nodes are happy, we can nuke the old ones from the Salt Mine, re-run highstate on the front-end proxies (which regenerates the configs), so the load is now distributed exclusively to the new servers. We can then just terminate the old instances at our leisure.
Front end proxies are migrated by just firing up new ones and jumping across the AWS Elastic IP. We use multi-AZ RDS instances for databases, so typically we can just fail-over to another zone and not have to do much there.
We don't reference hosts by DNS internally in most cases, opting to use AWS tags and Salt grains instead (which are all auto-assigned via salt-cloud). IPs of instances are stored in the Mine and easily queried. We mainly just use DNS for public-facing addresses, and those remain static due to our use of Elastic IPs when required.
It's GNU/Linux dammit!
(Score: 2) by zeigerpuppy on Thursday December 17 2015, @09:31PM
Thanks for the description. Useful for when our service expands.
At the moment in loving the combination of Xen and ZFS on our Debian servers.
I can clone a DomU really quickly (their root partitions are zvols) and it's easy snapshotting them.
However we are using internal DNS which I'm
Not sure if it's a strength or weakness yet!
(Score: 2) by boltronics on Friday December 18 2015, @12:48AM
Makes sense.
For instances I host for personal use (on my home server running my blog, e-mail, etc.), or for infrastructure hosted on my workplace premises, I use Xen instances too. I use LVM2 though instead of ZFS, which works nicely with xen-tools on Wheezy and I feel is better supported for now. But everything in those environment is pretty much static (and I can always easily schedule downtime after hours) so I do use a DNS server in those situations for instance addressing.
To be fair, I could use DNS for IaaS-hosted instances too, but it seems redundant and adds unnecessary complexity when there are easier ways.
It's GNU/Linux dammit!