SoylentNews is people
SoylentNews
As part of Linode's migration of servers to a new Data Center in Dallas, two of our servers were scheduled for migration at 10pm EDT on September 29, 2017. NCommander happened to be around when I sent out a reminder I'd received from Linode, so he 'hit the button' at 9:30pm tonight (Sept. 28) and did a manual migration ahead of time.
Unless you were on our IRC server (Internet Relay Chat) at the time, you probably didn't even notice... and even then, it was unavailable for only about 15-20 minutes. Redundancy for the win!
That leaves us with a single server, sodium, to migrate. It is currently scheduled for migration on Tuesday, October 3, 2017 at 10:00pm EDT. Since sodium is one of two front-end proxies for us (the other is magnesium which has already been migrated), I expect we'll be able to perform that migration without any site interruption.
Separately, and in parallel, we are slowly moving our servers from Ubuntu 14.04 LTS to Gentoo.
To the community, thank you for your patience as we work our way through this process. And, for those of you who may have been with us from the outset, and when up-time was measured in hours, please join me in congratulating the team for their dedication and hard work which has facilitated such an uneventful migration!
(Score: 2, Informative) by Kalas on Friday September 29 2017, @05:23AM (14 children)
Is there anyone here who didn't already know what IRC is?
Anywho keep up the great work. I'm forever grateful that we have a truly community-driven successor to the green site.
(Score: 0) by Anonymous Coward on Friday September 29 2017, @06:41AM (5 children)
There's always someone.
And congratulations on the well done migration.
(Score: 1) by anubi on Friday September 29 2017, @08:00AM (4 children)
You changed over so well I was completely unaware that you had did it!
I think its quite a feat to do something like that.
When I change something out, its down for at least an hour and sometimes several days if I'm under pressure to do it in a hurry. ( I don't know why it works that way, but the likelihood of nasty surprises is directly proportional to the pressure to get the job done in a hurry.)
"Prove all things; hold fast that which is good." [KJV: I Thessalonians 5:21]
(Score: 2) by ledow on Friday September 29 2017, @09:04AM (3 children)
To be honest, is this really an achievement nowadays?
I have a bunch of services that need to be up 24/7. I mean, it's not the very end of the world if they go down, but people will ask questions about why they were kicked off at 3am.
So what you do is run a redundant configuration, on a virtualised back-end.
I snapshot the VM I want to upgrade.
I restore the snapshot to another hypervisor and disable its network interface.
I upgrade the machine and test that everything went as well as I need it to.
I switch off the original VM and boot up the new one, keeping an eye that it all works as expected, and keeping the old snapshot to hand in case I need a quick restore.
In the literal SECONDS it takes to switch from one to the other, or back to the old snapshot, the front-end balancing, protocol redundancy, etc. should ensure that nobody sees any change whatsoever.
As they say, they have two proxy balancers, so as long as they don't take both down for upgrades/migration at the same time, you'll never see any difference as a user.
I have a Linux VM that acts as firewall, webfilter, router, VPN endpoint, etc. for my workplace on a leased line.
When I upgrade that, it lets you specify a failover server.
The failover server is a copy of the same VM on a different IP.
I take one down, upgrade it, test it, put it back online.
I take the other, do the same.
And that's on the PRIMARY GATEWAY to the net, live, during the day, while users are using it.
Physical hardware migration is - in essence - exactly the same. Migrate all the VMs off the machine you want to turn off (literally seconds if they're already replicating by default, which they should be). When it's idle, turn it off, do what you want.
If you have enough physical servers, then even the one you migrate it to going down shouldn't interrupt anything. VM failover from a recent replica should be almost instantaneous, and the gap covered by said balancers, etc.
This isn't amazing. This is just bog-standard IT now. Linux or Windows. VMWare or HyperV.
Please tell me that Soylent runs on a virtualised infrastructure?
(Score: 2) by richtopia on Friday September 29 2017, @02:31PM (1 child)
For an IT team perhaps this is commonplace, but in my experience even in industry migrations are painful, as many of the servers I depend on are covered by "Scott, that guy who knows computers".
I was actually trying to do a failover solution based on Raspberry Pis the other day, as they provide an excellent cheap demo for redundancy. While well within grasp of someone with some time and understanding of networking, it definitely is not turnkey and there are plenty of opportunities for things to go amiss.
(Score: 2) by ledow on Friday September 29 2017, @03:09PM
My life is usually a one- or two-man IT team. It's been more, but rarely do you need that many, precisely because you have the tools nowadays to make it that simple.
Snapshot is right-click snapshot.
Move is right-click move.
Turning off the network interface is Properties... Network interface... set to None.
Admittedly that's Windows HyperV, but VMWare has the same and I can't imagine it's more than a few commands on any CLI hypervisor (in fact, it's probably a lot worse trying to do that through the Windows Server Core CLI than if you have to write a script to do so on Xen or similar!).
This is commodity technology nowadays, and I'd hope someone, somewhere running a website like Soylent with multiple database servers and proxy balancers, etc. actually knew this stuff was possible.
(Score: 2) by NCommander on Monday October 02 2017, @07:51AM
While this is possible in theory, Linode doesn't support this, and in my experience, live migration on Amazon was a crapshoot at best, and didn't work across availability zones (which is the closest equivalent here). Each migration required a full powerdown, image transfer (~5-10 minutes), and then recovery. This usually worked without issue though neon's migration had serious network loss issues when we first did it.
Under normal circumstances, IPv4 failover also works exactly the way you describe, but IPv6 failover is stupidly black magic, and isn't supported by Linode, combined with the fact about 10-15% of all SN traffic is IPv6, the loadbalancers are a special case when we need to play the hardware migration song and dance.
Furthermore, full redundancy for things like off-lining a server for maintenance is still important and a lot of large websites seem to run on a single DB server with a web frontend and thus have bad things happen if/when anything major has to happen.
Still always moving
(Score: 2) by edIII on Friday September 29 2017, @08:07AM (1 child)
I have the VT100 theme turned on, so this site is the green site now. Super green.
Technically, lunchtime is at any moment. It's just a wave function.
(Score: 0) by Anonymous Coward on Friday September 29 2017, @09:38PM
Yep, super green. Easy on the eyes, I've been using it for SN since they made it available.
Annnnd -- much better all around than one of those original VT100 terminals (we used Ann Arbors back then).
(Score: 2) by ledow on Friday September 29 2017, @08:55AM (5 children)
I haven't joined an IRC channel or loaded an IRC client in... probably 8-10 years.
IRC is a very niche usage. I guarantee you that most people under 25/30 have never used it in their lives or even heard of it. Even among techies, I know few people that have ever used IRC.
Everything's done on Slack, now, isn't it? All those kinds of collaborative tools. IRC is really quite dated in comparison.
Though you and I may know what it is, it's not at all a given that anyone else does. Hell MSN Messenger has been dead for years (20 year olds will talk about it fondly as the thing they did online when they were a kid!). IRC has been out of mainstream internet usage for a lot longer than that.
(Score: 3, Interesting) by FatPhil on Friday September 29 2017, @09:09AM (2 children)
Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
(Score: 2) by ledow on Friday September 29 2017, @10:01AM (1 child)
Article from 2013:
https://techcrunch.com/2013/01/06/irc-has-lost-60-of-its-users-since-2003-but-life-as-a-robot-is-just-beginning/ [techcrunch.com]
"going from 1 million [users] in 2003 to about 400,000 today"
Sorry, but IRC is basically error-margin by now, even if they haven't included every network, private IRC, etc.
Stack claims something like 200m users (though it's likely most of them are free users, and probably only about 1m paying based on some quick maths from their income / subscription cost).
I agree it has a lot of advantages, nobody questions that, and I've run my own private IRC servers for various projects. But as a mass-market technology, it's not even on the radar. It's a techy-geek niche, like Usenet turned into.
(Score: 2) by FatPhil on Friday September 29 2017, @11:47AM
Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
(Score: 0) by Anonymous Coward on Friday September 29 2017, @10:09AM (1 child)
Slack is nothing more than IRC, running on one company's servers, with backlog. Except it eats a lot of resources from the computer running it.
(Score: 0) by Anonymous Coward on Friday September 29 2017, @05:10PM
Which bugs me to no end. Our internal communication switched to Slack, this was after going from Messenger for Business, Lync, Skype for business, Office Communicator, something built into Outlook, and that's just the Microsoft stuff. We also hopped to other systems. The whole time our IT department was just screaming to use IRC, which they had used originally, way way back in the day. One nice thing about Slack though, is that you can use your own IRC client, but that might be a feature allowed to us because we pay through the nose.