[Update: as of 20180603_135454 UTC, all 4 server upgrades have completed and all servers should be running normally. --martyb]
Our server provider, Linode, has made available a free upgrade for our servers. Generally, it's a storage upgrade and sometimes a bandwidth upgrade, too. We use only the tiniest fraction of our bandwidth allocation each month (something like < 1%), but the extra storage IS useful.
We (read: The Mighty Buzzard) have already taken 4 of our servers out of rotation this morning. We anticipate no community-visible interruptions of service as a result of these upgrades.
So far, migrations of sodium, hydrogen, and lithium have completed. Migration of neon has started and should complete within the next 10 minutes or so.
Migrations of the remaining servers may cause some community-visible effects. We will provide advance notice before we perform those upgrades. Very tentatively planned for Thursday.
(Score: 0) by Anonymous Coward on Sunday June 03 2018, @02:03PM
Great job! Thank you, Linode.
(Score: 4, Funny) by RandomFactor on Sunday June 03 2018, @02:05PM (3 children)
Hmmm, you could use a naming scheme to indicate which servers are improved. Like a prefix.
I would suggest "Di"
it seems logical to indicate a performance or speed boost, right?
В «Правде» нет известий, в «Известиях» нет правды
(Score: 2) by janrinok on Sunday June 03 2018, @02:28PM
I can see where you were going with this.
It's a pity that we don't have a monoxide server too...
(Score: 2) by martyb on Sunday June 03 2018, @02:36PM
Missed that one at first, then got me a good laugh! Thanks for that! You get a "+1 Funny" from me.
Wit is intellect, dancing.
(Score: 2) by realDonaldTrump on Sunday June 03 2018, @08:51PM
If they did "di" then sodium becomes disodium. You're at a fast food restaurant, you get the combo meal, great deal because it comes with the chips & drink, right? And in the chips, disodium inosinate. And you go, what is that? Turns out it's meat extract, it's the thing that makes meat delicious. I love disodium!!!
(Score: 2) by Subsentient on Sunday June 03 2018, @03:50PM (3 children)
I know how long a linode takes to boot. I've used them a lot. I've recently completed the storage upgrade on a friend's linode. It takes around 10 minutes at most.
You could do all of this far faster if you just gave us a day's warning and then rebooted them all at once, then everyone could forget about it.
"It is no measure of health to be well adjusted to a profoundly sick society." -Jiddu Krishnamurti
(Score: 1) by Ethanol-fueled on Sunday June 03 2018, @03:55PM (1 child)
Yeah but they do it in style, for no downtime. Ya gotta hand it to them, they treat their servers like Jew-bank servers. 100% uptime.
(Score: 2) by Subsentient on Sunday June 03 2018, @03:59PM
True. Good to see you, btw. Looking forward to more stories in your journal.
"It is no measure of health to be well adjusted to a profoundly sick society." -Jiddu Krishnamurti
(Score: 5, Informative) by martyb on Sunday June 03 2018, @04:58PM
For our smaller servers, yes, but we have some 8GB Linodes for which the migration is more than "10 minutes at most".
From our Linode manager page for hydrogen (8GB RAM; now 163840 MB storage):
Yes, I suppose we could do them all in parallel. I actually considered doing that with today's migrations! But if we migrate several instances at the same time, and something goes sideways... do YOU want to be the one that tries to sort things out with who knows what services still [not] running? I surely didn't. As there would be no difference in impact to our community (ie. no impact at all), we took the slow and steady route, successfully. And, had we not posted this, none would have known that we had even migrated!
What we actually did: TMB moved 4 servers out of the rotation this morning (call this Phase 1). I initiated and monitored the upgrades/migrations — one at a time — and these four servers are waiting for TMB to finish his fishing and move the servers back into the rotation. He does have his priorities!
And, because of how the site is set up, there was absolutely no impact/downtime to any services on the site.
Once these are back in the rotation, there are 3-4 more (I think) servers (call this Phase 2) that we can offline, migrate, and resume with no impact on the site.
The remaining servers host things like e-mail and IRC (Phase 3) for which the impact of being down for a migration is minimal and isolated. Just isn't worth the time, effort, and money (for still more redundant servers) to have full redundancy.
The plan for the Phase 3 servers is: post advance warning on the site, hit the button, and let everyone know when it's done. Even then, we'll probably take it one server at a time as it is convenient to still have, say, IRC working while we are migrating other servers.
Wit is intellect, dancing.