posted by
paulej72
on Saturday March 07 2015, @09:10AM
from the Xen-and-the-Art-of-Web-Site-Maintenance dept.
from the Xen-and-the-Art-of-Web-Site-Maintenance dept.
Our VM hosting provider, Linode, needs to update the Xen VM software on the hosts that serve up all of our VMs. Linode is reserving 2 hours of downtime but expects to take less time. Unfortunately we cannot change when this is being done, but at least it is during low demand times. Here is a list of when servers will be going down for maintenance:
- 2015-03-07 3:00:00 PM UTC - nitrogen: staff slash and tor exit node server
- 2015-03-07 3:00:00 PM UTC - fluorine: production frontend
- 2015-03-07 4:00:00 PM UTC - carbon: irc server
- 2015-03-07 6:00:00 PM UTC - boron: staff services
- 2015-03-07 6:00:00 PM UTC - lithium: dev server
- 2015-03-07 7:00:00 PM UTC - neon: production db
- 2015-03-07 7:00:00 PM UTC - helium: backup prod db
- 2015-03-08 7:00:00 PM UTC - hydrogen: backup frontend
- 2015-03-08 9:00:00 PM UTC - beryllium: mail and wiki server
We will be taking time to also upgrade the kernel prior to the downtime so the newest one will be loaded upon restart.
More info is available here: http://status.linode.com/incidents/2dyvn29ds5mz.
[For the USAians: downtime starts Saturday 03/07 @ 10 AM EST; 7 AM PST]
This discussion has been archived.
No new comments can be posted.
Reminder: Site Downtime Starts Today at 15:00 UTC
|
Log In/Create an Account
| Top
| 27 comments
| Search Discussion
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
(Score: 2) by WizardFusion on Monday March 02 2015, @04:05PM
Thanks for the heads up. Will the site not fail-over to the backup servers.?
(Score: 3, Informative) by paulej72 on Monday March 02 2015, @04:19PM
Team Leader for SN Development
(Score: 0) by Anonymous Coward on Monday March 02 2015, @04:39PM
If you look closely, you'll see that the backup db & prod db are going down at the same time, as is the backup frontend. The production frontend goes down earlier in the day, so it should be back up when those 3 go down, but what difference does it make if there's no database to talk to?
(Score: 4, Insightful) by Anonymous Coward on Monday March 02 2015, @04:20PM
not to do two things at once.
Best to wait for kernel upgrade till the host is done and everything is back to operation, else you introduce another complexity in troubleshooting.
(Score: 2, Informative) by Anonymous Coward on Monday March 02 2015, @10:00PM
If it had worked out that this came after the release of kernel 4.0, word is that reboots won't be necessary.
What we discussed previously in
Kernel Live-Patching Moving into the Linux Kernel [soylentnews.org]
is slated to be folded into Linux 4.0.
We'll soon be at the point that notifying the community of these events will be just a formality.
To anyone still running Windoze:
Do you still have to restart your OS when you change your screen resolution?
-- gewg_
(Score: 0, Touché) by Anonymous Coward on Tuesday March 03 2015, @09:30AM
Windoze must indeed still require reboots for ridiculously minor things.
Redmond's fans are so touchy.
-- gewg_
(Score: 2) by DeathMonkey on Friday March 06 2015, @06:36PM
Judging by the mod...Windoze must indeed still require reboots for ridiculously minor things.
Well, you definitely don't need to reboot to change resolution. That hasn't been the case for at least 15 years (and I don't actually recall it ever being the case).
That could also explain the mod.
(Score: 2) by danomac on Saturday March 07 2015, @06:01PM
To change resolutions, no.
If you change the DPI settings (to make fonts & icons larger) you have to log out and back in again, but no actual system restart. I actually can't remember the last time Windows wanted a system restart due to a resolution change. Windows ME maybe?
(Score: 3, Informative) by Reziac on Sunday March 08 2015, @02:25AM
Windows 3.1, actually. Even Win95 could change resolution without a restart.
And there is no Alkibiades to come back and save us from ourselves.
(Score: 1) by Horse With Stripes on Monday March 02 2015, @05:16PM
We got the same info from Linode about our VPSs. One VPS is going to be affected midday on Sat, but other than that most of the downtime will be out of our client's main hours of operation.
It's hard to pick a good time for this type of thing, but we got pretty lucky.
(Score: 3, Informative) by isostatic on Monday March 02 2015, @07:16PM
One of my 3 servers migrated today, between 1800 and 2000 UTC.
The actual apache downtime started at 02/Mar/2015:18:13:45 +0000, and lasted until 02/Mar/2015:18:40:24 +0000, according to access.log (there's hits every second)
So 27 minutes downtime.
I have a total of 7 servers operating -- 3 in linode (1 dev, 2 production), as well as 2 real-iron boxes in a data centre in the UK, and one in the U.S, and a dev one in the UK. The system is resilient so downtime isn't a major issue.
I'm very keen on Singapore opening up for linode instances though. Unfortunatly I feel I'm going to have to buy some real hardware and land them in the office instead unless they get a move on!
(Score: 1) by Horse With Stripes on Friday March 06 2015, @11:32PM
Our biggest server took its turn this morning. It was only down for 13 minutes. Can't complain about that.
(Score: 2) by sudo rm -rf on Thursday March 05 2015, @03:21PM
Is it at 03:00 UTC like in the headline or at 03:00 PM like in the summary (which would then be 15:00)?
... forget it, the last line says it is 7 PM PST, which is 15:00 UTC.
OT: Wasn't there a short time period after the french revolution where (at least in France) a day had 10 hours, an hour 100 minutes and a minute 100 seconds? If only that would have become a standard...
(Score: 0) by Anonymous Coward on Thursday March 05 2015, @03:53PM
Yeah, it would be nice if all times were given in 24h format (I actually missed the PM part completely and thought it would indeed be 03:00 UTC).
(Score: 2) by paulej72 on Thursday March 05 2015, @08:07PM
Team Leader for SN Development
(Score: 2) by VLM on Saturday March 07 2015, @08:40PM
Damn Linode to use AM/PM with UTC.
Yeah, they got me too. Oh, "6pm" on saturday? I guess I'll have something to check up on after dinner, but the rest of the day will be pretty boring.
Oh you mean 6pm as in 1800 UTC which was a couple hours ago. That was surprising when it happened.
Everything turned out fine, just happened when I least expected it.
Now if I could just get a freebsd "linode" at linode I'd be pretty happy dude.
(Score: 3, Informative) by arashi no garou on Thursday March 05 2015, @07:24PM
OT: Wasn't there a short time period after the french revolution where (at least in France) a day had 10 hours, an hour 100 minutes and a minute 100 seconds?
Yep, and the concept has been used in a few sci-fi and fantasy works over the years. Robert Jordan's The Wheel of Time series comes to mind with its 10-day weeks and 3-week months.
https://en.wikipedia.org/wiki/Decimal_time#France [wikipedia.org]
(Score: 2) by pogostix on Thursday March 05 2015, @11:19PM
100x100x10 doesn't line up nicely with 60x60x24
So if we want to make plans for breakfast 2 weeks from now it'd be hard to pick a time! I'd probably just say umm txt me that morning and we'll figure it out!
(Score: 0) by Anonymous Coward on Friday March 06 2015, @03:08PM
Not any worse than the other imperial measures.
Only if you are bad at math.
Anyway, "at sunrise" would be a pretty universal time specification.
(Score: 2) by c0lo on Friday March 06 2015, @09:05PM
Maybe... but only if you can arrange for a synchronized dawn/dusk for all the timezones.
https://www.youtube.com/@ProfSteveKeen https://soylentnews.org/~MichaelDavidCrawford
(Score: 0) by Anonymous Coward on Friday March 06 2015, @03:47AM
1/ Just say yanks, its easier to parse and everyone knows what it means.
2/ wtf is Strudar?
(Score: 1, Insightful) by Anonymous Coward on Friday March 06 2015, @11:42AM
Or simply say Americans. South America has the same time zones as North America, so you'd be right no matter which interpretation of "American" is used.
(Score: 5, Insightful) by Anonymous Coward on Friday March 06 2015, @12:55PM
That's an awful lot of servers for such a low revenue non-critical website. You gonna bankrupt yourselves in no time if you keep that up.
Just sayin.
(Score: 3, Insightful) by jdccdevel on Friday March 06 2015, @08:26PM
I was thinking the same thing, but I'm not at all familiar with the way the backend is structured, software wise.
That said, maybe they should look into moving some of those services to LXC [linuxcontainers.org] type containers. I've worked with them before, and it lets you run small services like IRC in what is, essentially, a high-powered chroot - style virtual machine. It works using overlay filesystems and kernel namespaces, so it shouldn't require any special VM hardware configuration in the node.
Given the site's current financial status, I would think that consolidating the carbon, boron, and maybe beryllium servers together would be a viable cost reduction strategy. It seems like those services would be relatively low traffic.
(Score: 5, Informative) by paulej72 on Friday March 06 2015, @09:31PM
Team Leader for SN Development
(Score: 2) by isostatic on Saturday March 07 2015, @12:42PM
The linked structure means you can get two servers with 1GB of ram for the same price as a server with 2GB. Aside from making sense,
I'd personally like more disk space on mine, and less ram, due to the services I run, however I appreciate that a bespoke model means higher cost, and the cost is way less than the datacentre hosting costs, even ignoring the capital cost upfront.
(Score: 2) by pkrasimirov on Saturday March 07 2015, @11:34AM
Can you make the browser ask for confirmation if I try to close the comment page but did not submit the text. Like Wikipedia does.