Stories
Slash Boxes
Comments

SoylentNews is people

posted by paulej72 on Saturday March 07 2015, @09:10AM   Printer-friendly
from the Xen-and-the-Art-of-Web-Site-Maintenance dept.

Our VM hosting provider, Linode, needs to update the Xen VM software on the hosts that serve up all of our VMs. Linode is reserving 2 hours of downtime but expects to take less time. Unfortunately we cannot change when this is being done, but at least it is during low demand times. Here is a list of when servers will be going down for maintenance:

  • 2015-03-07 3:00:00 PM UTC - nitrogen: staff slash and tor exit node server
  • 2015-03-07 3:00:00 PM UTC - fluorine: production frontend
  • 2015-03-07 4:00:00 PM UTC - carbon: irc server
  • 2015-03-07 6:00:00 PM UTC - boron: staff services
  • 2015-03-07 6:00:00 PM UTC - lithium: dev server
  • 2015-03-07 7:00:00 PM UTC - neon: production db
  • 2015-03-07 7:00:00 PM UTC - helium: backup prod db
  • 2015-03-08 7:00:00 PM UTC - hydrogen: backup frontend
  • 2015-03-08 9:00:00 PM UTC - beryllium: mail and wiki server

We will be taking time to also upgrade the kernel prior to the downtime so the newest one will be loaded upon restart.

More info is available here: http://status.linode.com/incidents/2dyvn29ds5mz.

[For the USAians: downtime starts Saturday 03/07 @ 10 AM EST; 7 AM PST]

This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 2) by WizardFusion on Monday March 02 2015, @04:05PM

    by WizardFusion (498) Subscriber Badge on Monday March 02 2015, @04:05PM (#151921) Journal

    Thanks for the heads up. Will the site not fail-over to the backup servers.?

    • (Score: 3, Informative) by paulej72 on Monday March 02 2015, @04:19PM

      by paulej72 (58) on Monday March 02 2015, @04:19PM (#151933) Journal
      Well our backups are not so much backup servers right now due to some issues on each, so no fail-over right now. Also the timing is such that the both db servers and one of the frontends (non working one) will be down all at the same time. We basically have been trying to kill off the feature that would allow frontends to work with out a DB for a time. This would be nice for this case, but makes it hard to keep files synchronized between frontends.
      --
      Team Leader for SN Development
    • (Score: 0) by Anonymous Coward on Monday March 02 2015, @04:39PM

      by Anonymous Coward on Monday March 02 2015, @04:39PM (#151940)

      If you look closely, you'll see that the backup db & prod db are going down at the same time, as is the backup frontend. The production frontend goes down earlier in the day, so it should be back up when those 3 go down, but what difference does it make if there's no database to talk to?

  • (Score: 4, Insightful) by Anonymous Coward on Monday March 02 2015, @04:20PM

    by Anonymous Coward on Monday March 02 2015, @04:20PM (#151934)

    not to do two things at once.
    Best to wait for kernel upgrade till the host is done and everything is back to operation, else you introduce another complexity in troubleshooting.

    • (Score: 2, Informative) by Anonymous Coward on Monday March 02 2015, @10:00PM

      by Anonymous Coward on Monday March 02 2015, @10:00PM (#152163)

      If it had worked out that this came after the release of kernel 4.0, word is that reboots won't be necessary.

      What we discussed previously in
      Kernel Live-Patching Moving into the Linux Kernel [soylentnews.org]
      is slated to be folded into Linux 4.0.

      We'll soon be at the point that notifying the community of these events will be just a formality.

      To anyone still running Windoze:
      Do you still have to restart your OS when you change your screen resolution?

      -- gewg_

      • (Score: 0, Touché) by Anonymous Coward on Tuesday March 03 2015, @09:30AM

        by Anonymous Coward on Tuesday March 03 2015, @09:30AM (#152407)

        Windoze must indeed still require reboots for ridiculously minor things.
        Redmond's fans are so touchy.

        -- gewg_

        • (Score: 2) by DeathMonkey on Friday March 06 2015, @06:36PM

          by DeathMonkey (1380) on Friday March 06 2015, @06:36PM (#153888) Journal

          Judging by the mod...Windoze must indeed still require reboots for ridiculously minor things.
           
          Well, you definitely don't need to reboot to change resolution. That hasn't been the case for at least 15 years (and I don't actually recall it ever being the case).
           
          That could also explain the mod.

      • (Score: 2) by danomac on Saturday March 07 2015, @06:01PM

        by danomac (979) on Saturday March 07 2015, @06:01PM (#154172)

        To anyone still running Windoze: Do you still have to restart your OS when you change your screen resolution?

        To change resolutions, no.

        If you change the DPI settings (to make fonts & icons larger) you have to log out and back in again, but no actual system restart. I actually can't remember the last time Windows wanted a system restart due to a resolution change. Windows ME maybe?

        • (Score: 3, Informative) by Reziac on Sunday March 08 2015, @02:25AM

          by Reziac (2489) on Sunday March 08 2015, @02:25AM (#154311) Homepage

          Windows 3.1, actually. Even Win95 could change resolution without a restart.

          --
          And there is no Alkibiades to come back and save us from ourselves.
  • (Score: 1) by Horse With Stripes on Monday March 02 2015, @05:16PM

    by Horse With Stripes (577) on Monday March 02 2015, @05:16PM (#151962)

    We got the same info from Linode about our VPSs. One VPS is going to be affected midday on Sat, but other than that most of the downtime will be out of our client's main hours of operation.

    It's hard to pick a good time for this type of thing, but we got pretty lucky.

  • (Score: 3, Informative) by isostatic on Monday March 02 2015, @07:16PM

    by isostatic (365) on Monday March 02 2015, @07:16PM (#152037) Journal

    One of my 3 servers migrated today, between 1800 and 2000 UTC.

    The actual apache downtime started at 02/Mar/2015:18:13:45 +0000, and lasted until 02/Mar/2015:18:40:24 +0000, according to access.log (there's hits every second)

    So 27 minutes downtime.

    I have a total of 7 servers operating -- 3 in linode (1 dev, 2 production), as well as 2 real-iron boxes in a data centre in the UK, and one in the U.S, and a dev one in the UK. The system is resilient so downtime isn't a major issue.

    I'm very keen on Singapore opening up for linode instances though. Unfortunatly I feel I'm going to have to buy some real hardware and land them in the office instead unless they get a move on!

  • (Score: 2) by sudo rm -rf on Thursday March 05 2015, @03:21PM

    by sudo rm -rf (2357) on Thursday March 05 2015, @03:21PM (#153527) Journal

    Is it at 03:00 UTC like in the headline or at 03:00 PM like in the summary (which would then be 15:00)?
    ... forget it, the last line says it is 7 PM PST, which is 15:00 UTC.
    OT: Wasn't there a short time period after the french revolution where (at least in France) a day had 10 hours, an hour 100 minutes and a minute 100 seconds? If only that would have become a standard...

    • (Score: 0) by Anonymous Coward on Thursday March 05 2015, @03:53PM

      by Anonymous Coward on Thursday March 05 2015, @03:53PM (#153543)

      Yeah, it would be nice if all times were given in 24h format (I actually missed the PM part completely and thought it would indeed be 03:00 UTC).

      • (Score: 2) by paulej72 on Thursday March 05 2015, @08:07PM

        by paulej72 (58) on Thursday March 05 2015, @08:07PM (#153631) Journal
        I have corrected the story title and comment at the bottom. Damn Linode to use AM/PM with UTC. The text in with the dates and servers was from Linode's email to me. I just added the purpose of each server. I missed the PM patrt when I created the first version of the title. Sorry for any confusion.
        --
        Team Leader for SN Development
        • (Score: 2) by VLM on Saturday March 07 2015, @08:40PM

          by VLM (445) Subscriber Badge on Saturday March 07 2015, @08:40PM (#154212)

          Damn Linode to use AM/PM with UTC.

          Yeah, they got me too. Oh, "6pm" on saturday? I guess I'll have something to check up on after dinner, but the rest of the day will be pretty boring.

          Oh you mean 6pm as in 1800 UTC which was a couple hours ago. That was surprising when it happened.

          Everything turned out fine, just happened when I least expected it.

          Now if I could just get a freebsd "linode" at linode I'd be pretty happy dude.

    • (Score: 3, Informative) by arashi no garou on Thursday March 05 2015, @07:24PM

      by arashi no garou (2796) on Thursday March 05 2015, @07:24PM (#153605)

      OT: Wasn't there a short time period after the french revolution where (at least in France) a day had 10 hours, an hour 100 minutes and a minute 100 seconds?

      Yep, and the concept has been used in a few sci-fi and fantasy works over the years. Robert Jordan's The Wheel of Time series comes to mind with its 10-day weeks and 3-week months.

      https://en.wikipedia.org/wiki/Decimal_time#France [wikipedia.org]

    • (Score: 2) by pogostix on Thursday March 05 2015, @11:19PM

      by pogostix (1696) on Thursday March 05 2015, @11:19PM (#153674)

      100x100x10 doesn't line up nicely with 60x60x24
      So if we want to make plans for breakfast 2 weeks from now it'd be hard to pick a time! I'd probably just say umm txt me that morning and we'll figure it out!

      • (Score: 0) by Anonymous Coward on Friday March 06 2015, @03:08PM

        by Anonymous Coward on Friday March 06 2015, @03:08PM (#153844)

        100x100x10 doesn't line up nicely with 60x60x24

        Not any worse than the other imperial measures.

        So if we want to make plans for breakfast 2 weeks from now it'd be hard to pick a time!

        Only if you are bad at math.

        Anyway, "at sunrise" would be a pretty universal time specification.

        • (Score: 2) by c0lo on Friday March 06 2015, @09:05PM

          by c0lo (156) Subscriber Badge on Friday March 06 2015, @09:05PM (#153944) Journal

          Anyway, "at sunrise" would be a pretty universal time specification.

          Maybe... but only if you can arrange for a synchronized dawn/dusk for all the timezones.

          --
          https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
  • (Score: 0) by Anonymous Coward on Friday March 06 2015, @03:47AM

    by Anonymous Coward on Friday March 06 2015, @03:47AM (#153713)

    1/ Just say yanks, its easier to parse and everyone knows what it means.
    2/ wtf is Strudar?

    • (Score: 1, Insightful) by Anonymous Coward on Friday March 06 2015, @11:42AM

      by Anonymous Coward on Friday March 06 2015, @11:42AM (#153801)

      Or simply say Americans. South America has the same time zones as North America, so you'd be right no matter which interpretation of "American" is used.

  • (Score: 5, Insightful) by Anonymous Coward on Friday March 06 2015, @12:55PM

    by Anonymous Coward on Friday March 06 2015, @12:55PM (#153820)

    That's an awful lot of servers for such a low revenue non-critical website. You gonna bankrupt yourselves in no time if you keep that up.

    Just sayin.

    • (Score: 3, Insightful) by jdccdevel on Friday March 06 2015, @08:26PM

      by jdccdevel (1329) on Friday March 06 2015, @08:26PM (#153928) Journal

      I was thinking the same thing, but I'm not at all familiar with the way the backend is structured, software wise.

      That said, maybe they should look into moving some of those services to LXC [linuxcontainers.org] type containers. I've worked with them before, and it lets you run small services like IRC in what is, essentially, a high-powered chroot - style virtual machine. It works using overlay filesystems and kernel namespaces, so it shouldn't require any special VM hardware configuration in the node.

      Given the site's current financial status, I would think that consolidating the carbon, boron, and maybe beryllium servers together would be a viable cost reduction strategy. It seems like those services would be relatively low traffic.

      • (Score: 5, Informative) by paulej72 on Friday March 06 2015, @09:31PM

        by paulej72 (58) on Friday March 06 2015, @09:31PM (#153958) Journal
        It has been mentioned in previous articles that we are consolidating beryllium, carbon and nitrogen onto boron. Our problem is more on the ram side of things. Some of the stuff we run seems to want a lot of ram so consolidation is a bit tricky, but we are working on it. The biggest thing is we are volunteers and we don't always have the time to dedicate to sysop jobs like this when we are finished with our days jobs. Plus some of the sysops we had when we went first started are no longer around to help out, so it makes it harder for those that are left.
        --
        Team Leader for SN Development
        • (Score: 2) by isostatic on Saturday March 07 2015, @12:42PM

          by isostatic (365) on Saturday March 07 2015, @12:42PM (#154105) Journal

          The linked structure means you can get two servers with 1GB of ram for the same price as a server with 2GB. Aside from making sense,

          I'd personally like more disk space on mine, and less ram, due to the services I run, however I appreciate that a bespoke model means higher cost, and the cost is way less than the datacentre hosting costs, even ignoring the capital cost upfront.

  • (Score: 2) by pkrasimirov on Saturday March 07 2015, @11:34AM

    by pkrasimirov (3358) Subscriber Badge on Saturday March 07 2015, @11:34AM (#154099)

    Can you make the browser ask for confirmation if I try to close the comment page but did not submit the text. Like Wikipedia does.