Slash Boxes

SoylentNews is people

posted by martyb on Friday September 29 2017, @04:12AM   Printer-friendly
from the moving-right-along dept.

As part of Linode's migration of servers to a new Data Center in Dallas, two of our servers were scheduled for migration at 10pm EDT on September 29, 2017. NCommander happened to be around when I sent out a reminder I'd received from Linode, so he 'hit the button' at 9:30pm tonight (Sept. 28) and did a manual migration ahead of time.

Unless you were on our IRC server (Internet Relay Chat) at the time, you probably didn't even notice... and even then, it was unavailable for only about 15-20 minutes. Redundancy for the win!

That leaves us with a single server, sodium, to migrate. It is currently scheduled for migration on Tuesday, October 3, 2017 at 10:00pm EDT. Since sodium is one of two front-end proxies for us (the other is magnesium which has already been migrated), I expect we'll be able to perform that migration without any site interruption.

Separately, and in parallel, we are slowly moving our servers from Ubuntu 14.04 LTS to Gentoo.

To the community, thank you for your patience as we work our way through this process. And, for those of you who may have been with us from the outset, and when up-time was measured in hours, please join me in congratulating the team for their dedication and hard work which has facilitated such an uneventful migration!

Original Submission

This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 2, Informative) by Kalas on Friday September 29 2017, @05:23AM (14 children)

    by Kalas (4247) on Friday September 29 2017, @05:23AM (#574679)

    IRC server (Internet Relay Chat)

    Is there anyone here who didn't already know what IRC is?
    Anywho keep up the great work. I'm forever grateful that we have a truly community-driven successor to the green site.

    • (Score: 0) by Anonymous Coward on Friday September 29 2017, @06:41AM (5 children)

      by Anonymous Coward on Friday September 29 2017, @06:41AM (#574701)

      There's always someone.

      And congratulations on the well done migration.

      • (Score: 1) by anubi on Friday September 29 2017, @08:00AM (4 children)

        by anubi (2828) on Friday September 29 2017, @08:00AM (#574709) Journal

        You changed over so well I was completely unaware that you had did it!

        I think its quite a feat to do something like that.

        When I change something out, its down for at least an hour and sometimes several days if I'm under pressure to do it in a hurry. ( I don't know why it works that way, but the likelihood of nasty surprises is directly proportional to the pressure to get the job done in a hurry.)

        "Prove all things; hold fast that which is good." [KJV: I Thessalonians 5:21]
        • (Score: 2) by ledow on Friday September 29 2017, @09:04AM (3 children)

          by ledow (5567) on Friday September 29 2017, @09:04AM (#574722) Homepage

          To be honest, is this really an achievement nowadays?

          I have a bunch of services that need to be up 24/7. I mean, it's not the very end of the world if they go down, but people will ask questions about why they were kicked off at 3am.

          So what you do is run a redundant configuration, on a virtualised back-end.

          I snapshot the VM I want to upgrade.
          I restore the snapshot to another hypervisor and disable its network interface.
          I upgrade the machine and test that everything went as well as I need it to.
          I switch off the original VM and boot up the new one, keeping an eye that it all works as expected, and keeping the old snapshot to hand in case I need a quick restore.

          In the literal SECONDS it takes to switch from one to the other, or back to the old snapshot, the front-end balancing, protocol redundancy, etc. should ensure that nobody sees any change whatsoever.

          As they say, they have two proxy balancers, so as long as they don't take both down for upgrades/migration at the same time, you'll never see any difference as a user.

          I have a Linux VM that acts as firewall, webfilter, router, VPN endpoint, etc. for my workplace on a leased line.
          When I upgrade that, it lets you specify a failover server.
          The failover server is a copy of the same VM on a different IP.
          I take one down, upgrade it, test it, put it back online.
          I take the other, do the same.

          And that's on the PRIMARY GATEWAY to the net, live, during the day, while users are using it.

          Physical hardware migration is - in essence - exactly the same. Migrate all the VMs off the machine you want to turn off (literally seconds if they're already replicating by default, which they should be). When it's idle, turn it off, do what you want.
          If you have enough physical servers, then even the one you migrate it to going down shouldn't interrupt anything. VM failover from a recent replica should be almost instantaneous, and the gap covered by said balancers, etc.

          This isn't amazing. This is just bog-standard IT now. Linux or Windows. VMWare or HyperV.

          Please tell me that Soylent runs on a virtualised infrastructure?

          • (Score: 2) by richtopia on Friday September 29 2017, @02:31PM (1 child)

            by richtopia (3160) Subscriber Badge on Friday September 29 2017, @02:31PM (#574806) Homepage Journal

            For an IT team perhaps this is commonplace, but in my experience even in industry migrations are painful, as many of the servers I depend on are covered by "Scott, that guy who knows computers".

            I was actually trying to do a failover solution based on Raspberry Pis the other day, as they provide an excellent cheap demo for redundancy. While well within grasp of someone with some time and understanding of networking, it definitely is not turnkey and there are plenty of opportunities for things to go amiss.

            • (Score: 2) by ledow on Friday September 29 2017, @03:09PM

              by ledow (5567) on Friday September 29 2017, @03:09PM (#574823) Homepage

              My life is usually a one- or two-man IT team. It's been more, but rarely do you need that many, precisely because you have the tools nowadays to make it that simple.

              Snapshot is right-click snapshot.
              Move is right-click move.
              Turning off the network interface is Properties... Network interface... set to None.

              Admittedly that's Windows HyperV, but VMWare has the same and I can't imagine it's more than a few commands on any CLI hypervisor (in fact, it's probably a lot worse trying to do that through the Windows Server Core CLI than if you have to write a script to do so on Xen or similar!).

              This is commodity technology nowadays, and I'd hope someone, somewhere running a website like Soylent with multiple database servers and proxy balancers, etc. actually knew this stuff was possible.

          • (Score: 2) by NCommander on Monday October 02 2017, @07:51AM

            by NCommander (2) Subscriber Badge <> on Monday October 02 2017, @07:51AM (#575804) Homepage Journal

            While this is possible in theory, Linode doesn't support this, and in my experience, live migration on Amazon was a crapshoot at best, and didn't work across availability zones (which is the closest equivalent here). Each migration required a full powerdown, image transfer (~5-10 minutes), and then recovery. This usually worked without issue though neon's migration had serious network loss issues when we first did it.

            Under normal circumstances, IPv4 failover also works exactly the way you describe, but IPv6 failover is stupidly black magic, and isn't supported by Linode, combined with the fact about 10-15% of all SN traffic is IPv6, the loadbalancers are a special case when we need to play the hardware migration song and dance.

            Furthermore, full redundancy for things like off-lining a server for maintenance is still important and a lot of large websites seem to run on a single DB server with a web frontend and thus have bad things happen if/when anything major has to happen.

            Still always moving
    • (Score: 2) by edIII on Friday September 29 2017, @08:07AM (1 child)

      by edIII (791) Subscriber Badge on Friday September 29 2017, @08:07AM (#574712)

      I have the VT100 theme turned on, so this site is the green site now. Super green.

      • (Score: 0) by Anonymous Coward on Friday September 29 2017, @09:38PM

        by Anonymous Coward on Friday September 29 2017, @09:38PM (#575080)

        Yep, super green. Easy on the eyes, I've been using it for SN since they made it available.

        Annnnd -- much better all around than one of those original VT100 terminals (we used Ann Arbors back then).

    • (Score: 2) by ledow on Friday September 29 2017, @08:55AM (5 children)

      by ledow (5567) on Friday September 29 2017, @08:55AM (#574719) Homepage

      I haven't joined an IRC channel or loaded an IRC client in... probably 8-10 years.

      IRC is a very niche usage. I guarantee you that most people under 25/30 have never used it in their lives or even heard of it. Even among techies, I know few people that have ever used IRC.

      Everything's done on Slack, now, isn't it? All those kinds of collaborative tools. IRC is really quite dated in comparison.

      Though you and I may know what it is, it's not at all a given that anyone else does. Hell MSN Messenger has been dead for years (20 year olds will talk about it fondly as the thing they did online when they were a kid!). IRC has been out of mainstream internet usage for a lot longer than that.

      • (Score: 3, Interesting) by FatPhil on Friday September 29 2017, @09:09AM (2 children)

        by FatPhil (863) <{pc-soylent} {at} {}> on Friday September 29 2017, @09:09AM (#574723) Homepage
        I think you'll find that IRC is more popular than the things that came after IRC but aren't fresh any more. It's the only one with any staying power. Probably being an open protocol helped. Anything coming out of MS has a suspended death sentence hanging over it, for example. And in a commodity market such as chat, expect private companies to sell systems with unnecessary gizmos that are briefly popular, but then subsumed by newer gizmos from competitors. IRC doesn't suffer from that desperate inanity, so I don't think it will ever be at threat of dying out.
        If vaccination works, then why doesn't eucharist protect kids against Christianity?
        • (Score: 2) by ledow on Friday September 29 2017, @10:01AM (1 child)

          by ledow (5567) on Friday September 29 2017, @10:01AM (#574731) Homepage

          Article from 2013:


          "going from 1 million [users] in 2003 to about 400,000 today"

          Sorry, but IRC is basically error-margin by now, even if they haven't included every network, private IRC, etc.

          Stack claims something like 200m users (though it's likely most of them are free users, and probably only about 1m paying based on some quick maths from their income / subscription cost).

          I agree it has a lot of advantages, nobody questions that, and I've run my own private IRC servers for various projects. But as a mass-market technology, it's not even on the radar. It's a techy-geek niche, like Usenet turned into.

      • (Score: 0) by Anonymous Coward on Friday September 29 2017, @10:09AM (1 child)

        by Anonymous Coward on Friday September 29 2017, @10:09AM (#574735)

        Slack is nothing more than IRC, running on one company's servers, with backlog. Except it eats a lot of resources from the computer running it.

        • (Score: 0) by Anonymous Coward on Friday September 29 2017, @05:10PM

          by Anonymous Coward on Friday September 29 2017, @05:10PM (#574927)

          Which bugs me to no end. Our internal communication switched to Slack, this was after going from Messenger for Business, Lync, Skype for business, Office Communicator, something built into Outlook, and that's just the Microsoft stuff. We also hopped to other systems. The whole time our IT department was just screaming to use IRC, which they had used originally, way way back in the day. One nice thing about Slack though, is that you can use your own IRC client, but that might be a feature allowed to us because we pay through the nose.

  • (Score: 0) by Anonymous Coward on Friday September 29 2017, @12:42PM

    by Anonymous Coward on Friday September 29 2017, @12:42PM (#574755)

    they say. Let's see how it goes... *fingers crossed*

    Glad to hear you're moving over to gentoo and away from ubuntu. I think both calls are good! ;)

  • (Score: 2) by iWantToKeepAnon on Friday September 29 2017, @09:27PM

    by iWantToKeepAnon (686) Subscriber Badge on Friday September 29 2017, @09:27PM (#575073) Homepage Journal

    Love the site and the community here; thanks for building it and all the blood sweat and tears in the process. :))

    I miss my gentoo workstation, I had a hard disk failure and I needed to get it back up quick so I went with a popular turnkey distro instead of investing the time to get gentoo tuned again. I might have the bandwidth soon to fix that :)

    "Happy families are all alike; every unhappy family is unhappy in its own way." -- Anna Karenina by Leo Tolstoy