Stories
Slash Boxes
Comments

SoylentNews is people

Meta
posted by Fnord666 on Thursday August 31 2017, @01:42AM   Printer-friendly

[Update 1]: Fluorine (the web front end) has been back in the rotation since last night and we'll be checking on and bringing up Neon (the db node) tonight. Cross your fingers because if we can't get Neon up and happy by Friday 10:00 PM EDT[*], we'll have to temporarily down the site and copy the db over to our dev server to even keep the site online until we can get a db node back up.

[Update 2]: NC: I successfully CPRed neon, and was able to bring the DB cluster back into sync. I've stopped helium's database services so we're running on neon only now, and getting ready to migrate it after installing updates and such. With luck nothing blows up.

[Update 3]: Nothing blew up. All should be copacetic except for needing to update Neon tomorrow sometime.

* That's the deadline they've given us to move Helium (our other db node) over to the Dallas 2 facility, or they'll do it automatically themselves.

As most of you are already aware, Linode is our web hosting provider. A recent email from them informed us:

We recently announced our new Dallas 2 facility. Over the coming months, we'll be migrating all Linodes to this new, state-of-the-art facility. We're reaching out to let you know your Linode has been entered into a migration queue to move from Dallas 1 to Dallas 2.

We were informed in a separate email that the neon and helium servers were scheduled for an automatic migration. Manual migration was possible, if preferred. That's no big deal as we have redundancy on those servers. The site should continue functioning without a hiccup.

About an hour ago, we received another email saying that fluorine (one of our two web front ends) was also scheduled for migration. That one is a bit more interesting as that server also runs ipnd1 and slashd2 — daemons for which we have no redundancy.

Well, NCommander, TheMightyBuzzard and I happened to be on IRC at the same time as the fluorine migration notice arrived. No time like the present! So fluorine has been migrated. While we were at it, why not migrate neon, too? About 10 minutes later and that was been completed, as well. We discussed whether to migrate helium as well, but decided to hold off.

We did not anticipate any problems... but we found some pages loaded slowly and we were occasionally getting 403 and 503 errors. There are some issues with slower communications between the data centers than what we had within the same data center. Thanks to redundancy, it is not critical we get everything back up and running for the site to run, but it would definitely be best to not run in this configuration indefinitely.

The current state of the world? "one web frontend and one db node are shitting themselves. we're limping along on one of each but with backups in case of emergency." and... "fluorine is technically up but not in the rotation for serving up pages. it's just doing slashd and ipnd."

Hat tip to NCommander and TheMightyBuzzard -- I really enjoy watching these guys in action -- they know their stuff and we are truly fortunate to have them volunteer on SoylentNews.

[1] Instant Payments Notification Daemon
[2] The Daemon that makes it all work


Original Submission

This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
(1)
  • (Score: 5, Informative) by Gaaark on Tuesday August 29 2017, @03:16AM (1 child)

    by Gaaark (41) Subscriber Badge on Tuesday August 29 2017, @03:16AM (#560649) Homepage Journal

    All the whining and shit comments aside, I think we have a pretty nice site here.

    Thanks to all the volunteers, etc.

    You're doing a good job.

    --
    --- That's not flying: that's... falling... with more luck than I have. ---
    • (Score: 2, Interesting) by Ethanol-fueled on Tuesday August 29 2017, @04:02AM

      by Ethanol-fueled (2792) Subscriber Badge on Tuesday August 29 2017, @04:02AM (#560664) Homepage

      Yep, one of the reasons why I hang out here a lot during my downtime is not only because of the free speech but clean and quick operation that isn't shitted-up by all kinds of crap, as well as due to the aggressive actions of the staff to fix problems as they happen. Hell, I can even excuse the fact that it's written in Perl.

      Now if only I could have a job with sane deadlines that allows me to post from work again without getting into too much shit, I'd have the energy to post more during weekdays and even write some damn good submissions.

  • (Score: 4, Funny) by The Mighty Buzzard on Tuesday August 29 2017, @03:20AM (5 children)

    Dunno what ever gave you that idea... I just poke things with a stick until they break, sharpen the stick, and poke them some more until they start working again.

    --
    Cobra Kai
    • (Score: 3, Interesting) by Gaaark on Tuesday August 29 2017, @03:25AM (2 children)

      by Gaaark (41) Subscriber Badge on Tuesday August 29 2017, @03:25AM (#560656) Homepage Journal

      Sounds like my honeymoon. XD

      --
      --- That's not flying: that's... falling... with more luck than I have. ---
      • (Score: 3, Touché) by MostCynical on Tuesday August 29 2017, @04:12AM (1 child)

        by MostCynical (2589) on Tuesday August 29 2017, @04:12AM (#560666)

        ...including the sharpening??

        --
        (Score: tau, Irrational)
        • (Score: 2) by Gaaark on Tuesday August 29 2017, @02:25PM

          by Gaaark (41) Subscriber Badge on Tuesday August 29 2017, @02:25PM (#560823) Homepage Journal

          "sharpening"..."fluffing": same same! :)

          --
          --- That's not flying: that's... falling... with more luck than I have. ---
    • (Score: 3, Interesting) by martyb on Tuesday August 29 2017, @03:28AM (1 child)

      by martyb (76) Subscriber Badge on Tuesday August 29 2017, @03:28AM (#560658) Journal

      Dunno what ever gave you that idea... I just poke things with a stick until they break, sharpen the stick, and poke them some more until they start working again.

      The trick is in the knowing where to poke, how often, how hard, etc. Plain and simple: you've got good aim! I've witnessed some mighty fine root problem analysis sessions by you and NC and PJ, AG, Dec, MJ, et al. on IRC! The problem identification, fault isolation, and bug fixing has been a joy to behold.

      --
      Wit is intellect, dancing.
  • (Score: 3, Funny) by DonkeyChan on Tuesday August 29 2017, @03:47AM (2 children)

    by DonkeyChan (5551) on Tuesday August 29 2017, @03:47AM (#560663)

    See subject

    • (Score: 2) by realDonaldTrump on Wednesday August 30 2017, @10:55PM

      by realDonaldTrump (6614) Subscriber Badge on Wednesday August 30 2017, @10:55PM (#561801) Homepage Journal

      I went to pages but they were blank. I touch Refresh, still blank. I touch Refresh again, PERFECT, there's the page! Not bad for an alt-left site. 🇺🇸

      --
      Text TRUMP to 88022 to join the 🚂 #TrumpTrain [facebook.com]
    • (Score: 0) by Anonymous Coward on Thursday August 31 2017, @12:56AM

      by Anonymous Coward on Thursday August 31 2017, @12:56AM (#561889)

      High fives must be 0.55555, eh?
      Not very good compared to high nines, 0.99999
      (grin)

  • (Score: 3, Interesting) by MostCynical on Tuesday August 29 2017, @04:02AM (1 child)

    by MostCynical (2589) on Tuesday August 29 2017, @04:02AM (#560665)

    always amazed by how well you guys pull off major system changes and updates (and migrations!) with barely noticeable effects on us ornery users.

    --
    (Score: tau, Irrational)
    • (Score: 3, Informative) by NCommander on Tuesday August 29 2017, @06:37AM

      by NCommander (2) Subscriber Badge <mcasadevall@soylentnews.org> on Tuesday August 29 2017, @06:37AM (#560691) Homepage Journal

      Redundant webservers for the win.

      Our general rule is never let one server's load go over 50%, for cases when we blow one up, though this is a rare instance of blowing two up (web frontend + db backend).

      --
      Still always moving
  • (Score: 2, Interesting) by pTamok on Tuesday August 29 2017, @07:33AM

    by pTamok (3042) on Tuesday August 29 2017, @07:33AM (#560704)

    Thank-you for doing such a good job.

    I very much appreciate the time and trouble you guys put into this place - you are in a very small and select group of websites I support with money and not just goodwill.

    Long may you continue to provide a shining example of doing things properly.

  • (Score: 2, Informative) by Anonymous Coward on Tuesday August 29 2017, @11:38AM (4 children)

    by Anonymous Coward on Tuesday August 29 2017, @11:38AM (#560762)

    Yo! This could be fixed (permanently) https://soylentnews.org/comments.pl?noupdate=1&sid=21257&page=1&cid=558747#commentwrap [soylentnews.org]

    That's having open tags in an article causes the front page to go wonky. Maybe there could be a check. I've personally seen that happen twice.

    Cheers. Consider yourself served.

    • (Score: 2) by The Mighty Buzzard on Tuesday August 29 2017, @03:36PM (3 children)

      We actually do check for balanced tags in submissions, comments, and journal entries, then balance them if they aren't. What we don't do is check that our editors didn't leave one open when they monkey with a story. Guess we need to start.

      --
      Cobra Kai
      • (Score: 2) by CoolHand on Tuesday August 29 2017, @03:50PM (2 children)

        by CoolHand (438) Subscriber Badge on Tuesday August 29 2017, @03:50PM (#560861) Journal
        Yeah, some of us editors ain't too bright... but we're strong as an ox!
        --
        Anyone who is capable of getting themselves made President should on no account be allowed to do the job-Douglas Adams
  • (Score: 1) by AlwaysNever on Tuesday August 29 2017, @09:51PM (2 children)

    by AlwaysNever (5817) on Tuesday August 29 2017, @09:51PM (#561175)

    What infrastructure does Linode run on: VMware, Linux-KVM, XenServer...?

  • (Score: 1, Interesting) by Anonymous Coward on Tuesday August 29 2017, @10:27PM (5 children)

    by Anonymous Coward on Tuesday August 29 2017, @10:27PM (#561195)

    Great work! The main HTTPS site runs well both technically and participation-wise, as far as I am concerned.

    The onion site ( http://7rmath4ro2of2a42.onion/ [7rmath4ro2of2a42.onion] ) however seems to need some love. Via the onion site, the pages often take minutes to load and are usually locked into one of the sections ( meta, politics., breaking, etc. ) where it's not possible to get to the main section.

    • (Score: 2) by The Mighty Buzzard on Wednesday August 30 2017, @10:16PM (4 children)

      Damnit, you're going to make me configure my desktop to use ToR and resolve .onion addresses, aren't you?

      --
      Cobra Kai
      • (Score: 2) by edIII on Wednesday August 30 2017, @10:55PM (3 children)

        by edIII (791) Subscriber Badge on Wednesday August 30 2017, @10:55PM (#561802)

        In all seriousness, I've been trying to find the time to talk to you guys on IRC on how to do it. My new system is Redmond-free as I can make it, and I have some virtualized Tails images that I would like to surf sites from. My first thought was surfing Soylent from hidden services.

        Also, all your work is very appreciated, as well as your talent. Thank you.

        • (Score: 2) by The Mighty Buzzard on Wednesday August 30 2017, @11:13PM

          Cheers. It's definitely on the list now but there's things more pressing ahead of it and I won't have time to get to those until at least October I don't expect because I'll be busy writing code for actual moneys. Maybe I can talk NCommander into finding the time to troubleshoot it since he's the one who originally set it up.

          --
          Cobra Kai
        • (Score: 2) by NCommander on Wednesday August 30 2017, @11:30PM (1 child)

          by NCommander (2) Subscriber Badge <mcasadevall@soylentnews.org> on Wednesday August 30 2017, @11:30PM (#561828) Homepage Journal

          Drop a message to us in #dev on irc.soylentnews.org. my IRC bouncer is usually connected as is buzzard and a few other places. I admit I don't use the onion site very often at all so if its broken, it sometimes goes unnoticed for awhile.

          --
          Still always moving
          • (Score: 0) by Anonymous Coward on Thursday August 31 2017, @12:33AM

            by Anonymous Coward on Thursday August 31 2017, @12:33AM (#561874)

            Java i2p-router or C++ i2pd.

            I2P needs some love too, and at least so far hasn't had the hidden service attacks that Tor has.

(1)