Stories
Slash Boxes
Comments

SoylentNews is people

Meta
posted by Fnord666 on Thursday August 31 2017, @01:42AM   Printer-friendly

[Update 1]: Fluorine (the web front end) has been back in the rotation since last night and we'll be checking on and bringing up Neon (the db node) tonight. Cross your fingers because if we can't get Neon up and happy by Friday 10:00 PM EDT[*], we'll have to temporarily down the site and copy the db over to our dev server to even keep the site online until we can get a db node back up.

[Update 2]: NC: I successfully CPRed neon, and was able to bring the DB cluster back into sync. I've stopped helium's database services so we're running on neon only now, and getting ready to migrate it after installing updates and such. With luck nothing blows up.

[Update 3]: Nothing blew up. All should be copacetic except for needing to update Neon tomorrow sometime.

* That's the deadline they've given us to move Helium (our other db node) over to the Dallas 2 facility, or they'll do it automatically themselves.

As most of you are already aware, Linode is our web hosting provider. A recent email from them informed us:

We recently announced our new Dallas 2 facility. Over the coming months, we'll be migrating all Linodes to this new, state-of-the-art facility. We're reaching out to let you know your Linode has been entered into a migration queue to move from Dallas 1 to Dallas 2.

We were informed in a separate email that the neon and helium servers were scheduled for an automatic migration. Manual migration was possible, if preferred. That's no big deal as we have redundancy on those servers. The site should continue functioning without a hiccup.

About an hour ago, we received another email saying that fluorine (one of our two web front ends) was also scheduled for migration. That one is a bit more interesting as that server also runs ipnd1 and slashd2 — daemons for which we have no redundancy.

Well, NCommander, TheMightyBuzzard and I happened to be on IRC at the same time as the fluorine migration notice arrived. No time like the present! So fluorine has been migrated. While we were at it, why not migrate neon, too? About 10 minutes later and that was been completed, as well. We discussed whether to migrate helium as well, but decided to hold off.

We did not anticipate any problems... but we found some pages loaded slowly and we were occasionally getting 403 and 503 errors. There are some issues with slower communications between the data centers than what we had within the same data center. Thanks to redundancy, it is not critical we get everything back up and running for the site to run, but it would definitely be best to not run in this configuration indefinitely.

The current state of the world? "one web frontend and one db node are shitting themselves. we're limping along on one of each but with backups in case of emergency." and... "fluorine is technically up but not in the rotation for serving up pages. it's just doing slashd and ipnd."

Hat tip to NCommander and TheMightyBuzzard -- I really enjoy watching these guys in action -- they know their stuff and we are truly fortunate to have them volunteer on SoylentNews.

[1] Instant Payments Notification Daemon
[2] The Daemon that makes it all work


Original Submission

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 2, Informative) by Anonymous Coward on Tuesday August 29 2017, @11:38AM (4 children)

    by Anonymous Coward on Tuesday August 29 2017, @11:38AM (#560762)

    Yo! This could be fixed (permanently) https://soylentnews.org/comments.pl?noupdate=1&sid=21257&page=1&cid=558747#commentwrap [soylentnews.org]

    That's having open tags in an article causes the front page to go wonky. Maybe there could be a check. I've personally seen that happen twice.

    Cheers. Consider yourself served.

    Starting Score:    0  points
    Moderation   +2  
       Informative=2, Total=2
    Extra 'Informative' Modifier   0  

    Total Score:   2  
  • (Score: 2) by The Mighty Buzzard on Tuesday August 29 2017, @03:36PM (3 children)

    by The Mighty Buzzard (18) Subscriber Badge <themightybuzzard@proton.me> on Tuesday August 29 2017, @03:36PM (#560848) Homepage Journal

    We actually do check for balanced tags in submissions, comments, and journal entries, then balance them if they aren't. What we don't do is check that our editors didn't leave one open when they monkey with a story. Guess we need to start.

    --
    My rights don't end where your fear begins.
    • (Score: 2) by CoolHand on Tuesday August 29 2017, @03:50PM (2 children)

      by CoolHand (438) on Tuesday August 29 2017, @03:50PM (#560861) Journal
      Yeah, some of us editors ain't too bright... but we're strong as an ox!
      --
      Anyone who is capable of getting themselves made President should on no account be allowed to do the job-Douglas Adams
      • (Score: 2) by The Mighty Buzzard on Tuesday August 29 2017, @06:07PM (1 child)

        by The Mighty Buzzard (18) Subscriber Badge <themightybuzzard@proton.me> on Tuesday August 29 2017, @06:07PM (#560978) Homepage Journal

        We love yall anyway.

        --
        My rights don't end where your fear begins.
        • (Score: 0) by Anonymous Coward on Wednesday August 30 2017, @12:36PM

          by Anonymous Coward on Wednesday August 30 2017, @12:36PM (#561459)

          Thanks for the quick reply. Love this place, it's incredibly good. Keep on humping! ++