Stories
Slash Boxes
Comments

SoylentNews is people

Meta
posted by Fnord666 on Thursday August 31 2017, @01:42AM   Printer-friendly

[Update 1]: Fluorine (the web front end) has been back in the rotation since last night and we'll be checking on and bringing up Neon (the db node) tonight. Cross your fingers because if we can't get Neon up and happy by Friday 10:00 PM EDT[*], we'll have to temporarily down the site and copy the db over to our dev server to even keep the site online until we can get a db node back up.

[Update 2]: NC: I successfully CPRed neon, and was able to bring the DB cluster back into sync. I've stopped helium's database services so we're running on neon only now, and getting ready to migrate it after installing updates and such. With luck nothing blows up.

[Update 3]: Nothing blew up. All should be copacetic except for needing to update Neon tomorrow sometime.

* That's the deadline they've given us to move Helium (our other db node) over to the Dallas 2 facility, or they'll do it automatically themselves.

As most of you are already aware, Linode is our web hosting provider. A recent email from them informed us:

We recently announced our new Dallas 2 facility. Over the coming months, we'll be migrating all Linodes to this new, state-of-the-art facility. We're reaching out to let you know your Linode has been entered into a migration queue to move from Dallas 1 to Dallas 2.

We were informed in a separate email that the neon and helium servers were scheduled for an automatic migration. Manual migration was possible, if preferred. That's no big deal as we have redundancy on those servers. The site should continue functioning without a hiccup.

About an hour ago, we received another email saying that fluorine (one of our two web front ends) was also scheduled for migration. That one is a bit more interesting as that server also runs ipnd1 and slashd2 — daemons for which we have no redundancy.

Well, NCommander, TheMightyBuzzard and I happened to be on IRC at the same time as the fluorine migration notice arrived. No time like the present! So fluorine has been migrated. While we were at it, why not migrate neon, too? About 10 minutes later and that was been completed, as well. We discussed whether to migrate helium as well, but decided to hold off.

We did not anticipate any problems... but we found some pages loaded slowly and we were occasionally getting 403 and 503 errors. There are some issues with slower communications between the data centers than what we had within the same data center. Thanks to redundancy, it is not critical we get everything back up and running for the site to run, but it would definitely be best to not run in this configuration indefinitely.

The current state of the world? "one web frontend and one db node are shitting themselves. we're limping along on one of each but with backups in case of emergency." and... "fluorine is technically up but not in the rotation for serving up pages. it's just doing slashd and ipnd."

Hat tip to NCommander and TheMightyBuzzard -- I really enjoy watching these guys in action -- they know their stuff and we are truly fortunate to have them volunteer on SoylentNews.

[1] Instant Payments Notification Daemon
[2] The Daemon that makes it all work


Original Submission

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 3, Interesting) by martyb on Tuesday August 29 2017, @03:28AM (1 child)

    by martyb (76) Subscriber Badge on Tuesday August 29 2017, @03:28AM (#560658) Journal

    Dunno what ever gave you that idea... I just poke things with a stick until they break, sharpen the stick, and poke them some more until they start working again.

    The trick is in the knowing where to poke, how often, how hard, etc. Plain and simple: you've got good aim! I've witnessed some mighty fine root problem analysis sessions by you and NC and PJ, AG, Dec, MJ, et al. on IRC! The problem identification, fault isolation, and bug fixing has been a joy to behold.

    --
    Wit is intellect, dancing.
    Starting Score:    1  point
    Moderation   +1  
       Interesting=1, Total=1
    Extra 'Interesting' Modifier   0  
    Karma-Bonus Modifier   +1  

    Total Score:   3  
  • (Score: 5, Informative) by The Mighty Buzzard on Tuesday August 29 2017, @03:44AM

    by The Mighty Buzzard (18) Subscriber Badge <themightybuzzard@proton.me> on Tuesday August 29 2017, @03:44AM (#560661) Homepage Journal

    I still maintain that the new datacenter is in Houston not Dallas. It's certainly dropping packets like it's under several feet of water.

    --
    My rights don't end where your fear begins.