Stories
Slash Boxes
Comments

SoylentNews is people

Meta
posted by Fnord666 on Thursday August 31 2017, @01:42AM   Printer-friendly

[Update 1]: Fluorine (the web front end) has been back in the rotation since last night and we'll be checking on and bringing up Neon (the db node) tonight. Cross your fingers because if we can't get Neon up and happy by Friday 10:00 PM EDT[*], we'll have to temporarily down the site and copy the db over to our dev server to even keep the site online until we can get a db node back up.

[Update 2]: NC: I successfully CPRed neon, and was able to bring the DB cluster back into sync. I've stopped helium's database services so we're running on neon only now, and getting ready to migrate it after installing updates and such. With luck nothing blows up.

[Update 3]: Nothing blew up. All should be copacetic except for needing to update Neon tomorrow sometime.

* That's the deadline they've given us to move Helium (our other db node) over to the Dallas 2 facility, or they'll do it automatically themselves.

As most of you are already aware, Linode is our web hosting provider. A recent email from them informed us:

We recently announced our new Dallas 2 facility. Over the coming months, we'll be migrating all Linodes to this new, state-of-the-art facility. We're reaching out to let you know your Linode has been entered into a migration queue to move from Dallas 1 to Dallas 2.

We were informed in a separate email that the neon and helium servers were scheduled for an automatic migration. Manual migration was possible, if preferred. That's no big deal as we have redundancy on those servers. The site should continue functioning without a hiccup.

About an hour ago, we received another email saying that fluorine (one of our two web front ends) was also scheduled for migration. That one is a bit more interesting as that server also runs ipnd1 and slashd2 — daemons for which we have no redundancy.

Well, NCommander, TheMightyBuzzard and I happened to be on IRC at the same time as the fluorine migration notice arrived. No time like the present! So fluorine has been migrated. While we were at it, why not migrate neon, too? About 10 minutes later and that was been completed, as well. We discussed whether to migrate helium as well, but decided to hold off.

We did not anticipate any problems... but we found some pages loaded slowly and we were occasionally getting 403 and 503 errors. There are some issues with slower communications between the data centers than what we had within the same data center. Thanks to redundancy, it is not critical we get everything back up and running for the site to run, but it would definitely be best to not run in this configuration indefinitely.

The current state of the world? "one web frontend and one db node are shitting themselves. we're limping along on one of each but with backups in case of emergency." and... "fluorine is technically up but not in the rotation for serving up pages. it's just doing slashd and ipnd."

Hat tip to NCommander and TheMightyBuzzard -- I really enjoy watching these guys in action -- they know their stuff and we are truly fortunate to have them volunteer on SoylentNews.

[1] Instant Payments Notification Daemon
[2] The Daemon that makes it all work


Original Submission

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 2) by NCommander on Wednesday August 30 2017, @11:30PM (1 child)

    by NCommander (2) Subscriber Badge <michael@casadevall.pro> on Wednesday August 30 2017, @11:30PM (#561828) Homepage Journal

    Drop a message to us in #dev on irc.soylentnews.org. my IRC bouncer is usually connected as is buzzard and a few other places. I admit I don't use the onion site very often at all so if its broken, it sometimes goes unnoticed for awhile.

    --
    Still always moving
    Starting Score:    1  point
    Karma-Bonus Modifier   +1  

    Total Score:   2  
  • (Score: 0) by Anonymous Coward on Thursday August 31 2017, @12:33AM

    by Anonymous Coward on Thursday August 31 2017, @12:33AM (#561874)

    Java i2p-router or C++ i2pd.

    I2P needs some love too, and at least so far hasn't had the hidden service attacks that Tor has.