Stories
Slash Boxes
Comments

SoylentNews is people

posted by Fnord666 on Thursday August 31 2017, @01:42AM   Printer-friendly

[Update 1]: Fluorine (the web front end) has been back in the rotation since last night and we'll be checking on and bringing up Neon (the db node) tonight. Cross your fingers because if we can't get Neon up and happy by Friday 10:00 PM EDT[*], we'll have to temporarily down the site and copy the db over to our dev server to even keep the site online until we can get a db node back up.

[Update 2]: NC: I successfully CPRed neon, and was able to bring the DB cluster back into sync. I've stopped helium's database services so we're running on neon only now, and getting ready to migrate it after installing updates and such. With luck nothing blows up.

[Update 3]: Nothing blew up. All should be copacetic except for needing to update Neon tomorrow sometime.

* That's the deadline they've given us to move Helium (our other db node) over to the Dallas 2 facility, or they'll do it automatically themselves.

As most of you are already aware, Linode is our web hosting provider. A recent email from them informed us:

We recently announced our new Dallas 2 facility. Over the coming months, we'll be migrating all Linodes to this new, state-of-the-art facility. We're reaching out to let you know your Linode has been entered into a migration queue to move from Dallas 1 to Dallas 2.

We were informed in a separate email that the neon and helium servers were scheduled for an automatic migration. Manual migration was possible, if preferred. That's no big deal as we have redundancy on those servers. The site should continue functioning without a hiccup.

About an hour ago, we received another email saying that fluorine (one of our two web front ends) was also scheduled for migration. That one is a bit more interesting as that server also runs ipnd1 and slashd2 — daemons for which we have no redundancy.

Well, NCommander, TheMightyBuzzard and I happened to be on IRC at the same time as the fluorine migration notice arrived. No time like the present! So fluorine has been migrated. While we were at it, why not migrate neon, too? About 10 minutes later and that was been completed, as well. We discussed whether to migrate helium as well, but decided to hold off.

We did not anticipate any problems... but we found some pages loaded slowly and we were occasionally getting 403 and 503 errors. There are some issues with slower communications between the data centers than what we had within the same data center. Thanks to redundancy, it is not critical we get everything back up and running for the site to run, but it would definitely be best to not run in this configuration indefinitely.

The current state of the world? "one web frontend and one db node are shitting themselves. we're limping along on one of each but with backups in case of emergency." and... "fluorine is technically up but not in the rotation for serving up pages. it's just doing slashd and ipnd."

Hat tip to NCommander and TheMightyBuzzard -- I really enjoy watching these guys in action -- they know their stuff and we are truly fortunate to have them volunteer on SoylentNews.

[1] Instant Payments Notification Daemon
[2] The Daemon that makes it all work


Original Submission

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 2) by edIII on Wednesday August 30 2017, @10:55PM (3 children)

    by edIII (791) on Wednesday August 30 2017, @10:55PM (#561802)

    In all seriousness, I've been trying to find the time to talk to you guys on IRC on how to do it. My new system is Redmond-free as I can make it, and I have some virtualized Tails images that I would like to surf sites from. My first thought was surfing Soylent from hidden services.

    Also, all your work is very appreciated, as well as your talent. Thank you.

    --
    Technically, lunchtime is at any moment. It's just a wave function.
    Starting Score:    1  point
    Karma-Bonus Modifier   +1  

    Total Score:   2  
  • (Score: 2) by The Mighty Buzzard on Wednesday August 30 2017, @11:13PM

    by The Mighty Buzzard (18) Subscriber Badge <themightybuzzard@proton.me> on Wednesday August 30 2017, @11:13PM (#561814) Homepage Journal

    Cheers. It's definitely on the list now but there's things more pressing ahead of it and I won't have time to get to those until at least October I don't expect because I'll be busy writing code for actual moneys. Maybe I can talk NCommander into finding the time to troubleshoot it since he's the one who originally set it up.

    --
    My rights don't end where your fear begins.
  • (Score: 2) by NCommander on Wednesday August 30 2017, @11:30PM (1 child)

    by NCommander (2) Subscriber Badge <michael@casadevall.pro> on Wednesday August 30 2017, @11:30PM (#561828) Homepage Journal

    Drop a message to us in #dev on irc.soylentnews.org. my IRC bouncer is usually connected as is buzzard and a few other places. I admit I don't use the onion site very often at all so if its broken, it sometimes goes unnoticed for awhile.

    --
    Still always moving
    • (Score: 0) by Anonymous Coward on Thursday August 31 2017, @12:33AM

      by Anonymous Coward on Thursday August 31 2017, @12:33AM (#561874)

      Java i2p-router or C++ i2pd.

      I2P needs some love too, and at least so far hasn't had the hidden service attacks that Tor has.