SoylentNews

We've discovered over the weekend that soylentnews.org was failing to resolve with some DNSSEC enabled resolvers. After debugging and manually checking our setup, the problem appears to be occurring due to an issue with the Linode DNS servers when accessed over IPv6. As such, some users may experience slow waiting times due to these DNS issues. I have filed a ticket with Linode about this, and will keep you guys up to date.

73 de NCommander

martyb writes:

Nearly two months ago, we received notice from Linode (which hosts the servers for SoylentNews) that they would be migrating our servers to a new data center in Dallas, TX. Our systems would gradually be scheduled for migration. We could either accept their scheduled date/time or trigger a manual migration. In theory, this should be a no-worry activity as we have redundancy on almost all of our servers and processes. But in practice, that is not always the case. Rather than take our chances, we were proactive and manually performed migrations as they became possible.

We had a couple hiccups with one server, but with NCommander, TMB, PJ on hand (among others), we were able to get that one straightened out with only limited impact to the site. We also lost access to our IRC server for about 20 minutes when that server was migrated.

So, with that backdrop, I'm pleased to announce that we completed the migration of our last Linode (hydrogen) to the new data center in Dallas this morning! Shoutout to TheMightyBuzzard for tweaking our load balancer to facilitate the migration, and for being on hand had things gone sideways.

Original Submission

martyb writes:

As part of Linode's migration of servers to a new Data Center in Dallas, two of our servers were scheduled for migration at 10pm EDT on September 29, 2017. NCommander happened to be around when I sent out a reminder I'd received from Linode, so he 'hit the button' at 9:30pm tonight (Sept. 28) and did a manual migration ahead of time.

Unless you were on our IRC server (Internet Relay Chat) at the time, you probably didn't even notice... and even then, it was unavailable for only about 15-20 minutes. Redundancy for the win!

That leaves us with a single server, sodium, to migrate. It is currently scheduled for migration on Tuesday, October 3, 2017 at 10:00pm EDT. Since sodium is one of two front-end proxies for us (the other is magnesium which has already been migrated), I expect we'll be able to perform that migration without any site interruption.

Separately, and in parallel, we are slowly moving our servers from Ubuntu 14.04 LTS to Gentoo.

To the community, thank you for your patience as we work our way through this process. And, for those of you who may have been with us from the outset, and when up-time was measured in hours, please join me in congratulating the team for their dedication and hard work which has facilitated such an uneventful migration!

Original Submission

Just a quick heads up to the SN community. As we previously announced, Linode is migrating customers to a new data center. We already did the first stage of migration with most of the production servers two weeks ago. Now we're working our way through the remainder of the servers. As of this writing, we've migrated both webservers, both DB servers, our development server, and the fallback load balancer.

Tonight at approximately midnight EDT, we're going to migrate beryllium, which hosts our IRC server, wiki, and mail server, and boron, which is our redundant KDC/internal DNS server. During this process, IRC and email from SoylentNews will be unavailable. The site itself will stay up during this process.

After this migration, we'll only have our primary load balancer to migrate, which we will likely do over the weekend. Thank you all for your understanding.

~ NCommander

[Update 1]: Fluorine (the web front end) has been back in the rotation since last night and we'll be checking on and bringing up Neon (the db node) tonight. Cross your fingers because if we can't get Neon up and happy by Friday 10:00 PM EDT^[*], we'll have to temporarily down the site and copy the db over to our dev server to even keep the site online until we can get a db node back up.

[Update 2]: NC: I successfully CPRed neon, and was able to bring the DB cluster back into sync. I've stopped helium's database services so we're running on neon only now, and getting ready to migrate it after installing updates and such. With luck nothing blows up.

[Update 3]: Nothing blew up. All should be copacetic except for needing to update Neon tomorrow sometime.

* That's the deadline they've given us to move Helium (our other db node) over to the Dallas 2 facility, or they'll do it automatically themselves.

martyb writes:

As most of you are already aware, Linode is our web hosting provider. A recent email from them informed us:

We recently announced our new Dallas 2 facility. Over the coming months, we'll be migrating all Linodes to this new, state-of-the-art facility. We're reaching out to let you know your Linode has been entered into a migration queue to move from Dallas 1 to Dallas 2.

We were informed in a separate email that the neon and helium servers were scheduled for an automatic migration. Manual migration was possible, if preferred. That's no big deal as we have redundancy on those servers. The site should continue functioning without a hiccup.

About an hour ago, we received another email saying that fluorine (one of our two web front ends) was also scheduled for migration. That one is a bit more interesting as that server also runs ipnd¹ and slashd² — daemons for which we have no redundancy.

Well, NCommander, TheMightyBuzzard and I happened to be on IRC at the same time as the fluorine migration notice arrived. No time like the present! So fluorine has been migrated. While we were at it, why not migrate neon, too? About 10 minutes later and that was been completed, as well. We discussed whether to migrate helium as well, but decided to hold off.

We did not anticipate any problems... but we found some pages loaded slowly and we were occasionally getting 403 and 503 errors. There are some issues with slower communications between the data centers than what we had within the same data center. Thanks to redundancy, it is not critical we get everything back up and running for the site to run, but it would definitely be best to not run in this configuration indefinitely.

The current state of the world? "one web frontend and one db node are shitting themselves. we're limping along on one of each but with backups in case of emergency." and... "fluorine is technically up but not in the rotation for serving up pages. it's just doing slashd and ipnd."

Hat tip to NCommander and TheMightyBuzzard -- I really enjoy watching these guys in action -- they know their stuff and we are truly fortunate to have them volunteer on SoylentNews.

[1] Instant Payments Notification Daemon
[2] The Daemon that makes it all work

Original Submission

The Mighty Buzzard writes:

Welcome, new trolls! We're pleased as punch to have you aboard, unfortunately as you may have noticed our moderators are unable to give you the moderations you've been working so hard for. Since we can't really do much about people not moderating more, we're going to be giving out more points so that the ones that do can give you the attention you so desperately crave.

Moderators: Starting a little after midnight UTC tonight, everyone will be getting ten points a day instead of five. The threshold for a mod-bomb, however, is going to remain at five. This change is not so you can pursue an agenda against registered users more effectively but so we can collectively handle the rather large uptick in anonymous trolling recently while still being able to have points remaining for upmodding quality comments. This is not an invitation to go wild downmodding; it's helping you to be able to stick to the "concentrate more on upmodding than downmodding" bit of the guidelines.

Also, this is not a heavily thought-out or permanent change. It is a quick, dirty adjustment that will be reviewed, tweaked, and likely changed before year's end. Questions? Comments?