2017-07-01 00:00:00 ..
2017-11-10 23:13:37 UTC
2017-11-10 20:41:20 UTC
We always have a place for talented people, visit the Get Involved section on the wiki to see how you can make SoylentNews better.
We've discovered over the weekend that soylentnews.org was failing to resolve with some DNSSEC enabled resolvers. After debugging and manually checking our setup, the problem appears to be occurring due to an issue with the Linode DNS servers when accessed over IPv6. As such, some users may experience slow waiting times due to these DNS issues. I have filed a ticket with Linode about this, and will keep you guys up to date.
73 de NCommander
Nearly two months ago, we received notice from Linode (which hosts the servers for SoylentNews) that they would be migrating our servers to a new data center in Dallas, TX. Our systems would gradually be scheduled for migration. We could either accept their scheduled date/time or trigger a manual migration. In theory, this should be a no-worry activity as we have redundancy on almost all of our servers and processes. But in practice, that is not always the case. Rather than take our chances, we were proactive and manually performed migrations as they became possible.
We had a couple hiccups with one server, but with NCommander, TMB, PJ on hand (among others), we were able to get that one straightened out with only limited impact to the site. We also lost access to our IRC server for about 20 minutes when that server was migrated.
So, with that backdrop, I'm pleased to announce that we completed the migration of our last Linode (hydrogen) to the new data center in Dallas this morning! Shoutout to TheMightyBuzzard for tweaking our load balancer to facilitate the migration, and for being on hand had things gone sideways.
As part of Linode's migration of servers to a new Data Center in Dallas, two of our servers were scheduled for migration at 10pm EDT on September 29, 2017. NCommander happened to be around when I sent out a reminder I'd received from Linode, so he 'hit the button' at 9:30pm tonight (Sept. 28) and did a manual migration ahead of time.
Unless you were on our IRC server (Internet Relay Chat) at the time, you probably didn't even notice... and even then, it was unavailable for only about 15-20 minutes. Redundancy for the win!
That leaves us with a single server, sodium, to migrate. It is currently scheduled for migration on Tuesday, October 3, 2017 at 10:00pm EDT. Since sodium is one of two front-end proxies for us (the other is magnesium which has already been migrated), I expect we'll be able to perform that migration without any site interruption.
Separately, and in parallel, we are slowly moving our servers from Ubuntu 14.04 LTS to Gentoo.
To the community, thank you for your patience as we work our way through this process. And, for those of you who may have been with us from the outset, and when up-time was measured in hours, please join me in congratulating the team for their dedication and hard work which has facilitated such an uneventful migration!
Just a quick heads up to the SN community. As we previously announced, Linode is migrating customers to a new data center. We already did the first stage of migration with most of the production servers two weeks ago. Now we're working our way through the remainder of the servers. As of this writing, we've migrated both webservers, both DB servers, our development server, and the fallback load balancer.
Tonight at approximately midnight EDT, we're going to migrate beryllium, which hosts our IRC server, wiki, and mail server, and boron, which is our redundant KDC/internal DNS server. During this process, IRC and email from SoylentNews will be unavailable. The site itself will stay up during this process.
After this migration, we'll only have our primary load balancer to migrate, which we will likely do over the weekend. Thank you all for your understanding.
[Update 1]: Fluorine (the web front end) has been back in the rotation since last night and we'll be checking on and bringing up Neon (the db node) tonight. Cross your fingers because if we can't get Neon up and happy by Friday 10:00 PM EDT[*], we'll have to temporarily down the site and copy the db over to our dev server to even keep the site online until we can get a db node back up.
[Update 2]: NC: I successfully CPRed neon, and was able to bring the DB cluster back into sync. I've stopped helium's database services so we're running on neon only now, and getting ready to migrate it after installing updates and such. With luck nothing blows up.
[Update 3]: Nothing blew up. All should be copacetic except for needing to update Neon tomorrow sometime.
* That's the deadline they've given us to move Helium (our other db node) over to the Dallas 2 facility, or they'll do it automatically themselves.
As most of you are already aware, Linode is our web hosting provider. A recent email from them informed us:
We recently announced our new Dallas 2 facility. Over the coming months, we'll be migrating all Linodes to this new, state-of-the-art facility. We're reaching out to let you know your Linode has been entered into a migration queue to move from Dallas 1 to Dallas 2.
We were informed in a separate email that the neon and helium servers were scheduled for an automatic migration. Manual migration was possible, if preferred. That's no big deal as we have redundancy on those servers. The site should continue functioning without a hiccup.
About an hour ago, we received another email saying that fluorine (one of our two web front ends) was also scheduled for migration. That one is a bit more interesting as that server also runs ipnd1 and slashd2 — daemons for which we have no redundancy.
Well, NCommander, TheMightyBuzzard and I happened to be on IRC at the same time as the fluorine migration notice arrived. No time like the present! So fluorine has been migrated. While we were at it, why not migrate neon, too? About 10 minutes later and that was been completed, as well. We discussed whether to migrate helium as well, but decided to hold off.
We did not anticipate any problems... but we found some pages loaded slowly and we were occasionally getting 403 and 503 errors. There are some issues with slower communications between the data centers than what we had within the same data center. Thanks to redundancy, it is not critical we get everything back up and running for the site to run, but it would definitely be best to not run in this configuration indefinitely.
The current state of the world? "one web frontend and one db node are shitting themselves. we're limping along on one of each but with backups in case of emergency." and... "fluorine is technically up but not in the rotation for serving up pages. it's just doing slashd and ipnd."
Hat tip to NCommander and TheMightyBuzzard -- I really enjoy watching these guys in action -- they know their stuff and we are truly fortunate to have them volunteer on SoylentNews.
 Instant Payments Notification Daemon
 The Daemon that makes it all work
Welcome, new trolls! We're pleased as punch to have you aboard, unfortunately as you may have noticed our moderators are unable to give you the moderations you've been working so hard for. Since we can't really do much about people not moderating more, we're going to be giving out more points so that the ones that do can give you the attention you so desperately crave.
Moderators: Starting a little after midnight UTC tonight, everyone will be getting ten points a day instead of five. The threshold for a mod-bomb, however, is going to remain at five. This change is not so you can pursue an agenda against registered users more effectively but so we can collectively handle the rather large uptick in anonymous trolling recently while still being able to have points remaining for upmodding quality comments. This is not an invitation to go wild downmodding; it's helping you to be able to stick to the "concentrate more on upmodding than downmodding" bit of the guidelines.
Also, this is not a heavily thought-out or permanent change. It is a quick, dirty adjustment that will be reviewed, tweaked, and likely changed before year's end. Questions? Comments?
This is a meta post concerning Soylentnews' background, finances, operations, staffing, story scheduling, and a conclusion. If this is not your cup-of-coffee++ (or tea, etc.), then please ignore this story — another will appear shortly.
In February of 2014, a group of ticked-off Slashdot users got together, said "Fuck Beta!", and launched an alternative web site focused on the community. It started with an out-of-date and unmaintained open source version of slashcode which was promptly forked and renamed 'Rehash'. We incorporated as a Public Benefit Corporation. We experienced site outages, questions of leadership, and faced predictions of failure. Thanks to persistence, dedication, many late nights (and some very early mornings), we persevered and are still here today.
Soylentnews is a place for people to engage in discussions about topics of interest to the community. Not all topics are of interest to everyone, of course. In large part it is up to the community to submit stories — the large majority of these do get accepted to the main page. This is all the more important during the "silly season" &mash; summer in the northern hemisphere — when many people are on vacation and fewer scholarly articles are published.
We are still an all-volunteer organization. Nobody here has made a profit off this site. In fact, Soylentnews is still in debt to the founders who put up the funds required to get us up and running. I am happy to report that we have finally made enough progress that some payback to the founders may be possible.
Here are the unaudited numbers from site subscriptions for the first half of our fiscal year (2017-01-01 through 2017-06-30):
Base goal: $3000
Stretch goal: $2000
Subscription count: 133
Gross subscription income : $3795
Net subscription income: $3645 (estimated - after payment processor fees)
Net over goal: $645
So, thanks to all you Soylentils who have donated, we have a surplus at the moment. The ultimate decision is up to the Board of Directors, but the current sense is that we should build a prudent reserve of some months' operating expenses before paying back the founders. In light of the foregoing, we are aiming for the same fundraising goals for the second half of the year... $3,000 base and $2,000 stretch goals. More in line with business norms, however, these are now being presented in the "Site News" box as quarterly goals: $1,500 base and $1,000 stretch goals, respectively.
We've been forthright and upfront right from the start and it is our continued commitment to keep you informed of any issues in the site's operations.
To wit, we recently received a notice from our web-hosting provider, Linode, that one of our servers had been reported as having been added to a spam-blocking list. Staff immediately responded and found a misconfiguration in our link-shortening service. (It was only supposed to shorten links originating on Soylentnews.org, but was accepting links for other domains, as well.) A dump of the database was taken, non-SN sites were purged, the shortening service was updated to correctly implement the restriction to only shorten links from soylentnews.org, and Linode was informed of these actions.
We also recently experienced a problem with our slashd daemon which, among many other tasks, hands out moderation points each night. This fell over on us for a couple of nights leading to our handing out mod points manually to all users. This seems to have been rectified — please let us know if you see a recurrence.
Lastly, one of the senior editorial staff has been on hiatus to deal with major illnesses in his family. His dedicated efforts in helping them has brought ill health upon himself, as well. I ask you to keep janrinok and his family in your thoughts and, if you are of a mind to do so, in your prayers.
There have been discussions in the past as how we should best handle circumstances when there is a dearth of acceptable stories in the queue. Do we post something marginal just to fill the time or should we hold out and only publish when we have enough suitable material to publish. Past efforts and comments have suggested the majority prefer we avoid posting stories just to fill time slots. In short: quality over quantity. Further, staff cannot work 24/7/365 without a break either. We all need a break sometimes and summer is a good time to take one. In other words, we have been running with reduced staffing for the past couple of months and will continue to do so for the next few months as well.
The result? Over the past month or so, we have experimented with further spacing out stories on holidays (Independence Day in the USA) and on weekends. Instead of the usual cadence of a story appearing every 90 minutes or so, we have tried slowing to posting a story every 2 hours or even every 2.5 hours.
My perception is that this has worked okay. At least I have not noticed any complaints in the comments. It could well be that I had missed something, too. So I put this question to the community: How has the story spacing been working out?
Please keep those story submissions coming, please continue to subscribe (you can offer more than the minimum suggested amount), and — most importantly — please keep reading and commenting! Discussion is
[Update: It looks like the slashd daemon which was scheduled for 2017-06-29_00:10:00 failed to run again. We are investigating.]
We ran into a problem with the process which hands out moderation points each night.
We received a couple reports that people had no mod points. A query of our DB showed that fully 80% of our users had the full complement of 5 mod points. That seemed strange — we have a daemon that runs every night and that, among many other things, replenishes your supply each morning at 00:10 UTC. Apparently that process fell over and went toes up. Complicating matters, if you had mod points left over from the prior day, those were still available to you.
I put out the call to the devs as I tried to sleuth out what was going on. Many thanks to mrpg who played guinea pig and offered a fresh perspective as we tried to isolate the issue. With the information that was gathered, TheMightyBuzzard and paulej72 quickly figured out what happened. Further, rather than wait for tonight's process to run, TheMightyBuzzard manually updated the DB and handed out mod points to everyone.
(Debugging was complicated by the fact that there was another issue that was clogging up the logs which made it doubly hard to determine what happened. Debugging efforts are continuing on that matter.)
We anticipate things should be back to normal tomorrow. We'll check in on this story first thing in the morning so if you run into any issues with this, please post a reply with details.
P.S. I remember in the early days of this site when more than 12 hours of continuous up-time was an accomplishment. It's a credit to the staff here that we are now at a state where a system issue is a rare event, rather than an everyday occurrence.
SoyentNews is staffed by volunteers who give of their time and knowledge to provide a forum where people can discuss stories submitted by the community. We have no outside funding source.
Per our advance announcement on Saturday, May 20th, we completed our site update... one day ahead of schedule! And, even more amazingly, the community came together and we had over four dozen people subscribe since then! THANK YOU! Read on for more details.
The Site Upgrade: I am happy to report things went smoothly. So smoothly, in fact, I didn't even notice the upgrade was being rolled out! I was on the site at the time, following along on IRC (Internet Relay Chat), and didn't even realize the updates they were discussing were not on some support server... these updates were on the main site! (Given that I have a long background in QA/test, that's high praise indeed!) Many thanks to The Mighty Buzzard, NCommander, a surprise visit in IRC by "NC|FromTheFuture", and the rest of the SN staff waiting at the ready to help out should things go sideways.
IRC Server Updates: As mentioned in the earlier article, we are continuing apace with moving to Gentoo for our base OS across all our servers. Before we update the OS, all of the facilities and services underlying SoylentNews need be ported over. To that end, Deucalion has been working diligently to port our IRC servers to run on Gentoo and to do so in a 'multi-server' arrangement. (Behind the scenes, SoylentNews staff primarily coordinate our efforts using IRC. Should something go wrong, we do have fall-backs in place, but they are much less efficient.) We will keep you informed as to our progress.
Folding@home: Our progress has been slower and competition has been greater as we reach the higher ranks. We are currently still on track to be one of the Top 300 F@H Teams in the World by May 28th, 2017 — barely fifteen months since we started! To put this in perspective, there are over 226,270 teams behind us. Please consider helping us in the fight against many debilitating diseases such as Huntington's, Parkinson's, and Alzheimer's. (Original Announcement.)
Site Suggestions: The prior story brought a wealth of comments. Several suggestions for the site look to be both helpful and reasonably feasible to implement.
One proposed change is to provide a means for a user to set an explicit time (or a reference comment) for which comments newer than that would be flagged as *new*.
Several people shared how they had failed to notice their subscription had expired. One suggestion recommended dimming the "Site News" box which shows the site funding status, based on your current subscription status. Dim the box (user preference) when your subscription is up-to-date; display full-intensity when your subscription has expired (or you are an AC). Another suggested we add a banner at the top of the main page to keep folks appraised as to their subscription status (and a link to re-subscribe).
Separately, when viewing an article which appears in a nexus other than "The Main Page", some of the links on the page are particular to just that nexus, and not the site in general. This story, for example, is in the "Meta" nexus.
Staffing: It is my pleasure to introduce a new member of our staff, Xyem, who came on board on May 16th and has already made contributions to our code base! Please join me in welcoming him aboard.
Funding - In a word: WOW! The actual dollar amounts deposited into our accounts remain to be tabulated, but the current estimated tally, (as shown in the Beg-o-meter on the main page in the "Site News" slashbox) tells the tale. As of the time of writing this story, we have reached our base funding goal!
It bears mentioning that the base funding goal only covers our ongoing operations expenses. We have no prudent reserve should something goes sideways. Further, when SoylentNews started, there were setup expenses that were funded out-of-pocket by our founders. That was over three years ago and they have more than graciously allowed us to continue operating so far without insisting on getting repaid. Sadly, this all went down so long ago I don't recall the exact amount, but I believe it was on the order of $5K, total, that is owed two to people. It would thrill me to no end to know that they have been made whole. It is also important, as a Public Benefit Corporation that we be beholden to noone so that we can continue an an entity that provides a forum where the community can have open discussions on topics of interest. The community submits the stories, writes the comments, and moderates the comments. We are here for you.
It bears mentioning, for those who might not be aware, one is able to subscribe multiple times and/or specify a larger amount on the subscription page than the amounts offered. So far this year, NINE people have subscribed at $100.00 and one especially generous person subscribed at $250.00! Oh, and thanks to this upgrade, we have regained support for subscriptions via Bitcoin!
So, we have a stretch goal of $2000.00 which, if we were able to reach it, would allow us to make a significant step towards making the founders whole and allow SoylentNews to stand on its own.
Funding tl;dr: For tax and accounting purposes, all values are based on actual transactions to our bank account. Entirely separate is what we record internally to the site based on user's interactions with the UI, and there are some historical issues which we are addressing. The amounts appearing the "Site News" slashbox are, therefore, close approximations.
[*] We just discovered a few days ago that PayPal charges different fees depending on your local currency. For example, Alice (in America) subscribes to SoylentNews for one year with the suggested amount of $20.00 US using a credit card drawn on a US bank. Günther (from Germany) also chooses to subscribe for one year and at the suggested amount of $20.00 US. He, too uses his credit card, but it is drawn from an account denominated in Euros. You can see where this is headed, right? It appears there are additional fees charged for the conversion to $USD. See PayPal's merchant fees page for the low-down. Pay special attention to the fact that the additional fees are denominated in the user's local currency, not in $USD.
PayPal does inform us of the actual amount requested, the fees charged, and the net amount we receive. (We get similar info from Stripe, but of course, in a different format.) That information is now stored in our site database. But it wasn't always this way. In the very early days, we were mostly just trying to keep the site from crashing because the code on which this site was based had not been supported in several years and was rife with problems. As things stabilized over the ensuing months and years, we could finally bring our attention to other areas of the code. Since accounting was performed strictly by what happened through our bank account, there was little concern about what was happening internal to the site's inherited accounting code. And wouldn't you know it, the historical data had the gross subscription amount, but failed to accurately account for fees. Net amount was set to be the same as the gross amount. We are in the process of rectifying this, but it will take some time. Hence, the amounts shown in the "Site News" slashbox are an approximation.
To summarize, the site upgrade went smoothly, we have one of the top folding@home sites in the world(!), we are still working to improve the site, the community has been amazing in meeting our ongoing funding needs, and we are hoping we can start repaying our founders.
[Ed Note 2: Damn devs have made a liar out of me... moved it back to the original schedule noted below. - cmn32480]
[TMB Note: Site update complete. Bumped so folks will notice.]
It has been a few months since we last updated SoylentNews, and we've not been content to rest on our laurels. Our next site update is tentatively scheduled for Sunday, 2017-05-21, depending on staff availability. We'll update this story when we know for sure when it will take place.
Since this post was started, other things have come to light, so there's a bit of everything in here. Read on for the full scoop:
In this latest update (scheduled around 00:00 UTC on 5.21.2017, but we are flexible), we have made the following improvements:
Separately, the team has made great strides in moving to running on Gentoo. We are taking this step very methodically, making sure we have a solid foundation in place on one server before we even think of rolling it out to the rest of our systems. Yes, that means we will be free from systemd. Kudos to NCommander, Mechanicjay, Audioguy, TheMightyBuzzard, Paulej72, and Deucalion.
It's amazing how spare compute cycles add up! SoylentNews has a Folding@Home team which is helping researchers find a cure for diseases such as Huntington's, Alzheimer's, and Parkinson's — among many others. Our team was launched on Feb. 12, 2016. In just over 15 months, we have amassed well over 300 million points which places us at Team 304 out of 226564! Barring any surprises, and continuing at our current rate, we are on track to break past 300 and into the 2xx's on or about May 28th, 2017.
We are always open to receiving new team members. Contact Sir Finkus for more information, either via email at this site, or via the #Soylent or #folding channels on our IRC -- Internet Relay Chat server.
New account creation has been relatively consistent and steady over the past year averaging out to a new account pretty much every day. It is a pleasure to inform the community that, on May 18th, account number 6600 was registered on the site.
Lastly, it is my sad duty to inform the community that our cash intake has been seriously deficient so far this year. Our budget for the six-month period of Jan 1, 2017 through June 30, 2017 is $3,000 and we are currently at approximately half that, with less than 6 weeks to go.
We have in excess of 100 users who have been active on the site within the last 30 days whose subscription has lapsed. It is easy enough to do — I have failed to notice my own subscription's end on more than one occasion!
Plain and simple, the site needs to pay its bills. Please look at your subscription page and consider making a contribution. The dollar amounts shown in the text-entry fields are the minimum amount required for that subscription duration. We've had a few users anonymously contribute significantly more than that in the past.
Some have chosen to give a gift subscription to NCommander (UID: 2) as a sign of support. However you choose to make a contribution, please do so now.
Two months ago, I polled the community for advice on the underlying operating system that should power SoylentNews (SN). After reading comments, and some recent experiences in my personal and professional life, we are migrating to Gentoo as the operating system of choice. As of right now, we've already migrated our development box, lithium, over, and using it as a shakedown to see how painful the overall migration will be. I'm pleased to report that, aside from varnish (an HTTP accelerator), the process went relatively smoothly.
For those who weren't here for the original article querying the community (linked above), let me recap the situation. At the time that I wrote that article, SN was mostly standardized on Ubuntu 14.04, with a single CentOS 6.7 box lurking in our midst. In the course of testing updates and other projects, the staff and myself felt that Ubuntu (and Debian) had lost a lot of the advantages that had made it a rock solid choice for the last three years of powering SN, combined with the fact that the upgrade process would not have been trivial due to the systemd migration.
Though greatly disliked by all of us, systemd being part of Ubuntu 16.04 LTS (Long Term Stable) was not a deal breaker. More importantly was the perception that the release lacked stability and we had a serious sense that the upgrade would be problematic. I felt it was time to reopen the scenario to see if we were better off migrating to a different distro, or abandoning Linux entirely. As such the original article was penned to see what the community's feelings on the subject were. The overwhelming consensus was that I was not alone with my feelings on the latest LTS, and many thought FreeBSD would be a good choice for us. Ultimately, we decided to trial Gentoo over FreeBSD for four reasons
I'll break these first three item by item
FreeBSD is divided into two parts, the core system which has basic utilities, and the ports collection which has all the add on software like Apache and such. In theory, these two components can be updated independently of each other allowing a stable base while migrating to newer software versions with relative ease. Ports can be installed from binaries, or manually compiled to suit one's taste in a relatively automated fashion bringing together the best of a binary and source based distribution. On paper, it looks perfect.
In further research, I've found that port upgrades are fragile at the best of times. Unlike Debian's APT which has strict package dependence and shared library management, port upgrades are very much upgrade and pray and its possible to hose a system in this way. The situation is similar to using EPEL on CentOS, or using Slackware that port upgrades can leave artifacts, and there's often considerable manual intervention to keep things chugging around. This is compounded by the fact that the version of Kerberos we need is in the ports collection due to incompatibility between MIT Kerberos V (which we use) and Heimdal Kerberos which ships out of the box. For those of you familiar with Active Directory, this is roughly on par with the effort required to rebuild AD from scratch along side a pre-existing forest. This meant unless we rebuilt the entire Kerberos domain (a drastic and painful option to say the least) that we could easily break a node because a ports upgrade went sideways.
Furthermore, mixing binary and source ports also have several ways it can go wrong which is problematic. To ease our system maintenance burden, its long been a goal of the admin team to have rehash and its dependencies built and deployed through package management instead of the rather horrorifying script+rsync that we use now. While we could have technically achieved this with Ubuntu by running our own buildd (or using a PPA), the sheer amount of dependencies combined with the pain of rebuilding the world ultimately doomed this to the "would be nice" pile list of ideas.
On top of this, the split architecture of FreeBSD would also mean that upgrades are no longer "one command and done" as they are with Ubuntu and Debian. Instead it becomes a matter of determining what, if any, core system upgrades are available, deploy them, then deploy/rebuild ports as needed. None of this by itself would be deal-breaking, but when compounded with the other reasons it tipped me away from this option.
For any production website, having backups is the thing you must have, not the thing you wish you had. With the exception of our development box, all our systems are backed up to off-site storage on a machine called oxygen via rsnapshot nightly (and yes, we do test our backups). However, due to the way SoylentNews is situated, there is the possibility that if an attacker ever successfully breached SN, its possible they may be able to gain access to oxygen, and rm -rf / everything.
For this reason alone, we used two separate sets of backups in case of system failure or node compromise. As mentioned many times before, SN is hosted on a number of VPSes by Linode who I continue to highly recommend for anyone's VPS needs. One very useful and handy feature of Linode is that they offer snapshotting and node backups as part of their hosting services for reasonable prices, and critical system boxes are backed up with them as a second-level of defense. Unfortunately, Linode's backup services require that their system understand the underlying filesystem format used by the OS so they can snapshot it easily. As of writing, they do not support FreeBSD's UFS or ZFS. A migration would mean we'd have to sink additional costs in a new backup system to supplement oxygen.
I'm going to get flamed for this reason, but recent events have sort of drilled this home for me, both at SoylentNews and as my work as a freelancer. During the last round of security updates, I've been fighting to get several of CentOS's security issues fixed. Red Hat (and CentOS) offer ten year support for their products but in many ways it is the wrong approach to system stability and security. A real-world issue I ran into with CentOS's support is that they ship rather old issues of dovecot, a relatively popular IMAP server.
Now, in theory, as long as security patches are backported, this shouldn't be a problem. In practice however, it means you're essentially tethered to the security features as offered at the time of the release. For example, a good number of our users are likely familiar with the Logjam attack. The mitigation for Logjam is to regenerate DH parameters to larger sizes, and change to a non-common prime. Relatively straightforward, right?
Well, not so much. Dovecot 2.0 (which is what CentOS 6.7 ships with) doesn't allow for setting of custom DH parameters, or even tweaking anything beyond the most basic TLS settings. To a lesser extent, we also had this problem with Postfix (we can't disable client side negotiation). The solution in both cases is to upgrade. That would be great, if we could in-place upgrade CentOS, or reliably upgrade the RPMs without hosing YUM at a later date. In practice, we've been forced into doing a number of arcane hacks to get most of the survey tools to report anything better than a "C" grade, with the situation worsening as time goes on. Before people say "well that's a problem with dovecot", and not CentOS, you can't get OCSP stapling (which is an important security feature to help fix SSL's revocation system) with Apache out of the box. You need to either patch Apache 2.2 in place, or upgrade to 2.4.
This problem also has shown its head on Ubuntu. To Canonical's credit, their security team actually has gone through the work of mainlining newer security features in popular products; Ubuntu 14.04's Apache 2.2 supports OCSP stapling because they patch Apache in their binaries. However this practice only goes so far. Deploying CAA records to SoylentNews in the last round of tweaks was an exercise in frustration because only the most recent versions of BIND knows how to handle the CAA record type. Once again, we're in serious voodoo territory if we tried to upgrade BIND outside of a distro release.
This brings me to my final point: trying to follow industry best-practices falls apart if you can't easily update your stack. Release based distros at best (with Ubuntu) update once every six months, or once every year or so for longer term support from other distros. That's a very long time in the security world. Furthermore, each major upgrade is an event and a large time sink in and of itself. As such I've (grudgingly) come to the conclusion that if you actually want to have real security, you need to update frequently. Furthermore, by having smaller upgrades at a given time instead of them in one large pile, you have a better chance of not getting overwhelmed at those release points.
Gentoo ultimately won by being both rolling-release based, and source based. It meant that we could easily upgrade the entire stack (including rehash's special dependencies) as a single emerge world, and then deploy. It also edged out the other options by not forcing systemd on us (and OpenRC is an absolute pleasure to debug and maintain in comparison). We've also discussed the issue at length and have determined how we're going to approach the rather daunting task ahead of us.
The first step, which was already completed, was to migrate our development system over to Gentoo to get an idea of how much pain we're going to be in. This was accomplished by booting the system in rescue mode, moving "/" (i.e. the root of our filesystem) to "/old-rootfs", extracting a stage3, cooking the kernel, and rebooting. audioguy and TheMightyBuzzard worked out the correct set of USE flags for our environment, and I used the serial console to do the actual changeover. Aside from Varnish breaking, the migration was actually relatively smooth if time consuming. Right now, we're still wrestling with varnish, but after kicking MySQL cluster's init scripts and copying configs, it sputtered to life and dev.soylentnews.org popped back onto the internet.
The next steps is to create ebuilds for hesinfo (a Hesiod support tool that Gentoo doesn't ship in their hesiod package), and then to create a custom stage3 with our kernel config and base system with catalyst. We're going to work out the set of packages we need and configure lithium to work as a binary package source for portage. In effect, every package we need will be compiled once on lithium, then published via a private portage repository. Other machines will simply be able to emerge world and download the pre-tested and compiled binaries in one fell swoop keeping the software stack across SoylentNews consistent across the organization. As an added bonus, we can now easily migrate our custom set of compilation scripts to ebuilds and have sane package and dependency tracking for the entire site infrastructure.
Since most of the site infrastructure is fully redundant, we don't expect too much downtime or breakage as we begin migrating other boxes from Ubuntu. As usual, we'll keep the community apprised of our status, as well as if we need to schedule actual site downtime during this period. While some of us might thing we're insane, I will just note for the record we took a similarly drastic step of migrating to a IPv6-only backend two years ago in the name of administration sanity, and serving SN needs best. As always, I'll be reading and commenting below.
In the continuing saga of website tinkering and people's love of update posts, I'm back with some backend configuration changes. Right now, things have been relatively quiet on the backend side of things. We've got some good news, and some bad news in this update. That being said, we've made a few small updates over the weekend. Rapid fire style, let's go through them:
CAA records define which certificate authorities (CAs) are allowed to sign your domains. They essentially act as a CA whitelist, and the most recent revisions of the Certificate Authority/Browser Baseline Requirements mandates that CAs check for CAA records and respect them. In line with this policy, we've white-listed Let's Encrypt and Gandi's CAs to issue certificates for SoylentNews for the time being as these are the two CAs currently in use here.
In a fun bit of fail, this is the second time I've tried to deploy CAA, and fortunately managed to succeed this go around. The problem stems from the fact that many versions of BIND except the very latest don't recognize the "CAA" record type, and cause the zone file to not process correctly if it's present. As we're still using an older version of BIND as our master server, I had to manually create TYPE257 records as seen below:
soylentnews.org. 3586 IN TYPE257 \# 16 0005697373756567616E64692E6E6574 soylentnews.org. 3586 IN TYPE257 \# 22 000569737375656C657473656E63727970742E6F7267 soylentnews.org. 3586 IN TYPE257 \# 35 0005696F6465666D61696C746F3A61646D696E40736F796C656E746E 6577732E6F7267 soylentnews.org. 3586 IN TYPE257 \# 12 0009697373756577696C643B
Both htbridge.com and ssllabs.com show that the CAA records are properly encoded, and show an additional green bar that they're in place.
Almost two years ago, the Logjam attack on the DH key exchange was discovered and publicized. As part of our general hardening of SoylentNews, we regenerated all the DH parameters to prevent logjam from being a viable attack vector. Unfortunately, we overlooked the mail STARTTLS services on mail.soylentnews.org, and only caught it when I was checking various security things. The DH parameter files have been regenerated. Under normal circumstances, Logjam can't be exploited unless the underlying SSL cipher is relatively weak. As part of previous hardening, we kicked SSLv3 and many insecure ciphers to the curb, but unfortunately RSA_CBC_IDEA was accidentally left in place as a valid protocol for STARTTLS transport. Based on my understanding of the logjam attack, 1024-bit ciphers like RSA_CBC_IDEA are still difficult to exploit, and its likely only a nation state could successfully have breached it.
Given only SN staff have mail accounts, and that users are encouraged to change their passwords after creating an account, I think its safe to assume that we're relatively OK as far as data security and integrity go since email in general at best is opportunistically encrypted, and should always be assumed to be monitor-able (via a STRIPTLS attack). That being said, if you haven't changed your password from account creation though, it's likely a good idea to do so now.
We discovered our IMAP server has been serving a self-signed certificate during this check as well. We'll be replacing this with a properly signed certificate within the near future. I have other things on this topic that will be noted in a future post, so keep a look out for that.
Disabling HTTP Methods
A routine check of the site's security headers showed that we were accepting HTTP TRACE and other methods we don't need on production. The configuration for nginx has been modified to put a bullet in this behavior. We're still checking to make sure we got this everywhere, but we should be good on at least the production servers for now. This has bumped the site security rating up to an A on the HTBridge; we're still missing the referral security header, but we need to check to make sure there's no user impact before deploying it.
3DES Put Out To Pasture
As always in the world of encryption, various algorithms eventually become insecure and weakened as cryptanalysis gets more and more advanced. A few months ago, the SWEET32 attack against 3DES was discovered which drastically weakens the security of 3DES via the birthday paradox problem. In practice, SWEET32 requires a second exploit to even be usable as SoylentNews only allowed 3DES connections as a last resort if AES wasn't supported. As every major browser has supported AES for years, we decided to put 3DES out to pasture and have removed it from the allowed list of ciphers for SN.
Not too much to note in this round of administration games, but we're working to make overhaul changes to the stack to allow the potential for HPKP key pinning in the near future, as well as deploying TLSA/DANE support for both HTTPS and SMTP on SN. As part of this process, we'll also be enabling HSTS across subdomains, and reissuing our SSL certificates to enable OCSP Must-Staple. We'll keep you guys updated as we move towards that goal!
If you haven't noticed already, today is April 1st, otherwise known as April Fool's Day.
Writers are a creative lot and very occasionally use this day as an excuse to let their creative juices flow — as they are often granted some discretion by those who watch over their work.
In years past, I've had times where I decided I'd just give up on surfing the web at all, given the plethora of nonsense that got posted. "Sure, it might have seemed funny when you wrote it, but it sure didn't do much of anything for me." I know I am not the only one who has felt this way. Once in a while, though, I do stumble upon a real gem that makes me bust out laughing.
Given all the noise of the half-baked stories out there, the well-crafted prank stories can be hard to find. Let's use this story to post links to the best April Fool's Stories. Funny stories are, of course, desired. So too are bizarre, but genuine, stories. Bring it on!
So the Dev Team has been hard at work fixing up issues with with the 17_02 release. We compiled all of your comments from the 17_02 Meta stories into a large bug and feature request list. We have been working on getting these issues fixed as soon as we can.
You may have noticed some changes over the last week that went out to fix some issues, and we just released some more fixes today.
Here is a list of the major fixes since the last story:
And here are the latest updates:
So if you see any new bugs that you think are related to these changes, or just want to let us know about an ongoing issue, please feel free to comment below.
Here are the currently known bugs that we are working on:
And here are the feature requests:
Discussion to a minimum here, please, so it doesn't distract from having an all-in-one-place list of things from this release that still need addressed.