Covers the period:
2017-01-01 00:00:00 ..
2017-05-28 14:01:42 UTC
(SPIDs: [586..706]) --martyb
We always have a place for talented people, visit the Get Involved section on the wiki to see how you can make SoylentNews better.
SoyentNews is staffed by volunteers who give of their time and knowledge to provide a forum where people can discuss stories submitted by the community. We have no outside funding source.
Per our advance announcement on Saturday, May 20th, we completed our site update... one day ahead of schedule! And, even more amazingly, the community came together and we had over four dozen people subscribe since then! THANK YOU! Read on for more details.
The Site Upgrade: I am happy to report things went smoothly. So smoothly, in fact, I didn't even notice the upgrade was being rolled out! I was on the site at the time, following along on IRC (Internet Relay Chat), and didn't even realize the updates they were discussing were not on some support server... these updates were on the main site! (Given that I have a long background in QA/test, that's high praise indeed!) Many thanks to The Mighty Buzzard, NCommander, a surprise visit in IRC by "NC|FromTheFuture", and the rest of the SN staff waiting at the ready to help out should things go sideways.
IRC Server Updates: As mentioned in the earlier article, we are continuing apace with moving to Gentoo for our base OS across all our servers. Before we update the OS, all of the facilities and services underlying SoylentNews need be ported over. To that end, Deucalion has been working diligently to port our IRC servers to run on Gentoo and to do so in a 'multi-server' arrangement. (Behind the scenes, SoylentNews staff primarily coordinate our efforts using IRC. Should something go wrong, we do have fall-backs in place, but they are much less efficient.) We will keep you informed as to our progress.
Folding@home: Our progress has been slower and competition has been greater as we reach the higher ranks. We are currently still on track to be one of the Top 300 F@H Teams in the World by May 28th, 2017 — barely fifteen months since we started! To put this in perspective, there are over 226,270 teams behind us. Please consider helping us in the fight against many debilitating diseases such as Huntington's, Parkinson's, and Alzheimer's. (Original Announcement.)
Site Suggestions: The prior story brought a wealth of comments. Several suggestions for the site look to be both helpful and reasonably feasible to implement.
One proposed change is to provide a means for a user to set an explicit time (or a reference comment) for which comments newer than that would be flagged as *new*.
Several people shared how they had failed to notice their subscription had expired. One suggestion recommended dimming the "Site News" box which shows the site funding status, based on your current subscription status. Dim the box (user preference) when your subscription is up-to-date; display full-intensity when your subscription has expired (or you are an AC). Another suggested we add a banner at the top of the main page to keep folks appraised as to their subscription status (and a link to re-subscribe).
Separately, when viewing an article which appears in a nexus other than "The Main Page", some of the links on the page are particular to just that nexus, and not the site in general. This story, for example, is in the "Meta" nexus.
Staffing: It is my pleasure to introduce a new member of our staff, Xyem, who came on board on May 16th and has already made contributions to our code base! Please join me in welcoming him aboard.
Funding - In a word: WOW! The actual dollar amounts deposited into our accounts remain to be tabulated, but the current estimated tally, (as shown in the Beg-o-meter on the main page in the "Site News" slashbox) tells the tale. As of the time of writing this story, we have reached our base funding goal!
It bears mentioning that the base funding goal only covers our ongoing operations expenses. We have no prudent reserve should something goes sideways. Further, when SoylentNews started, there were setup expenses that were funded out-of-pocket by our founders. That was over three years ago and they have more than graciously allowed us to continue operating so far without insisting on getting repaid. Sadly, this all went down so long ago I don't recall the exact amount, but I believe it was on the order of $5K, total, that is owed two to people. It would thrill me to no end to know that they have been made whole. It is also important, as a Public Benefit Corporation that we be beholden to noone so that we can continue an an entity that provides a forum where the community can have open discussions on topics of interest. The community submits the stories, writes the comments, and moderates the comments. We are here for you.
It bears mentioning, for those who might not be aware, one is able to subscribe multiple times and/or specify a larger amount on the subscription page than the amounts offered. So far this year, NINE people have subscribed at $100.00 and one especially generous person subscribed at $250.00! Oh, and thanks to this upgrade, we have regained support for subscriptions via Bitcoin!
So, we have a stretch goal of $2000.00 which, if we were able to reach it, would allow us to make a significant step towards making the founders whole and allow SoylentNews to stand on its own.
Funding tl;dr: For tax and accounting purposes, all values are based on actual transactions to our bank account. Entirely separate is what we record internally to the site based on user's interactions with the UI, and there are some historical issues which we are addressing. The amounts appearing the "Site News" slashbox are, therefore, close approximations.
[*] We just discovered a few days ago that PayPal charges different fees depending on your local currency. For example, Alice (in America) subscribes to SoylentNews for one year with the suggested amount of $20.00 US using a credit card drawn on a US bank. Günther (from Germany) also chooses to subscribe for one year and at the suggested amount of $20.00 US. He, too uses his credit card, but it is drawn from an account denominated in Euros. You can see where this is headed, right? It appears there are additional fees charged for the conversion to $USD. See PayPal's merchant fees page for the low-down. Pay special attention to the fact that the additional fees are denominated in the user's local currency, not in $USD.
PayPal does inform us of the actual amount requested, the fees charged, and the net amount we receive. (We get similar info from Stripe, but of course, in a different format.) That information is now stored in our site database. But it wasn't always this way. In the very early days, we were mostly just trying to keep the site from crashing because the code on which this site was based had not been supported in several years and was rife with problems. As things stabilized over the ensuing months and years, we could finally bring our attention to other areas of the code. Since accounting was performed strictly by what happened through our bank account, there was little concern about what was happening internal to the site's inherited accounting code. And wouldn't you know it, the historical data had the gross subscription amount, but failed to accurately account for fees. Net amount was set to be the same as the gross amount. We are in the process of rectifying this, but it will take some time. Hence, the amounts shown in the "Site News" slashbox are an approximation.
To summarize, the site upgrade went smoothly, we have one of the top folding@home sites in the world(!), we are still working to improve the site, the community has been amazing in meeting our ongoing funding needs, and we are hoping we can start repaying our founders.
[Ed Note 2: Damn devs have made a liar out of me... moved it back to the original schedule noted below. - cmn32480]
[TMB Note: Site update complete. Bumped so folks will notice.]
It has been a few months since we last updated SoylentNews, and we've not been content to rest on our laurels. Our next site update is tentatively scheduled for Sunday, 2017-05-21, depending on staff availability. We'll update this story when we know for sure when it will take place.
Since this post was started, other things have come to light, so there's a bit of everything in here. Read on for the full scoop:
In this latest update (scheduled around 00:00 UTC on 5.21.2017, but we are flexible), we have made the following improvements:
Separately, the team has made great strides in moving to running on Gentoo. We are taking this step very methodically, making sure we have a solid foundation in place on one server before we even think of rolling it out to the rest of our systems. Yes, that means we will be free from systemd. Kudos to NCommander, Mechanicjay, Audioguy, TheMightyBuzzard, Paulej72, and Deucalion.
It's amazing how spare compute cycles add up! SoylentNews has a Folding@Home team which is helping researchers find a cure for diseases such as Huntington's, Alzheimer's, and Parkinson's — among many others. Our team was launched on Feb. 12, 2016. In just over 15 months, we have amassed well over 300 million points which places us at Team 304 out of 226564! Barring any surprises, and continuing at our current rate, we are on track to break past 300 and into the 2xx's on or about May 28th, 2017.
We are always open to receiving new team members. Contact Sir Finkus for more information, either via email at this site, or via the #Soylent or #folding channels on our IRC -- Internet Relay Chat server.
New account creation has been relatively consistent and steady over the past year averaging out to a new account pretty much every day. It is a pleasure to inform the community that, on May 18th, account number 6600 was registered on the site.
Lastly, it is my sad duty to inform the community that our cash intake has been seriously deficient so far this year. Our budget for the six-month period of Jan 1, 2017 through June 30, 2017 is $3,000 and we are currently at approximately half that, with less than 6 weeks to go.
We have in excess of 100 users who have been active on the site within the last 30 days whose subscription has lapsed. It is easy enough to do — I have failed to notice my own subscription's end on more than one occasion!
Plain and simple, the site needs to pay its bills. Please look at your subscription page and consider making a contribution. The dollar amounts shown in the text-entry fields are the minimum amount required for that subscription duration. We've had a few users anonymously contribute significantly more than that in the past.
Some have chosen to give a gift subscription to NCommander (UID: 2) as a sign of support. However you choose to make a contribution, please do so now.
Two months ago, I polled the community for advice on the underlying operating system that should power SoylentNews (SN). After reading comments, and some recent experiences in my personal and professional life, we are migrating to Gentoo as the operating system of choice. As of right now, we've already migrated our development box, lithium, over, and using it as a shakedown to see how painful the overall migration will be. I'm pleased to report that, aside from varnish (an HTTP accelerator), the process went relatively smoothly.
For those who weren't here for the original article querying the community (linked above), let me recap the situation. At the time that I wrote that article, SN was mostly standardized on Ubuntu 14.04, with a single CentOS 6.7 box lurking in our midst. In the course of testing updates and other projects, the staff and myself felt that Ubuntu (and Debian) had lost a lot of the advantages that had made it a rock solid choice for the last three years of powering SN, combined with the fact that the upgrade process would not have been trivial due to the systemd migration.
Though greatly disliked by all of us, systemd being part of Ubuntu 16.04 LTS (Long Term Stable) was not a deal breaker. More importantly was the perception that the release lacked stability and we had a serious sense that the upgrade would be problematic. I felt it was time to reopen the scenario to see if we were better off migrating to a different distro, or abandoning Linux entirely. As such the original article was penned to see what the community's feelings on the subject were. The overwhelming consensus was that I was not alone with my feelings on the latest LTS, and many thought FreeBSD would be a good choice for us. Ultimately, we decided to trial Gentoo over FreeBSD for four reasons
I'll break these first three item by item
FreeBSD is divided into two parts, the core system which has basic utilities, and the ports collection which has all the add on software like Apache and such. In theory, these two components can be updated independently of each other allowing a stable base while migrating to newer software versions with relative ease. Ports can be installed from binaries, or manually compiled to suit one's taste in a relatively automated fashion bringing together the best of a binary and source based distribution. On paper, it looks perfect.
In further research, I've found that port upgrades are fragile at the best of times. Unlike Debian's APT which has strict package dependence and shared library management, port upgrades are very much upgrade and pray and its possible to hose a system in this way. The situation is similar to using EPEL on CentOS, or using Slackware that port upgrades can leave artifacts, and there's often considerable manual intervention to keep things chugging around. This is compounded by the fact that the version of Kerberos we need is in the ports collection due to incompatibility between MIT Kerberos V (which we use) and Heimdal Kerberos which ships out of the box. For those of you familiar with Active Directory, this is roughly on par with the effort required to rebuild AD from scratch along side a pre-existing forest. This meant unless we rebuilt the entire Kerberos domain (a drastic and painful option to say the least) that we could easily break a node because a ports upgrade went sideways.
Furthermore, mixing binary and source ports also have several ways it can go wrong which is problematic. To ease our system maintenance burden, its long been a goal of the admin team to have rehash and its dependencies built and deployed through package management instead of the rather horrorifying script+rsync that we use now. While we could have technically achieved this with Ubuntu by running our own buildd (or using a PPA), the sheer amount of dependencies combined with the pain of rebuilding the world ultimately doomed this to the "would be nice" pile list of ideas.
On top of this, the split architecture of FreeBSD would also mean that upgrades are no longer "one command and done" as they are with Ubuntu and Debian. Instead it becomes a matter of determining what, if any, core system upgrades are available, deploy them, then deploy/rebuild ports as needed. None of this by itself would be deal-breaking, but when compounded with the other reasons it tipped me away from this option.
For any production website, having backups is the thing you must have, not the thing you wish you had. With the exception of our development box, all our systems are backed up to off-site storage on a machine called oxygen via rsnapshot nightly (and yes, we do test our backups). However, due to the way SoylentNews is situated, there is the possibility that if an attacker ever successfully breached SN, its possible they may be able to gain access to oxygen, and rm -rf / everything.
For this reason alone, we used two separate sets of backups in case of system failure or node compromise. As mentioned many times before, SN is hosted on a number of VPSes by Linode who I continue to highly recommend for anyone's VPS needs. One very useful and handy feature of Linode is that they offer snapshotting and node backups as part of their hosting services for reasonable prices, and critical system boxes are backed up with them as a second-level of defense. Unfortunately, Linode's backup services require that their system understand the underlying filesystem format used by the OS so they can snapshot it easily. As of writing, they do not support FreeBSD's UFS or ZFS. A migration would mean we'd have to sink additional costs in a new backup system to supplement oxygen.
I'm going to get flamed for this reason, but recent events have sort of drilled this home for me, both at SoylentNews and as my work as a freelancer. During the last round of security updates, I've been fighting to get several of CentOS's security issues fixed. Red Hat (and CentOS) offer ten year support for their products but in many ways it is the wrong approach to system stability and security. A real-world issue I ran into with CentOS's support is that they ship rather old issues of dovecot, a relatively popular IMAP server.
Now, in theory, as long as security patches are backported, this shouldn't be a problem. In practice however, it means you're essentially tethered to the security features as offered at the time of the release. For example, a good number of our users are likely familiar with the Logjam attack. The mitigation for Logjam is to regenerate DH parameters to larger sizes, and change to a non-common prime. Relatively straightforward, right?
Well, not so much. Dovecot 2.0 (which is what CentOS 6.7 ships with) doesn't allow for setting of custom DH parameters, or even tweaking anything beyond the most basic TLS settings. To a lesser extent, we also had this problem with Postfix (we can't disable client side negotiation). The solution in both cases is to upgrade. That would be great, if we could in-place upgrade CentOS, or reliably upgrade the RPMs without hosing YUM at a later date. In practice, we've been forced into doing a number of arcane hacks to get most of the survey tools to report anything better than a "C" grade, with the situation worsening as time goes on. Before people say "well that's a problem with dovecot", and not CentOS, you can't get OCSP stapling (which is an important security feature to help fix SSL's revocation system) with Apache out of the box. You need to either patch Apache 2.2 in place, or upgrade to 2.4.
This problem also has shown its head on Ubuntu. To Canonical's credit, their security team actually has gone through the work of mainlining newer security features in popular products; Ubuntu 14.04's Apache 2.2 supports OCSP stapling because they patch Apache in their binaries. However this practice only goes so far. Deploying CAA records to SoylentNews in the last round of tweaks was an exercise in frustration because only the most recent versions of BIND knows how to handle the CAA record type. Once again, we're in serious voodoo territory if we tried to upgrade BIND outside of a distro release.
This brings me to my final point: trying to follow industry best-practices falls apart if you can't easily update your stack. Release based distros at best (with Ubuntu) update once every six months, or once every year or so for longer term support from other distros. That's a very long time in the security world. Furthermore, each major upgrade is an event and a large time sink in and of itself. As such I've (grudgingly) come to the conclusion that if you actually want to have real security, you need to update frequently. Furthermore, by having smaller upgrades at a given time instead of them in one large pile, you have a better chance of not getting overwhelmed at those release points.
Gentoo ultimately won by being both rolling-release based, and source based. It meant that we could easily upgrade the entire stack (including rehash's special dependencies) as a single emerge world, and then deploy. It also edged out the other options by not forcing systemd on us (and OpenRC is an absolute pleasure to debug and maintain in comparison). We've also discussed the issue at length and have determined how we're going to approach the rather daunting task ahead of us.
The first step, which was already completed, was to migrate our development system over to Gentoo to get an idea of how much pain we're going to be in. This was accomplished by booting the system in rescue mode, moving "/" (i.e. the root of our filesystem) to "/old-rootfs", extracting a stage3, cooking the kernel, and rebooting. audioguy and TheMightyBuzzard worked out the correct set of USE flags for our environment, and I used the serial console to do the actual changeover. Aside from Varnish breaking, the migration was actually relatively smooth if time consuming. Right now, we're still wrestling with varnish, but after kicking MySQL cluster's init scripts and copying configs, it sputtered to life and dev.soylentnews.org popped back onto the internet.
The next steps is to create ebuilds for hesinfo (a Hesiod support tool that Gentoo doesn't ship in their hesiod package), and then to create a custom stage3 with our kernel config and base system with catalyst. We're going to work out the set of packages we need and configure lithium to work as a binary package source for portage. In effect, every package we need will be compiled once on lithium, then published via a private portage repository. Other machines will simply be able to emerge world and download the pre-tested and compiled binaries in one fell swoop keeping the software stack across SoylentNews consistent across the organization. As an added bonus, we can now easily migrate our custom set of compilation scripts to ebuilds and have sane package and dependency tracking for the entire site infrastructure.
Since most of the site infrastructure is fully redundant, we don't expect too much downtime or breakage as we begin migrating other boxes from Ubuntu. As usual, we'll keep the community apprised of our status, as well as if we need to schedule actual site downtime during this period. While some of us might thing we're insane, I will just note for the record we took a similarly drastic step of migrating to a IPv6-only backend two years ago in the name of administration sanity, and serving SN needs best. As always, I'll be reading and commenting below.
In the continuing saga of website tinkering and people's love of update posts, I'm back with some backend configuration changes. Right now, things have been relatively quiet on the backend side of things. We've got some good news, and some bad news in this update. That being said, we've made a few small updates over the weekend. Rapid fire style, let's go through them:
CAA records define which certificate authorities (CAs) are allowed to sign your domains. They essentially act as a CA whitelist, and the most recent revisions of the Certificate Authority/Browser Baseline Requirements mandates that CAs check for CAA records and respect them. In line with this policy, we've white-listed Let's Encrypt and Gandi's CAs to issue certificates for SoylentNews for the time being as these are the two CAs currently in use here.
In a fun bit of fail, this is the second time I've tried to deploy CAA, and fortunately managed to succeed this go around. The problem stems from the fact that many versions of BIND except the very latest don't recognize the "CAA" record type, and cause the zone file to not process correctly if it's present. As we're still using an older version of BIND as our master server, I had to manually create TYPE257 records as seen below:
soylentnews.org. 3586 IN TYPE257 \# 16 0005697373756567616E64692E6E6574 soylentnews.org. 3586 IN TYPE257 \# 22 000569737375656C657473656E63727970742E6F7267 soylentnews.org. 3586 IN TYPE257 \# 35 0005696F6465666D61696C746F3A61646D696E40736F796C656E746E 6577732E6F7267 soylentnews.org. 3586 IN TYPE257 \# 12 0009697373756577696C643B
Both htbridge.com and ssllabs.com show that the CAA records are properly encoded, and show an additional green bar that they're in place.
Almost two years ago, the Logjam attack on the DH key exchange was discovered and publicized. As part of our general hardening of SoylentNews, we regenerated all the DH parameters to prevent logjam from being a viable attack vector. Unfortunately, we overlooked the mail STARTTLS services on mail.soylentnews.org, and only caught it when I was checking various security things. The DH parameter files have been regenerated. Under normal circumstances, Logjam can't be exploited unless the underlying SSL cipher is relatively weak. As part of previous hardening, we kicked SSLv3 and many insecure ciphers to the curb, but unfortunately RSA_CBC_IDEA was accidentally left in place as a valid protocol for STARTTLS transport. Based on my understanding of the logjam attack, 1024-bit ciphers like RSA_CBC_IDEA are still difficult to exploit, and its likely only a nation state could successfully have breached it.
Given only SN staff have mail accounts, and that users are encouraged to change their passwords after creating an account, I think its safe to assume that we're relatively OK as far as data security and integrity go since email in general at best is opportunistically encrypted, and should always be assumed to be monitor-able (via a STRIPTLS attack). That being said, if you haven't changed your password from account creation though, it's likely a good idea to do so now.
We discovered our IMAP server has been serving a self-signed certificate during this check as well. We'll be replacing this with a properly signed certificate within the near future. I have other things on this topic that will be noted in a future post, so keep a look out for that.
Disabling HTTP Methods
A routine check of the site's security headers showed that we were accepting HTTP TRACE and other methods we don't need on production. The configuration for nginx has been modified to put a bullet in this behavior. We're still checking to make sure we got this everywhere, but we should be good on at least the production servers for now. This has bumped the site security rating up to an A on the HTBridge; we're still missing the referral security header, but we need to check to make sure there's no user impact before deploying it.
3DES Put Out To Pasture
As always in the world of encryption, various algorithms eventually become insecure and weakened as cryptanalysis gets more and more advanced. A few months ago, the SWEET32 attack against 3DES was discovered which drastically weakens the security of 3DES via the birthday paradox problem. In practice, SWEET32 requires a second exploit to even be usable as SoylentNews only allowed 3DES connections as a last resort if AES wasn't supported. As every major browser has supported AES for years, we decided to put 3DES out to pasture and have removed it from the allowed list of ciphers for SN.
Not too much to note in this round of administration games, but we're working to make overhaul changes to the stack to allow the potential for HPKP key pinning in the near future, as well as deploying TLSA/DANE support for both HTTPS and SMTP on SN. As part of this process, we'll also be enabling HSTS across subdomains, and reissuing our SSL certificates to enable OCSP Must-Staple. We'll keep you guys updated as we move towards that goal!
If you haven't noticed already, today is April 1st, otherwise known as April Fool's Day.
Writers are a creative lot and very occasionally use this day as an excuse to let their creative juices flow — as they are often granted some discretion by those who watch over their work.
In years past, I've had times where I decided I'd just give up on surfing the web at all, given the plethora of nonsense that got posted. "Sure, it might have seemed funny when you wrote it, but it sure didn't do much of anything for me." I know I am not the only one who has felt this way. Once in a while, though, I do stumble upon a real gem that makes me bust out laughing.
Given all the noise of the half-baked stories out there, the well-crafted prank stories can be hard to find. Let's use this story to post links to the best April Fool's Stories. Funny stories are, of course, desired. So too are bizarre, but genuine, stories. Bring it on!
So the Dev Team has been hard at work fixing up issues with with the 17_02 release. We compiled all of your comments from the 17_02 Meta stories into a large bug and feature request list. We have been working on getting these issues fixed as soon as we can.
You may have noticed some changes over the last week that went out to fix some issues, and we just released some more fixes today.
Here is a list of the major fixes since the last story:
And here are the latest updates:
So if you see any new bugs that you think are related to these changes, or just want to let us know about an ongoing issue, please feel free to comment below.
Here are the currently known bugs that we are working on:
And here are the feature requests:
Discussion to a minimum here, please, so it doesn't distract from having an all-in-one-place list of things from this release that still need addressed.
Martyb here once again with today's update on our site's update. I apologize that this is not as well written as my prior updates as exhaustion and outside issues are demanding of my time. I ask you to please bear with me.
Quick Recap: As you may recall, our servers were getting melted trying to serve up highly-commented stories. Further, this made for an unacceptably long delay between the time one would request a story and when it would finally get returned for display. Our devs have implemented alternative display modes, "Threaded-TOS" and "Threaded-TNG" which use a much more efficient means of processing comments and getting them to you. These changes went "live" on Saturday, February 25.
Like anything new, we expected there would be issues. We very much appreciate your patience as we tried to work through these as they were reported. And did you ever let us know!
Paulej72 (aka PJ) and TheMightyBuzzard (TMB) have been laboring mightily to keep up with the issues that have been reported as well as a few they found independently. Similarly, as their fixes have gone live, our community has had to deal with a changing landscape of "what happens when I do this?" To add to this, comments being such a major part of the site's purpose, there are "knobs" in several places where users can customize which comments are presented to them and how they appear. The permutations are many and wide-raging. As bug fixes have been made, the impact of changing these has had different effects over time.
I've been astounded at how much the community has been supportive of our efforts, how well problems have been described and isolated, and how quickly the devs have been able to fix bugs as they have been noted. Even more impressive was the discussion in our last update story on possible alternative means of implementing the <spoiler> tag. I'm proud to be part of this community — you rock!
Stories: While all this activity has been happening, stories have still been posted to our site for your reading and commenting pleasure. We are working with a reduced editorial staff at the moment. Us long-timers have been posting as we can, but I would like to personally thanks our new editors fnord666 and charon for their heroic efforts getting stories posted, and takyon for his continued efforts at providing well-written stories. I have noticed submissions from new folks as well, and the heartens me immensely! (Note: I hesitate to call out people in particular for fear I will overlook someone; any omission is purely my fault and I would appreciate being called out on it if I have failed to list your contribution.)
Plans: This development blitz has, however, come at a cost. For those who were with this site at its inception, there was a "day of rest" imposed on the developers who had worked basically non-stop trying to get our site up and somewhat stable. I have suggested a similar break to our dev staff. Recall we are all volunteers doing this in our spare time. PJ has plans coming up and will be unavailable on Friday and Saturday. From what I've seen, TMB is well nigh a crispy critter at this point and most certainly needs a break. And, quite frankly, I've put a lot of personal stuff on hold while working on this update and could use a break, as well. In short, we are tired.
So, PJ is around for a bit (in his free time while at work) for today and TMB is getting a well-deserved breather. NCommander is nearing burnout has been tied up with an outside project that demands his full attention and has been unable to help much. I'll poke in from time to time, but I really need some time off, too.
What I ask from the community is that we do something similar. Step back for a moment. Look at the forest and not just the trees. Play around with the different display Modes. Try setting a different "Breakthrough" and/or "Threshold". Things should be much more stable today, so that will make it easier to gain a "mental model" of what does what.
The other thing I would ask is for the community to pull together and try to address issues together. Someone posts an issue about struggling with having to click on all the little chevrons? Inquire about their user preference settings, and suggest a different value for Threshold/Breakthrough. My sense is that some are more adept at using the new features and they can help others to get a better understanding of how things work. With those issues addressed, we can more clearly identify and isolate underlying problems and focus our energies more productively.
tl;dr We're not done yet, we truly appreciate your patience and forbearance during this transition, we need a break, and you guys rock in helping others in the community understand and use the new stuff. As always, keep our toes to the fire — we are here for you — let us catch our breath and we'll be better able to move forward.
Martyb here once again with an update on our progress with the site upgrade.
Our development team (paulej72, TheMightyBuzzard, and NCommander) have been hard at it trying to isolate and quash the bugs that have been reported. Having looked at some of the site source code (Perl) I can attest there are places where the comparison of Perl code to line noise is an apt description. Also, some of the code we inherited was written by, um, creative people who did not write the most readable code. Further the code documents what it does, but is just a wee bit short on the why. Translation: we have an amazing dev team here who have slogged many many hours trying to isolate and correct the issues that have arisen. If you've ever been bleary-eyed after a several-day coding sprint, you have an idea of things. I hereby express my personal thanks to the brain-numbing hard work these guys have put in for this site. And now on to where things stand.
We had an issue with getting a single comment to display correctly in "Flat" mode which appears to have been caused by issues with specifying the correct page it appeared in. Also, there was a rewrite of this code so things should be better, but watch out for regressions.
There are known issues with accessing the site via TOR most likely because we added a very restrictive Content Security Policy.
The new comment viewing modes "Threaded-TOS" and "Threaded-TNG" have been tweaked.
There is a strong voice to replicate the old "Threaded" behavior and it appears that may be feasible, now that we better understand how the community used it in the past. No promises, but it is being looked into.
We are close to making some changes for the defaults for Anonymous Cowards (non logged-in users), so if you have a preference, please speak up and make your voice known.
Oh, we have had reports of seemingly random 503 (Site Unavailable) errors. If you should experience one, please reply to this story with a description of what you were doing and a copy/paste of the entire error message. That will greatly help in our identifying, isolating, and hopefully fixing whatever gremlin is in the gears.
We have not forgotten about replacing chevrons with single/double plus/minus, but had some fires to put out that postponed action on these.
I expect I've left out a thing (or three) — please reply with a comment to (gently) remind us if you see a problem persisting, or if you find something new. it is most helpful to provide your user nickname, the date/time (and timezone), steps taken to cause the problem, and (ideally) suggestions on how you expected it to behave. Reports so far have for the most part been amazingly detailed and helpful — thanks!
Penultimately (I like that word!), I must express my sincere appreciation to the community who has been amazingly supportive and helpful in this transition. One benefit of the upgrade is you should see quicker page-load times on highly-commented stories. Our servers are experiencing a much lighter load to serve up those pages, too. Speaking of servers, I noticed that several of you have renewed your subscription to the site which is the primary way we can afford to keeps the lights on. Please accept my sincere and heart-felt thanks! The "Site News" slashbox has been updated to reflect our current situation.
Lastly, I must express my sincere gratitude to the community. I continue to be amazed at the breadth of knowledge that is freely shared here. Nary a day goes by that I don't learn something new. And many days when I am just blown away. Some long-held ideas have been challenged, and in some cases changed, thanks to what I've read here. Thank you!
Dev Note: Deployed a fix tonight for broken comment links that was due to yesterday's deploy. Alos deployed a partial fix for Flat comments and single comments. TMB will be working on getting it fixed up fully but I thought we needed what we had out now. -- paulej72
Hi there. Martyb again with an update of our progress on issues arising from the site update. (The new comment grouping and display code was necessitated by huge server loads as well as long delays on constructing and returning highly-commented articles.)
First off, please accept my sincere thanks to all of you who made the time to comment in the prior stories and/or engaged us on the #dev channel on IRC. Really! Thank you for your passion and willingness to provide steps to reproduce and ideas for overcoming the issues that have been found.
ACs: If you access the site as an Anonymous Coward, be aware that we have NOT forgotten you. We are still trying to ascertain what features work best for the most people and are holding off changing (and rechanging and...) settings until we have a better idea of what to change those settings to be. So, please speak up on anything that you continue to find problematic and help guide us to making a choice that works the best for the most.
Scrolling Within a Comment: From what I saw in the reports from Monday, one of the key issues had to do with the scrolling within comments. We heard you. Oh, did we ever! Scrolling within comments was quickly removed and replaced by setting a limit on how long a comment could be submitted. This was especially problematic on mobiles and tablets.
Display Modes: Another of the often discussed issues had to do with "Display Mode." This can be set in your preferences (for logged-in users) and ad hoc when you load a story.
Display Mode - Defaults: If, prior to the release you had chosen "Flat", then you were transitioned to "Flat" (Doh!) If you had anything else as your selection for "Display Mode", you were transitioned to "Threaded-TOS". That mode was intended, as best as we were able to do using only CSS, to replicate the behavior previously supported by the old "Threaded" mode. You CAN change this. Many have reported that changing "Threaded-TOS" to "Threaded-TNG" and setting a lower value for "Breakthrough" (in this mode, "Threshold" is ignored) seemed to do the trick.
Display Mode - ad hoc setting: For the ad hoc case, just load the story as you normally would. Below the actual story text and before the comments is a set of controls. If you are having issues with the current default of "Threaded-TOS" click on that text and change it to "Threaded-TNG". if you find you have way too many icons to click in order to read comments, choose a smaller value for Breakthrough (-1 displays all; in this mode Threshold value is ignored).
Spoiler: Another popular topic of discussion was the way the new <spoiler> tag was implemented. We've heard you, but have not as yet decided on a course of action on how to update its functionality... Stay tuned!
*NEW* and/or Dimming: A surprising (to me at least) number of folks had issues with how we flagged old/new comments. For logged-in users, again go to the "Comments" tab of your "preferences" page, scroll down a little, and there are checkboxes that you can toggle:
Highlight New Comments [ ] Highlight new comments with *NEW* tag
Dim Read Comments [ ] Dim already read comments
Please give those a try and see if that works for you. Our first implementation of "Dimming" was a bit too strong for most folk's liking - this has been reduced so as to be less jarring. As for the "*NEW*" text, there were several positive comments that on mobile devices especially, one could quickly search for the text and rapidly navigate comments to find out what was new. There was a suggestion that uppercase-only looks like YELLING. Yes, it does. On the other hand, whatever text is selected for display has to be a reasonably unlikely string to appear in the normal course of reading comments. (False positives, anyone?)
There were some suggestions on changing the color of the comment title to flag it as new. This sounds pretty simple, but the devil is in the details. We have some in our community who are color-blind and others who have very limited vision, if any at all. For them, any color changes could be well nigh invisible. But it gets worse. On the "Homepage" tab of the "Preferences" page, there are currently 11 different themes that one can choose as your default. Setting a new comment to have a lighter (or darker) title bar would not work across all of those disparate themes.
Chevrons: And as for those chevrons that control the display of a single comment and of a comment tree, yes we heard you. Work is underway to see if we can replace those images with single/double plus/minus characters.
Penultimately, I would like to add a call-out to Paulej72 who took point yesterday (giving TheMightyBuzzard a well-deserved respite) and worked tirelessly into the night to address the issues that were raised.
Lastly, again many thanks to you, our community, who have guided us through this transition. Your feedback matters. We listen and for those who have been following along, I hope you can see the changes and the progress. We continue to strive to earn your trust and support. Thank you!
Dev Note: Currently there is an issue with Flat mode and viewing single comments such as https://soylentnews.org/comments.pl?sid=18223&cid=472653. It just came to our attention and we will be working on it to fix it. This issue will cause you to get a server error. Workarounds are to either switch modes to anything other than Flat or avoid going to single comment views.
Continuation of: Site Update 2/27
So, the recent site update got a lot of news, and comments. Predictably, there was a lot of comments split on the fence both ways. I've been out sick and haven't been actively involved in SN in a few days, but I did review the updated changes on dev before they went out. I'm still not up to responding to you guys personally, and TMB/Paul have had things covered, so I'm just going to write a blanket story. So, let's open this and say THIS ISN'T THE FINAL SET OF HOW THINGS WILL BE. I'm leaving my comments above the fold to make it clear what's going on. I'd put that in a blink tag on if that was still in the HTML standard.
The changes to commenting were primarily driven on technical grounds. To do D1.5, the site had to load a mass load of comments and do server side processing to thread them. To give you an example, on a cold page load, before we apply caching a few points in the site would take over a minute to load, render and thread. The only thing that prevented the site from becoming unusable in 503s is that the frontend has a lot of caching. Even with that, we can't cache every single bit of the site at once. In a "cold cache" scenario such as after a varnish or DB update, the site would be borderline unusable until those caches could be loaded. So let me make this clear that this change wasn't a change for changes sake. There was (and is) a need to revamp the commenting.
We noted that this change was coming in other meta stories, and even had a landing article on dev for people coming to check it out. No one did. How we use commenting on dev and how we use it on production are two different things; you can't realistically test these things in real world conditions without updating production.
As TMB stated, we couldn't get the same behavior without making the site cry in the corner, and this was fairly extensively tested on dev before it went live. For older users to the site, you may remember this is not the first time we've changed comments, and rather predictably, the roll out of Improved Commenting actually was fairly buggy. This is a more drastic update.
Right now, we're going to keep improving and changing things to address as many things as possible. To that extent, there will be a daily article for at least this week if not longer to allow for feedback as we work to make things better. If, at the end of all the tweaking, we can't satisfy the vast majority of folks, a revert remains as an available option. We've built this entire site on listening to the community, and taking their feedback into account. That isn't going to change now. I'm hoping we've earned enough trust from you guys collectively to be allowed to at least experiment for a bit.
I'm going to leave the rest of the article for the dev crew to use. Due to personal real life issues, I'm likely not going to be around much, so if you don't see me, that's why. I have full faith in the staff in helping manage and keep things going.
Hi! I'm martyb (aka Bytram) your friendly neighborhood QA/test guy chiming in with my 2¢ on the upgrade/rollout.
Firstly, I apologize that you are seeing ANY issues with the site upgrade. I took this update very seriously and was, unfortunately, only able to perform about half of the testing that I wanted to see done before we went live. That said, there are some issues that were reported that I had not foreseen, so this has been a learning experience for me, too.
Secondly, I'd like to point out what you are NOT seeing -- the many MANY changes that TMB and PJ made as a result of feedback arising from testing. That said, comments are THE thing that makes this site. It's not the timeliness or fine writing of the stories — as I see it, this site is all about providing a venue for discussion.
Look past the fold for the rest of my comments.
Though there were a whole lot of tests that I was able to perform, there were many others that I had still not gotten to yet. I apologize that some of you had to scrape your knuckles on some very rough edges that made it through. In preparation for rollout I had written a series of programs to allow me to automate some aspects of submitting comments in different hierarchies which were key in identifying shortcomings in testing the correct operation of the expand/collapse and hide/show features. I was by no means able to perform an exhaustive test of all of the permutations but I was able to catch a number of issues and I'm sure TMB and PJ will attest that I beat on them pretty hard to make some changes. So far, I've seen no comments complaining about those controls functioning as they should, so YAY on that.
What has not been tested, and for which I hereby request the help of the community, are the user preferences whereby one can provide modifiers to certain aspects of comments. To access these, go to your preferences page, and then click on the "Comments" tab.
Here, you will see a set of modifiers grouped under the header: "Points Modification." The comment's actual score remains unchanged, but these modifiers allow you to provide a nudge to different categories so you could, say, favor "Funny" comments by adding +2 to the score calculation, and hiding all comments modded "Offtopic" by changing that modifier to "-6".
The "Reason Modifiers" are:
Insightful Offtopic Spam Interesting Flamebait Disagree Funny Troll Touché Informative Redundant
The "People Modifiers" are:
Friend Fan Foe Freak Friends-of-Friends fof Foes-of-Friends
And so on with modifiers for Anonymous postings, Karma Bonus, New User Modifiers, Small Comment Modifiers, and Long Comment Modifier.
I would appreciate these being explored and verified as to their correct operation. If you choose to help, please mention in the comments which control you tested, and what happened when you set it to -6, -2, +2, and +6.
These values are suggested so as to explore settings that make a given category nearly hidden (a "+5 Interesting" comment with the "Interesting" modifier set to -6 results in an effective score of -1) — set your threshold/breakthrough to 0 and those comments should not be displayed. Conversely, you can set the "Troll" modifier to +6 so even a "-1 Troll" comment would receive an effective score of +5 and should always appear in the comments you see displayed.
Lastly, but of extreme importance in my mind, is how impressed I am by the community feedback. Issues were stated, explained why it was problematic, steps required to reproduce, steps taken as an attempt at a workaround -- THIS is what keeps me going and donating my time to this site. We are working together to make this the best site we can. I'm proud to be a member of this community. Together I'm sure we can get the remaining issues worked out to people's satisfaction. And, as NCommander stated, if we are not able to do so, there is a fallback to the old approach. I must admit that some of the new features were a bit jarring to me (I started reading at the green site before it even had UIDs) so there's some long-practice reading/viewing skills that are being challenged, but overall I'm liking the changes. I hope you do, too.
Okay, I know it's been a long time since we did one of these but life does intrude on volunteer dev time. Hopefully this one will be worth the wait. Bear with me if I seem a bit off today, I'm writing this with a really fun head cold.
First, what didn't make it into this update but is directly upcoming. Bitpay is still down on account of them changing the API without notifying existing customers or versioning the new API and leaving the old one still up and functional. It's the first thing I'm going to work on after we get this update rolled out but it will basically require a complete rewrite. Don't expect it any earlier than two months from now because we like to test the complete hell out of any code that deals with your money.
Also, adding a Jobs nexus didn't quite make the cut because we're not entirely sure how/if we want to work it. One thing we are certain about, it would not be for headhunters or HR drones to spam us silly but for registered members who have a specific vacancy they need to fill and would like to throw it open to the community.
The API still has some broken bits but it's been low priority compared to what I've been busy with. I'm thinking I'll jump on it after Bitpay unless paulej72 cracks the whip and makes me fix bugs/implement features instead.
There were several other things that I had lined up for post-Bitpay but I can't remember them just now what with my head feeling like it's stuffed full of dirty gym socks.
Now let's throw the list of what did make it out there and go over it in more detail afterwards.
Morning Update: Really digging the constructive criticism. Some quality thoughts in there. Keep them coming and we'll see how fast we can get a few done. --TMB
Before the specifics, I know some of you are going to see the new Threaded modes and be like "that's pretty awesome" and some of you are going to call us dev types very bad names. Well, this ain't the other site. We're not saying "You Shall Use This Because It's New And Shiny". We're saying something had to be done about page load times approaching a full minute on heavily trafficked stories and the way we pulled and rendered comments made up nearly all of that time.
So, the first thing we did was we stopped pulling every single comment and then removing the ones we didn't want to display. Mostly that means that the comment counts in the dropdown menus for Threshold and Breakthrough are on a per-page basis now.
Third, we paginated every mode. I know it was nice being able to see every comment on one page but that meant pulling and rendering every comment and that simply doesn't work if a story has over a hundred comments.
The removal of sorting by score we can't roll back though. Its loss was a necessity due to the way we pull and sort only the comments that the user actually requests. Previously, we were pulling every single comment for a story and then removing the ones we didn't want. That was both bloody stupid and slow as hell, so it had to go. Unfortunately it means we have to do things slightly differently. It may make a triumphant return eventually but it would require some moderately tricky coding with the particular way our code is laid out.
Oh and if you have objections to the new Threaded modes, by all means bitch about specifics in comments here and we'll see what we can do to address them. After having spent so much time recently bashing on exactly these bits of code, we're quite familiar with them and changes/additions shouldn't take too terribly long to whip out.
Now to the specifics.
Flat: Flat is still flat but now with a collapse/expand button that functions like the ones from Improved Threaded.
Threaded-TOS: If you can find significant differences between Improved Threaded and Threaded-TOS, let us know because it's probably a bug. The idea was to make it as much like Improved Threaded as technically possible with just CSS but paginated like Nested so we don't have to render more than 100 comments at a go. We defaulted everyone on Nested/Threaded/Improved threaded to Threaded-TOS to minimize the aggravation of unexpected change. Oh, and Breakthrough now takes precedence over Threshold, so high scoring comments will always be visible even if they're responding to blatant trolling.
Threaded-TNG: All comment trees start fully branched out but with the individual comments either expanded or collapsed. "Comment Below Threshold" functionality is gone. Breakthrough gets compared to a comment's score to decide if it gets expanded or collapsed. Play with it a couple minutes; it's not terribly hard to grok. Why do we need this mode if TOS covers most all of the best bits of the three old modes? Because I like it. You don't have to use it. Shut up.
Why not leave the old comment rendering modes in as well as the new ones? Because by rewriting them we got a rendering speed increase around a factor of two+, to go with the factor of two+ increase we got by pulling only the necessary comments instead of every last comment a story has with every page load. This has been becoming necessary as we increasingly go way above the 100 comment mark on busy stories. It's not cool for you lot to have to wait forty-five seconds to load a page of comments and it's even less cool to peg a cpu core for forty-five seconds to deliver it to you. If you ever again find a story that takes 10+s to load, something's going wrong and we'd appreciate a heads up. We think there's still some room in the code for improvement but this was the lowest-hanging fruit.
Now on to the rest of the details.
The Content Security Policy should cover what's required for operation of this site (plus allowing for Stripe payments) and nothing else. If your browser honors CSPs, it should not be possible to get smacked with XSS or inline script injection on this site any more; even if we write code buggy enough to allow it, which we have once or twice.
On dimmed comments... This only functions for logged in users currently as it would take some serious work to get it functioning for individual ACs, even using cookies. What it does is when you load a page of comments, it picks the highest comment ID from that story and marks that comment as read by you. Switching between pages of comments or changing your Threshold/sort order should not update which comments you have read, even if new ones have come in since your last read comment ID was set. Hitting the "Mark All as Read" button or hitting your browser's Refresh button on the main story page should take the stored comment ID and set the opacity to 60% on all the comments with a comment ID equal to or less than that. It's not entirely accurate but it's pretty damned close and it doesn't bloat the db much at all. Oh and read histories get wiped after two weeks of not being updated for a particular user/story combination to save on db space as well.
The new comment badge functions exactly opposite of dimmed comments. It puts "* NEW *" in the title bar of comments you haven't read yet. It's there strictly so you can have the same functionality but dislike the aesthetics of comment dimming. You can technically use both if you really want new comments to stand out but that would just be weird.
Returning to where you last moderated works like this. If you moderate one comment, you'll get sent back to that comment. If you moderate several in one go, you should get sent to the one farthest down the page. Moderating does not update the comment ID of what you've read for dimming purposes.
Returning to where you just made a comment? That's pretty self-explanatory. It also should not update the comment ID of what you've read for dimming purposes.
The Politics nexus. This does not mean we're looking to have even more political stories. The balance of tech/science/etc... to political stories is not going to change nor will the quality of accepted political submissions. It's primarily a way to let people who are sick and bloody tired of seeing politics here set a preference and never see political stories again. It's also handy if you wish to see what political stories we've run recently as clicking on the nexus link on the left of the page will show you only those stories.
The Reviews nexus has been brought up three separate times that I can remember by different groups of people, so we decided to go ahead with it. It's going to be a book/film/software/hardware/etc... review and discussion place. By my understanding, though I'm not really involved, it's getting its own space because some folks wanted to start what amounts to a site book club. Tech books will of course be welcome but it's open to all genres of printed and bound words. Ditto non-book reviews. Just don't go sending in a review of something we normally wouldn't publish news about on the site. Not enough people are going to be interested in your review of the barber shop down the street from your house, so it won't get published.
Spoiler tags, <spoiler>text you don't want casually seen</spoiler>, work both in stories and comments and are just a bit of css trickery that hide the text between them until the person viewing them hovers over the *SPOILER* text. There's a slight delay, so don't think it's not working because it's not immediate. That's intentional so you don't accidentally trigger showing the contained text by briefly crossing it.
By popular demand, <del> tags were also added.
That's all worth mentioning in this site update. Look for another one hopefully in May or late April. If you find any bugs, please slap them up as issues on our github repo or email them to firstname.lastname@example.org.
Two of SoylentNews' staff submitted stories noting our three-year anniversary; one a site summary of where we are and a summary of what we've done, and the other a detailed presentation of the very early days and how SoylentNews got started.
Three years ago, today, SoylentNews announced its presence to the world. Much has happened along the way of our providing a place for a community to grow and to engage in discussion.
It started as a fork of five-year-old, open-sourced code which had suffered under benign neglect. Perl, Apache, MySQL, and other products had continued on. So we had to deal with dependencies on unsupported and back-level versions of code. A great deal of effort went into bringing the site up-to-date with current versions of that base. See below for mechanicjay's illuminating first-hand account of how that all got started.
Those of you who were with us then can attest to the fact that site outages were a regular occurrence. Bugs were found and eradicated. New bugs were made, and found as well. We invited the community to vote to name the site. We created documents of incorporation and had them dutifully filed. On July 4th, 2014 we received notice of officially becoming SoylentNews PBC. But I get ahead of myself.
Not content with just running a clone of the old code, the staff embarked on a large number of improvements to the site. Support for Unicode characters (via UTF-8) was an early improvement. Refinements to moderation took place — you could now moderate and comment in the same story. Moderation points were issued to every registered user every single day. An API was written and made available. We have our own Folding@Home team (currently ranked 314 of 226132 teams in the world) which contributes spare compute cycles to help find cures to maladies such as Huntington's Disease. (See the Main F@H site and our team page.) We sent out a call for new editors to help our beleaguered editing team which was approaching burnout; several of you answered the call and we are greatly enriched by their viewpoints and their questioning of the status quo.
And what have we wrought? Our own place on the world-wide web, supported and run entirely by the community. For the numerate in our midst, here are some statistics for the site. As of the time of this writing (20170217_002919 UTC), SoylentNews has:
But that's not all! Unwilling to rest on their laurels, our development team has been hard at work bringing improvements to the site — along with some bug fixes. If you want to play with the current, in-development, subject-to-change-without-notice version of the site, hop on over to our development server. Do be aware several specially-crafted stories were created and posted there so as to evoke certain test conditions, so please respect the admonitions stated on those stories. Have an observation, question, or found a bug? We'd love to hear your feedback in the #dev channel on our IRC server.
We could not have done it alone — a great many of you have contributed to the site. There is the administrative tasks of paying the bills and handling legal obligations. Sysops support to keep our boxes up and running. Writing code and patching bugs (while minimizing the bug writing). Suggesting and testing new code/features and providing constructive feedback. Making financial contributions by signing up for subscriptions. Submitting story submissions for the editors to poke and prod at. All of this in support of a goal to provide a place where people can submit comments and engage in discussions with other interesting and intelligent people on the 'net. As with any community, there have been some 'heated' discussions. And most refreshing of all, are those discussions where nuggets of wisdom and brilliance appear — and make the whole effort worthwhile.
So, on behalf of the rest of the all-volunteer staff here at SoylentNews, let me say thank you. For your support, engagement, and questioning — we are a better site because of you. May we continue to earn your trust and support for many years to come.
In the comments, please feel free to mention anything significant that happened over these years which were inadvertently omitted as well as to tell us what we can do better.
So, to wind this up, I have one last question: "emacs or vi?" =)
For our third year, I have some Reflections on our third day.
In some of the pre-history of SoylentNews, here is some of the stuff that gets lost in the mists of time around the first coordinated development effort -- running on a VM, on a laptop in my basement under the slashcott.org domain. The slashcott had been announced and was to commence in some number of days. A bunch of folks thought it would be an awesome idea to get an independent version of slash running in time for the slashcott -- what could go wrong?
3 years and ton of life changes for me, makes some of this a little fuzzy, but I'll do my best to put things together. I've relied heavily on my email archive of that time which helped spur a bunch of memories. Hopefully this will be a coherent tale. (Maybe for next year I'll mine my personal IRC logs from when we were still on freenode).
At first there was a bunch of coordination in the ##slashcode channel on freenode, a bunch of emails were also buzzing around trying to coordinate some things and ideas. My first email to Barrabas was on 02/06/2014 [6 Feb 2014 for our non-US readers]. The issue at hand was that "slashcode" had been hastily open sourced 5 years prior, then pretty well abandoned. Not only did you need to build the perl modules from scratch, but it would only build against Apache 1.x. Once you managed to run that gauntlet, even compiled and installed, things barely ran and were pretty horribly broken. Anyway, it soon became apparent that robinld, NCommander and myself were making the most progress on getting something running, as I recall Robin was the first to success in getting an installed running site, but his VM was stuck behind a corporate firewall.
In the meantime, I had gotten the domain slashcott.org registered while trying to build things myself. At some point, a bunch of us decided to combine forces, Robin shipped me his VM, I got it running on my laptop (as it was the only 64-bit thing I had at the time), we got myself and Ncommander ssh'ed in and we started hacking. For some reason, RedHat vm's were horribly laggy on my openSuse VirtualBox host and work was slow and painful, but progress started to be made.
The only bug I've ever fixed in the code base was a critical piece of the new account email/password generation stuff, as I recall the generated password wasn't actually getting written to the DB. (sadly the evidence of my contribution has been lost, I think I shipped the fix to either robin or ncommander, so they have credit in the git history). Regardless, it was a critical piece - I have an email dated 02/08/2014 with my new account/password, which worked -- it was a huge boon and let us start to let a couple people in to start hammering away to find front-end bugs (of which there were countless). The next big thing I see from mining my email is the first "Nightly stories email", which came out on 02/11/2014 (from the slashcott.org domain). I think we ended up with about 50ish users on slashcott.org (gosh I hope I still have that vmdk stashed somewhere).
On the night of 02/11/2014 (or very early morning of 02/12/2014), after giving up and going to bed (I had a new born and was teaching an undergrad class on the side in addition to my regular 9-5 -- I was beyond toasted after a week). The VM locked up hard (it had done this a couple times, but I was always available to poke it with a stick and bring it back. As I was unavailable and no one had exchanged important things like phone numbers yet, NCommander made the executive decision to spin up a linode, which was great. The laggy VM on the laptop wasn't meant to last forever, though I admit I had visions (delusions?) of hosting the site myself on some real hardware at some point. In retrospect, Linode has been an amazing way to run this site and absolutely the right decision.
I got my new account on the li694-22 domain, on the 02/12/2014, that new account email was for mechanicjay, UID 7 -- which is where I live on the site to this day. I kept the slashcott.org server in sync with code changes for a bit, and was a pretty handy testing platform, until the "official" dev box came online on 02/14/2014. At some point during this week, we had landed on the soylentnews.org domain and that's where we went live on 02/17/2014.
So there you have it, we went from a group of independent pissed off people with no organization and an abandoned broken codebase to launching an honest-to-goodness site in ELEVEN fucking days.
So, in previous posts, I've talked about the fact that SoylentNews currently is powered on Ubuntu 14.04 + a single CentOS 6 box. Right now, the sysops have been somewhat deadlocked on what we should do going forward for our underlying operating system, and I am hoping to get community advice. Right now, the "obvious" choice of what to do is simply do-release-upgrade to Ubuntu 16.04. We've done in-place upgrades before without major issue, and I'm relatively certain we could upgrade without breaking the world. However, from my personal experience, 16.04 introduces systemd support into the stack and is not easily removable. Furthermore, at least in my personal experience, working with journalctl and such has caused me considerable headaches which I detailed in a comment awhile ago.
Discounting systemd itself, I've also found that Ubuntu 16.04 seems less "polished", for want of a better word. I've found I've had to do considerably more fiddling and tweaking to get it to work as a server distro than I had to do with previous releases, as well as had weird issues with LDAP. The same was also true when I worked with recent versions with Debian. As such, there's been a general feeling with the sysops that it's time to go somewhere else.
Below the fold are basically the options as we see them, and I hope if the community can provide some interesting insight or guidance.
Right now, we have about three years before security updates for 14.04 stop, and we are absolutely forced to migrate or upgrade. However, we're already hitting pain due to outdated software; I managed to briefly hose the DNS setup over the weekend trying to deploy CAA records for SN due to our version of BIND being outdated. When TLS 1.3 gets standardized, we're going to have a similar problem with our frontend load balancers. As such, I want to get a plan in place for migration so we can start upgrading over the next year instead of panicking and having to do something at the last moment
As with any discussion for server operating system, knowing what our workloads and such is an important consideration. In short, this is what we use for SN, and the software we have to support
In addition, we use mandatory application controls (AppArmor) to limit the amount of stuff a given process can access for critical services to try and help harden security. We'd like to maintain support for this feature to whatever we migrate, either continuing with AppArmor, switching to SELinux, or using jails/zones if we switch operating systems entirely.
Right now, we've floated a few options, but we're willing to hear more.
The first choice is simply migrate over to a distribution where systemd is not present or completely optional. As of writing, Arch Linux, Gentoo, and Slackware are three such options. Our requirements for a Linux distribution is a good record of updates and security support as I don't wish to be upgrading the system once a week to a new release.
I'm aware of the Devuan project, and at first glance, it would seem like an obvious choice; Debian without systemd is the de-facto tagline. However, I've got concerns about the long-term suitability of the distribution, as well as an intentional choice to replace much of the time-tested Debian infrastructure such as the testing archive with a git-powered Jenkins instance in it's place. Another option would be slackware, but Slackware has made no indication that they won't adapt systemd, and is historically very weak with in-place upgrading and package management in general. Most of the other distributions on without-systemd.org are either LiveCDs, or are very small minority distros that I would be hesitant to bet the farm on with.
On the other side of the coin, and an option favored by at least some of the staff is to migrate to Gentoo or Arch, which are rolling-release. For those unaware, a rolling release distribution basically always has the latest version of everything. Security updates are handled simply by updating to the latest upstream package for the most part. I'm not a huge fan of this option, as we're dependent on self-built software, and it's not unheard of for "emerge world" to break things during upgrades due to feature changes and such. It would essentially require us to manually be checking release notes, and crossing our fingers every time we did a major upgrade. We could reduce some of this pain by simply migrating all our infrastructure to the form of ebuilds so that at least they would get rebuild as part of upgrading, but I'm very very hesitant about this option as a whole, especially for multiple machines.
Another way we could handle the problem is simply jump off the Linux ship entirely. From a personal perspective, I'm not exactly thrilled on the way Linux as a collective whole has gone for several years, and I see the situation only getting worse with time. As an additional benefit, switching off Linux gives us the possiblity of using real containers and ZFS, which would allow us to further isolate components of the stack, and give us the option to do rollbacks if ever necessary on a blocked upgrade; something that is difficult to impossible with most Linux distributions. As such, I've been favoring this option personally, though I'm not sold enough to make the jump. Two major options attract me of these two:
FreeBSD has been around a long time, and has both considerable developer support, and support for a lot of features we'd like such as ZFS, jails, and a sane upstream. FreeBSD is split into two components, the core stack which is what constitutes a release, and the ports collection which is add-on software. Both can be upgraded (somewhat) independently of each other, so we won't have as much pain with outdated server components. We'd also have the ability to easy create jails for things like rehash, MySQL, and such and easily isolate these components from each other in a way that's more iron-clad than AppArmor or SELinux.
illumos is descended from OpenSolaris, and forked after Oracle closed up the source code for Solaris 11. Development has continued on it (at a, granted, slower place). Being the originator of ZFS, it has class A support for it, as well as zones which are functionally equivalent to FreeBSD jails. illumos also has support for SMF, which is essentially advanced service management and tracking without all the baggage systemd creates and tendrils throughout the stack. Zones can also be branded to run Linux binaries to some extent so we can handle migrating the core system over by simply installing illumos, restoring a backup into a branded zone, and then piecemeal decommissioning of said zone. As such, as an upgrade choice, this is fairly attractive. If we migrate to illumos, we'll either use the SmartOS distribution, or OpenIndiana.
Right now, we're basically on the fence with all options, so hopefully the community can provide their own input, or suggest other options we're not aware of. I look forward to your comments below!
Earlier today, we ran an article detailing that Oracle released 270 critical security updates for many of its products, including MySQL cluster which we use here to provide high uptime and reliability for SoylentNews. Needless to say, it was time to upgrade both NDB backends, and the four MySQLd frontends. While the upgrade did not go completely smoothly due to the fact that MySQL strict mode got enabled, and broke the site briefly, our total downtime was less than five minutes or so. Right now, we had to do a full flush and purge of all caches, which means the site is running a bit larky until they can repopulate but I'm pleased to announce we're up to date and secure!
ndb_mgm> show Cluster Configuration --------------------- [ndbd(NDB)] 2 node(s) id=2 @redacted (mysql-5.7.17 ndb-7.5.5, Nodegroup: 0) id=3 @redacted (mysql-5.7.17 ndb-7.5.5, Nodegroup: 0, *) [ndb_mgmd(MGM)] 2 node(s) id=101 @redacted (mysql-5.7.17 ndb-7.5.5) id=102 @redacted (mysql-5.7.17 ndb-7.5.5) [mysqld(API)] 4 node(s) id=11 @redacted (mysql-5.7.17 ndb-7.5.5) id=12 @redacted (mysql-5.7.17 ndb-7.5.5) id=13 @redacted (mysql-5.7.17 ndb-7.5.5) id=14 @redacted (mysql-5.7.17 ndb-7.5.5)
If you notice any unusual breakages or slowdowns, please let me know in the comments. Otherwise, keep calm and carry on!