Slash Boxes

SoylentNews is people


Log In

Log In

Create Account  |  Retrieve Password

Site News

Join our Folding@Home team:
Main F@H site
Our team page

Funding Goal
For 6-month period:
2019-01-01 to 2019-06-30
(All amounts are estimated)
Base Goal:


Covers transactions:
2019-01-01 00:00:00 ..
2019-01-12 01:33:11 UTC
(SPIDs: [1022..1031])
Last Update:
2019-01-13 15:56:06 UTC

Support us: Subscribe Here
and buy SoylentNews Swag

We always have a place for talented people, visit the Get Involved section on the wiki to see how you can make SoylentNews better.

posted by NCommander on Monday February 27 2017, @12:51PM   Printer-friendly
from the we-knew-there-would-be-pitchforks dept.

Continuation of: Site Update 2/27

So, the recent site update got a lot of news, and comments. Predictably, there was a lot of comments split on the fence both ways. I've been out sick and haven't been actively involved in SN in a few days, but I did review the updated changes on dev before they went out. I'm still not up to responding to you guys personally, and TMB/Paul have had things covered, so I'm just going to write a blanket story. So, let's open this and say THIS ISN'T THE FINAL SET OF HOW THINGS WILL BE. I'm leaving my comments above the fold to make it clear what's going on. I'd put that in a blink tag on if that was still in the HTML standard.

The changes to commenting were primarily driven on technical grounds. To do D1.5, the site had to load a mass load of comments and do server side processing to thread them. To give you an example, on a cold page load, before we apply caching a few points in the site would take over a minute to load, render and thread. The only thing that prevented the site from becoming unusable in 503s is that the frontend has a lot of caching. Even with that, we can't cache every single bit of the site at once. In a "cold cache" scenario such as after a varnish or DB update, the site would be borderline unusable until those caches could be loaded. So let me make this clear that this change wasn't a change for changes sake. There was (and is) a need to revamp the commenting.

We noted that this change was coming in other meta stories, and even had a landing article on dev for people coming to check it out. No one did. How we use commenting on dev and how we use it on production are two different things; you can't realistically test these things in real world conditions without updating production.

As TMB stated, we couldn't get the same behavior without making the site cry in the corner, and this was fairly extensively tested on dev before it went live. For older users to the site, you may remember this is not the first time we've changed comments, and rather predictably, the roll out of Improved Commenting actually was fairly buggy. This is a more drastic update.

Right now, we're going to keep improving and changing things to address as many things as possible. To that extent, there will be a daily article for at least this week if not longer to allow for feedback as we work to make things better. If, at the end of all the tweaking, we can't satisfy the vast majority of folks, a revert remains as an available option. We've built this entire site on listening to the community, and taking their feedback into account. That isn't going to change now. I'm hoping we've earned enough trust from you guys collectively to be allowed to at least experiment for a bit.

I'm going to leave the rest of the article for the dev crew to use. Due to personal real life issues, I'm likely not going to be around much, so if you don't see me, that's why. I have full faith in the staff in helping manage and keep things going.

~ NCommander

Hi! I'm martyb (aka Bytram) your friendly neighborhood QA/test guy chiming in with my 2¢ on the upgrade/rollout.

Firstly, I apologize that you are seeing ANY issues with the site upgrade. I took this update very seriously and was, unfortunately, only able to perform about half of the testing that I wanted to see done before we went live. That said, there are some issues that were reported that I had not foreseen, so this has been a learning experience for me, too.

Secondly, I'd like to point out what you are NOT seeing -- the many MANY changes that TMB and PJ made as a result of feedback arising from testing. That said, comments are THE thing that makes this site. It's not the timeliness or fine writing of the stories — as I see it, this site is all about providing a venue for discussion.

Look past the fold for the rest of my comments.

Though there were a whole lot of tests that I was able to perform, there were many others that I had still not gotten to yet. I apologize that some of you had to scrape your knuckles on some very rough edges that made it through. In preparation for rollout I had written a series of programs to allow me to automate some aspects of submitting comments in different hierarchies which were key in identifying shortcomings in testing the correct operation of the expand/collapse and hide/show features. I was by no means able to perform an exhaustive test of all of the permutations but I was able to catch a number of issues and I'm sure TMB and PJ will attest that I beat on them pretty hard to make some changes. So far, I've seen no comments complaining about those controls functioning as they should, so YAY on that.

What has not been tested, and for which I hereby request the help of the community, are the user preferences whereby one can provide modifiers to certain aspects of comments. To access these, go to your preferences page, and then click on the "Comments" tab.

Here, you will see a set of modifiers grouped under the header: "Points Modification." The comment's actual score remains unchanged, but these modifiers allow you to provide a nudge to different categories so you could, say, favor "Funny" comments by adding +2 to the score calculation, and hiding all comments modded "Offtopic" by changing that modifier to "-6".

The "Reason Modifiers" are:
Insightful Offtopic Spam Interesting Flamebait Disagree Funny Troll Touché Informative Redundant

The "People Modifiers" are:
Friend Fan Foe Freak Friends-of-Friends fof Foes-of-Friends

And so on with modifiers for Anonymous postings, Karma Bonus, New User Modifiers, Small Comment Modifiers, and Long Comment Modifier.

I would appreciate these being explored and verified as to their correct operation. If you choose to help, please mention in the comments which control you tested, and what happened when you set it to -6, -2, +2, and +6.

These values are suggested so as to explore settings that make a given category nearly hidden (a "+5 Interesting" comment with the "Interesting" modifier set to -6 results in an effective score of -1) — set your threshold/breakthrough to 0 and those comments should not be displayed. Conversely, you can set the "Troll" modifier to +6 so even a "-1 Troll" comment would receive an effective score of +5 and should always appear in the comments you see displayed.

Lastly, but of extreme importance in my mind, is how impressed I am by the community feedback. Issues were stated, explained why it was problematic, steps required to reproduce, steps taken as an attempt at a workaround -- THIS is what keeps me going and donating my time to this site. We are working together to make this the best site we can. I'm proud to be a member of this community. Together I'm sure we can get the remaining issues worked out to people's satisfaction. And, as NCommander stated, if we are not able to do so, there is a fallback to the old approach. I must admit that some of the new features were a bit jarring to me (I started reading at the green site before it even had UIDs) so there's some long-practice reading/viewing skills that are being challenged, but overall I'm liking the changes. I hope you do, too.

posted by The Mighty Buzzard on Sunday February 26 2017, @01:52AM   Printer-friendly
from the not-actually-an-NCommander-post dept.

Okay, I know it's been a long time since we did one of these but life does intrude on volunteer dev time. Hopefully this one will be worth the wait. Bear with me if I seem a bit off today, I'm writing this with a really fun head cold.

First, what didn't make it into this update but is directly upcoming. Bitpay is still down on account of them changing the API without notifying existing customers or versioning the new API and leaving the old one still up and functional. It's the first thing I'm going to work on after we get this update rolled out but it will basically require a complete rewrite. Don't expect it any earlier than two months from now because we like to test the complete hell out of any code that deals with your money.

Also, adding a Jobs nexus didn't quite make the cut because we're not entirely sure how/if we want to work it. One thing we are certain about, it would not be for headhunters or HR drones to spam us silly but for registered members who have a specific vacancy they need to fill and would like to throw it open to the community.

The API still has some broken bits but it's been low priority compared to what I've been busy with. I'm thinking I'll jump on it after Bitpay unless paulej72 cracks the whip and makes me fix bugs/implement features instead.

There were several other things that I had lined up for post-Bitpay but I can't remember them just now what with my head feeling like it's stuffed full of dirty gym socks.

Now let's throw the list of what did make it out there and go over it in more detail afterwards.

  • Tweaked the themes a bit where they were off.
  • Changed or fixed some adminy/editory stuff that most of you will never see or care about.
  • Fixed a mess of minor bugs not worth noting individually.
  • Improved Rehash installation. It should almost be possible to just follow directions and have a site working in an hour or two now.
  • Added a very restrictive Content Security Policy.
  • Added a link to the Hall of Fame. It was always there, just not linked to.
  • Return to where you just moderated after moderating. (yay!)
  • Return to where you just were after commenting. (yay some more!)
  • Added a field for department on submissions. Editors get final say but if you have a good one, go for it.
  • Added a Community Reviews nexus.
  • Added a Politics nexus.
  • Added <spoiler> tags for the Reviews nexus in case you want to talk about a novel without ruining it for everyone else. They function everywhere though.
  • Changed really freaking long comments to have a scrollbar now instead of being click-to-show.
  • Massively sped up comment rendering on heavily commented stories.
  • Dimming of comments you've already read. (You can turn this off with the controls on the "Comments" tab of your preferences page if it annoys you.)
  • Added a "*NEW*" badge to new comments in case you don't like dimming but still want to easily see new posts. (Disable it the same place as above.)
  • Removed Nested, Threaded, and Improved threaded comment rendering modes (Necessary due to the changes required for the massive speed-up)
  • Added Threaded-TOS and Threaded-TNG comment rendering modes. (TOS is the default)
  • All comment modes now feature collapsible/expandable comments. (Without javascript)

Morning Update: Really digging the constructive criticism. Some quality thoughts in there. Keep them coming and we'll see how fast we can get a few done. --TMB

Before the specifics, I know some of you are going to see the new Threaded modes and be like "that's pretty awesome" and some of you are going to call us dev types very bad names. Well, this ain't the other site. We're not saying "You Shall Use This Because It's New And Shiny". We're saying something had to be done about page load times approaching a full minute on heavily trafficked stories and the way we pulled and rendered comments made up nearly all of that time.

So, the first thing we did was we stopped pulling every single comment and then removing the ones we didn't want to display. Mostly that means that the comment counts in the dropdown menus for Threshold and Breakthrough are on a per-page basis now.

Next we did away with templates for comments. Wildcarded, case insensitive search and replace, even in perl, is horribly slow and that's a large part of how templates worked. The html and related logic is now hardcoded into the source. This did mean though that we had to entirely rewrite all the comment modes logic. Flat and Threaded-TOS are pretty much identical to the old Flat and Threaded, so there shouldn't be any surprises there except that we got rid of the javascript in Improved Threaded and gave every mode collapsible comments with nothing but CSS. Threaded-TNG is new-ish however. It's essentially Nested but without Threshold or every top-level comment being fully visible. If Nested users absolutely cannot live with that, we'll preempt working on the bitcoin rewrite and slap a Nested mode in as well. It shouldn't take but a week, testing included.

Third, we paginated every mode. I know it was nice being able to see every comment on one page but that meant pulling and rendering every comment and that simply doesn't work if a story has over a hundred comments.

The removal of sorting by score we can't roll back though. Its loss was a necessity due to the way we pull and sort only the comments that the user actually requests. Previously, we were pulling every single comment for a story and then removing the ones we didn't want. That was both bloody stupid and slow as hell, so it had to go. Unfortunately it means we have to do things slightly differently. It may make a triumphant return eventually but it would require some moderately tricky coding with the particular way our code is laid out.

Oh and if you have objections to the new Threaded modes, by all means bitch about specifics in comments here and we'll see what we can do to address them. After having spent so much time recently bashing on exactly these bits of code, we're quite familiar with them and changes/additions shouldn't take too terribly long to whip out.

Now to the specifics.

The buttons on the upper left of each comment don't work exactly like the Javascript version did but we do like how they work. The double chevron either shows or hides the comment tree beneath a comment but it does not change their collapsed/expanded state. The single chevron controls the expanded/collapsed state of each comment individually. Adding another button to expand/collapse every individual comment beneath a given comment may be doable but we haven't figured out how so far. It is high on the wish list but not high enough to delay the release any longer than it already has been.

Flat: Flat is still flat but now with a collapse/expand button that functions like the ones from Improved Threaded.

Threaded-TOS: If you can find significant differences between Improved Threaded and Threaded-TOS, let us know because it's probably a bug. The idea was to make it as much like Improved Threaded as technically possible with just CSS but paginated like Nested so we don't have to render more than 100 comments at a go. We defaulted everyone on Nested/Threaded/Improved threaded to Threaded-TOS to minimize the aggravation of unexpected change. Oh, and Breakthrough now takes precedence over Threshold, so high scoring comments will always be visible even if they're responding to blatant trolling.

Threaded-TNG: All comment trees start fully branched out but with the individual comments either expanded or collapsed. "Comment Below Threshold" functionality is gone. Breakthrough gets compared to a comment's score to decide if it gets expanded or collapsed. Play with it a couple minutes; it's not terribly hard to grok. Why do we need this mode if TOS covers most all of the best bits of the three old modes? Because I like it. You don't have to use it. Shut up.

What happened to Nested? What's old is new again. Threaded-TNG more or less is Nested but with the fun bits of Improved Threaded bolted on as well and without the annoyance of having to allow Javascript to run. Minus Threshold functionality. If you spot any serious differences between the two besides those, give us a heads up, because we didn't. It's a very easy mode to code on though, so if you absolutely cannot live without Threshold it's not at all difficult to clone it, add Threshold back in, and call it Nested.

Why not leave the old comment rendering modes in as well as the new ones? Because by rewriting them we got a rendering speed increase around a factor of two+, to go with the factor of two+ increase we got by pulling only the necessary comments instead of every last comment a story has with every page load. This has been becoming necessary as we increasingly go way above the 100 comment mark on busy stories. It's not cool for you lot to have to wait forty-five seconds to load a page of comments and it's even less cool to peg a cpu core for forty-five seconds to deliver it to you. If you ever again find a story that takes 10+s to load, something's going wrong and we'd appreciate a heads up. We think there's still some room in the code for improvement but this was the lowest-hanging fruit.

Now on to the rest of the details.

The Content Security Policy should cover what's required for operation of this site (plus allowing for Stripe payments) and nothing else. If your browser honors CSPs, it should not be possible to get smacked with XSS or inline script injection on this site any more; even if we write code buggy enough to allow it, which we have once or twice.

On dimmed comments... This only functions for logged in users currently as it would take some serious work to get it functioning for individual ACs, even using cookies. What it does is when you load a page of comments, it picks the highest comment ID from that story and marks that comment as read by you. Switching between pages of comments or changing your Threshold/sort order should not update which comments you have read, even if new ones have come in since your last read comment ID was set. Hitting the "Mark All as Read" button or hitting your browser's Refresh button on the main story page should take the stored comment ID and set the opacity to 60% on all the comments with a comment ID equal to or less than that. It's not entirely accurate but it's pretty damned close and it doesn't bloat the db much at all. Oh and read histories get wiped after two weeks of not being updated for a particular user/story combination to save on db space as well.

The new comment badge functions exactly opposite of dimmed comments. It puts "* NEW *" in the title bar of comments you haven't read yet. It's there strictly so you can have the same functionality but dislike the aesthetics of comment dimming. You can technically use both if you really want new comments to stand out but that would just be weird.

Returning to where you last moderated works like this. If you moderate one comment, you'll get sent back to that comment. If you moderate several in one go, you should get sent to the one farthest down the page. Moderating does not update the comment ID of what you've read for dimming purposes.

Returning to where you just made a comment? That's pretty self-explanatory. It also should not update the comment ID of what you've read for dimming purposes.

The Politics nexus. This does not mean we're looking to have even more political stories. The balance of tech/science/etc... to political stories is not going to change nor will the quality of accepted political submissions. It's primarily a way to let people who are sick and bloody tired of seeing politics here set a preference and never see political stories again. It's also handy if you wish to see what political stories we've run recently as clicking on the nexus link on the left of the page will show you only those stories.

The Reviews nexus has been brought up three separate times that I can remember by different groups of people, so we decided to go ahead with it. It's going to be a book/film/software/hardware/etc... review and discussion place. By my understanding, though I'm not really involved, it's getting its own space because some folks wanted to start what amounts to a site book club. Tech books will of course be welcome but it's open to all genres of printed and bound words. Ditto non-book reviews. Just don't go sending in a review of something we normally wouldn't publish news about on the site. Not enough people are going to be interested in your review of the barber shop down the street from your house, so it won't get published.

Spoiler tags, <spoiler>text you don't want casually seen</spoiler>, work both in stories and comments and are just a bit of css trickery that hide the text between them until the person viewing them hovers over the *SPOILER* text. There's a slight delay, so don't think it's not working because it's not immediate. That's intentional so you don't accidentally trigger showing the contained text by briefly crossing it.

By popular demand, <del> tags were also added.

That's all worth mentioning in this site update. Look for another one hopefully in May or late April. If you find any bugs, please slap them up as issues on our github repo or email them to

posted by martyb on Friday February 17 2017, @02:06AM   Printer-friendly
from the long-and-winding-road dept.

Two of SoylentNews' staff submitted stories noting our three-year anniversary; one a site summary of where we are and a summary of what we've done, and the other a detailed presentation of the very early days and how SoylentNews got started.

Three Whole Years -- Thanks to You!

Three years ago, today, SoylentNews announced its presence to the world. Much has happened along the way of our providing a place for a community to grow and to engage in discussion.

It started as a fork of five-year-old, open-sourced code which had suffered under benign neglect. Perl, Apache, MySQL, and other products had continued on. So we had to deal with dependencies on unsupported and back-level versions of code. A great deal of effort went into bringing the site up-to-date with current versions of that base. See below for mechanicjay's illuminating first-hand account of how that all got started.


Those of you who were with us then can attest to the fact that site outages were a regular occurrence. Bugs were found and eradicated. New bugs were made, and found as well. We invited the community to vote to name the site. We created documents of incorporation and had them dutifully filed. On July 4th, 2014 we received notice of officially becoming SoylentNews PBC. But I get ahead of myself.

Not content with just running a clone of the old code, the staff embarked on a large number of improvements to the site. Support for Unicode characters (via UTF-8) was an early improvement. Refinements to moderation took place — you could now moderate and comment in the same story. Moderation points were issued to every registered user every single day. An API was written and made available. We have our own Folding@Home team (currently ranked 314 of 226132 teams in the world) which contributes spare compute cycles to help find cures to maladies such as Huntington's Disease. (See the Main F@H site and our team page.) We sent out a call for new editors to help our beleaguered editing team which was approaching burnout; several of you answered the call and we are greatly enriched by their viewpoints and their questioning of the status quo.

And what have we wrought? Our own place on the world-wide web, supported and run entirely by the community. For the numerate in our midst, here are some statistics for the site. As of the time of this writing (20170217_002919 UTC), SoylentNews has:

  • conducted 96 polls
  • posted 2098 user journal articles
  • registered 6496 user nicknames
  • published 15660 stories
  • received 18611 story submissions
  • posted 462690 comments
  • had 52699108 hits on stories

But that's not all! Unwilling to rest on their laurels, our development team has been hard at work bringing improvements to the site — along with some bug fixes. If you want to play with the current, in-development, subject-to-change-without-notice version of the site, hop on over to our development server. Do be aware several specially-crafted stories were created and posted there so as to evoke certain test conditions, so please respect the admonitions stated on those stories. Have an observation, question, or found a bug? We'd love to hear your feedback in the #dev channel on our IRC server.

We could not have done it alone — a great many of you have contributed to the site. There is the administrative tasks of paying the bills and handling legal obligations. Sysops support to keep our boxes up and running. Writing code and patching bugs (while minimizing the bug writing). Suggesting and testing new code/features and providing constructive feedback. Making financial contributions by signing up for subscriptions. Submitting story submissions for the editors to poke and prod at. All of this in support of a goal to provide a place where people can submit comments and engage in discussions with other interesting and intelligent people on the 'net. As with any community, there have been some 'heated' discussions. And most refreshing of all, are those discussions where nuggets of wisdom and brilliance appear — and make the whole effort worthwhile.

So, on behalf of the rest of the all-volunteer staff here at SoylentNews, let me say thank you. For your support, engagement, and questioning — we are a better site because of you. May we continue to earn your trust and support for many years to come.

In the comments, please feel free to mention anything significant that happened over these years which were inadvertently omitted as well as to tell us what we can do better.

So, to wind this up, I have one last question: "emacs or vi?" =)

Reflections on our First Days

For our third year, I have some Reflections on our third day.

In some of the pre-history of SoylentNews, here is some of the stuff that gets lost in the mists of time around the first coordinated development effort -- running on a VM, on a laptop in my basement under the domain. The slashcott had been announced and was to commence in some number of days. A bunch of folks thought it would be an awesome idea to get an independent version of slash running in time for the slashcott -- what could go wrong?

3 years and ton of life changes for me, makes some of this a little fuzzy, but I'll do my best to put things together. I've relied heavily on my email archive of that time which helped spur a bunch of memories. Hopefully this will be a coherent tale. (Maybe for next year I'll mine my personal IRC logs from when we were still on freenode).

At first there was a bunch of coordination in the ##slashcode channel on freenode, a bunch of emails were also buzzing around trying to coordinate some things and ideas. My first email to Barrabas was on 02/06/2014 [6 Feb 2014 for our non-US readers]. The issue at hand was that "slashcode" had been hastily open sourced 5 years prior, then pretty well abandoned. Not only did you need to build the perl modules from scratch, but it would only build against Apache 1.x. Once you managed to run that gauntlet, even compiled and installed, things barely ran and were pretty horribly broken. Anyway, it soon became apparent that robinld, NCommander and myself were making the most progress on getting something running, as I recall Robin was the first to success in getting an installed running site, but his VM was stuck behind a corporate firewall.

In the meantime, I had gotten the domain registered while trying to build things myself. At some point, a bunch of us decided to combine forces, Robin shipped me his VM, I got it running on my laptop (as it was the only 64-bit thing I had at the time), we got myself and Ncommander ssh'ed in and we started hacking. For some reason, RedHat vm's were horribly laggy on my openSuse VirtualBox host and work was slow and painful, but progress started to be made.

The only bug I've ever fixed in the code base was a critical piece of the new account email/password generation stuff, as I recall the generated password wasn't actually getting written to the DB. (sadly the evidence of my contribution has been lost, I think I shipped the fix to either robin or ncommander, so they have credit in the git history). Regardless, it was a critical piece - I have an email dated 02/08/2014 with my new account/password, which worked -- it was a huge boon and let us start to let a couple people in to start hammering away to find front-end bugs (of which there were countless). The next big thing I see from mining my email is the first "Nightly stories email", which came out on 02/11/2014 (from the domain). I think we ended up with about 50ish users on (gosh I hope I still have that vmdk stashed somewhere).

On the night of 02/11/2014 (or very early morning of 02/12/2014), after giving up and going to bed (I had a new born and was teaching an undergrad class on the side in addition to my regular 9-5 -- I was beyond toasted after a week). The VM locked up hard (it had done this a couple times, but I was always available to poke it with a stick and bring it back. As I was unavailable and no one had exchanged important things like phone numbers yet, NCommander made the executive decision to spin up a linode, which was great. The laggy VM on the laptop wasn't meant to last forever, though I admit I had visions (delusions?) of hosting the site myself on some real hardware at some point. In retrospect, Linode has been an amazing way to run this site and absolutely the right decision.

I got my new account on the li694-22 domain, on the 02/12/2014, that new account email was for mechanicjay, UID 7 -- which is where I live on the site to this day. I kept the server in sync with code changes for a bit, and was a pretty handy testing platform, until the "official" dev box came online on 02/14/2014. At some point during this week, we had landed on the domain and that's where we went live on 02/17/2014.

So there you have it, we went from a group of independent pissed off people with no organization and an abandoned broken codebase to launching an honest-to-goodness site in ELEVEN fucking days.

Original Submission #1Original Submission #2

posted by NCommander on Tuesday February 07 2017, @11:45AM   Printer-friendly
from the insert-systemd-rant-here dept.

So, in previous posts, I've talked about the fact that SoylentNews currently is powered on Ubuntu 14.04 + a single CentOS 6 box. Right now, the sysops have been somewhat deadlocked on what we should do going forward for our underlying operating system, and I am hoping to get community advice. Right now, the "obvious" choice of what to do is simply do-release-upgrade to Ubuntu 16.04. We've done in-place upgrades before without major issue, and I'm relatively certain we could upgrade without breaking the world. However, from my personal experience, 16.04 introduces systemd support into the stack and is not easily removable. Furthermore, at least in my personal experience, working with journalctl and such has caused me considerable headaches which I detailed in a comment awhile ago.

Discounting systemd itself, I've also found that Ubuntu 16.04 seems less "polished", for want of a better word. I've found I've had to do considerably more fiddling and tweaking to get it to work as a server distro than I had to do with previous releases, as well as had weird issues with LDAP. The same was also true when I worked with recent versions with Debian. As such, there's been a general feeling with the sysops that it's time to go somewhere else.

Below the fold are basically the options as we see them, and I hope if the community can provide some interesting insight or guidance.

Right now, we have about three years before security updates for 14.04 stop, and we are absolutely forced to migrate or upgrade. However, we're already hitting pain due to outdated software; I managed to briefly hose the DNS setup over the weekend trying to deploy CAA records for SN due to our version of BIND being outdated. When TLS 1.3 gets standardized, we're going to have a similar problem with our frontend load balancers. As such, I want to get a plan in place for migration so we can start upgrading over the next year instead of panicking and having to do something at the last moment

The SN Software Stack

As with any discussion for server operating system, knowing what our workloads and such is an important consideration. In short, this is what we use for SN, and the software we have to support

  • nginx - Loadbalancing/SSL Termination
  • Apache 2.2 + mod_perl - rehash (we run it with a separate instance of Apache and Perl, and not the system copy)
  • MySQL Cluster for production
  • MySQL standard for secondary services
  • Kerberos + Hesiod - single-signon/authetication
  • Postfix+Squirrelmail - ... mail

In addition, we use mandatory application controls (AppArmor) to limit the amount of stuff a given process can access for critical services to try and help harden security. We'd like to maintain support for this feature to whatever we migrate, either continuing with AppArmor, switching to SELinux, or using jails/zones if we switch operating systems entirely.

The Options

Right now, we've floated a few options, but we're willing to hear more.

A non-systemd Linux distro

The first choice is simply migrate over to a distribution where systemd is not present or completely optional. As of writing, Arch Linux, Gentoo, and Slackware are three such options. Our requirements for a Linux distribution is a good record of updates and security support as I don't wish to be upgrading the system once a week to a new release.

Release-based distributions

I'm aware of the Devuan project, and at first glance, it would seem like an obvious choice; Debian without systemd is the de-facto tagline. However, I've got concerns about the long-term suitability of the distribution, as well as an intentional choice to replace much of the time-tested Debian infrastructure such as the testing archive with a git-powered Jenkins instance in it's place. Another option would be slackware, but Slackware has made no indication that they won't adapt systemd, and is historically very weak with in-place upgrading and package management in general. Most of the other distributions on are either LiveCDs, or are very small minority distros that I would be hesitant to bet the farm on with.

Rolling-release distributions

On the other side of the coin, and an option favored by at least some of the staff is to migrate to Gentoo or Arch, which are rolling-release. For those unaware, a rolling release distribution basically always has the latest version of everything. Security updates are handled simply by updating to the latest upstream package for the most part. I'm not a huge fan of this option, as we're dependent on self-built software, and it's not unheard of for "emerge world" to break things during upgrades due to feature changes and such. It would essentially require us to manually be checking release notes, and crossing our fingers every time we did a major upgrade. We could reduce some of this pain by simply migrating all our infrastructure to the form of ebuilds so that at least they would get rebuild as part of upgrading, but I'm very very hesitant about this option as a whole, especially for multiple machines.

Switch to FreeBSD/illumos/Other

Another way we could handle the problem is simply jump off the Linux ship entirely. From a personal perspective, I'm not exactly thrilled on the way Linux as a collective whole has gone for several years, and I see the situation only getting worse with time. As an additional benefit, switching off Linux gives us the possiblity of using real containers and ZFS, which would allow us to further isolate components of the stack, and give us the option to do rollbacks if ever necessary on a blocked upgrade; something that is difficult to impossible with most Linux distributions. As such, I've been favoring this option personally, though I'm not sold enough to make the jump. Two major options attract me of these two:


FreeBSD has been around a long time, and has both considerable developer support, and support for a lot of features we'd like such as ZFS, jails, and a sane upstream. FreeBSD is split into two components, the core stack which is what constitutes a release, and the ports collection which is add-on software. Both can be upgraded (somewhat) independently of each other, so we won't have as much pain with outdated server components. We'd also have the ability to easy create jails for things like rehash, MySQL, and such and easily isolate these components from each other in a way that's more iron-clad than AppArmor or SELinux.


illumos is descended from OpenSolaris, and forked after Oracle closed up the source code for Solaris 11. Development has continued on it (at a, granted, slower place). Being the originator of ZFS, it has class A support for it, as well as zones which are functionally equivalent to FreeBSD jails. illumos also has support for SMF, which is essentially advanced service management and tracking without all the baggage systemd creates and tendrils throughout the stack. Zones can also be branded to run Linux binaries to some extent so we can handle migrating the core system over by simply installing illumos, restoring a backup into a branded zone, and then piecemeal decommissioning of said zone. As such, as an upgrade choice, this is fairly attractive. If we migrate to illumos, we'll either use the SmartOS distribution, or OpenIndiana.

Final Notes

Right now, we're basically on the fence with all options, so hopefully the community can provide their own input, or suggest other options we're not aware of. I look forward to your comments below!

~ NCommander

posted by NCommander on Friday January 20 2017, @04:43PM   Printer-friendly
from the hot-upgrading-database-servers-ftw dept.

Earlier today, we ran an article detailing that Oracle released 270 critical security updates for many of its products, including MySQL cluster which we use here to provide high uptime and reliability for SoylentNews. Needless to say, it was time to upgrade both NDB backends, and the four MySQLd frontends. While the upgrade did not go completely smoothly due to the fact that MySQL strict mode got enabled, and broke the site briefly, our total downtime was less than five minutes or so. Right now, we had to do a full flush and purge of all caches, which means the site is running a bit larky until they can repopulate but I'm pleased to announce we're up to date and secure!

ndb_mgm> show
Cluster Configuration
[ndbd(NDB)]	2 node(s)
id=2	@redacted (mysql-5.7.17 ndb-7.5.5, Nodegroup: 0)
id=3	@redacted (mysql-5.7.17 ndb-7.5.5, Nodegroup: 0, *)

[ndb_mgmd(MGM)]	2 node(s)
id=101	@redacted (mysql-5.7.17 ndb-7.5.5)
id=102	@redacted (mysql-5.7.17 ndb-7.5.5)

[mysqld(API)]	4 node(s)
id=11	@redacted (mysql-5.7.17 ndb-7.5.5)
id=12	@redacted (mysql-5.7.17 ndb-7.5.5)
id=13	@redacted (mysql-5.7.17 ndb-7.5.5)
id=14	@redacted (mysql-5.7.17 ndb-7.5.5)

If you notice any unusual breakages or slowdowns, please let me know in the comments. Otherwise, keep calm and carry on!

~ NCommander

posted by martyb on Friday January 20 2017, @07:00AM   Printer-friendly
from the making-a-legacy dept.

A couple months ago we ran a story asking the SoylentNews community for volunteers to help with editing and the community did not let us down; we received a full dozen inquiries! You've probably noticed a few new names art the top of the stories and quite frankly, their contributions made it possible for the staff to survive the holiday season — many, many thanks!

If, for whatever reason, you did not want to be an Editor, but still wish to contribute, there are many other areas:

Submit stories
Click the Submit Story link in the "Navigation" slashbox on the left-hand side of the main page. It is not necessary to write perfect prose (though we sure appreciate it when we see it!) If you find a story that you find interesting and think that others might also enjoy it, too, send it in! We publish, on average, about 450 stories a month. If 1% of the community submitted a story or two each month, it would make a huge difference!
Post comments
You don't need to be a subject-matter expert to comment on a story! (Though we sure do appreciate when such people chime in!) Sometimes the best discussions come about simply because someone asked a question.
Perform Moderation
Moderation is like Olympic Scoring. Everything from a "-1" (not worth the electrons used to store it) to a "5" (one of the best on the site). Each registered user gets 5 mod points per day. Concentrate on promoting the good rather than hiding the bad... we want to make sure the most insightful, interesting, and informative comments are visible.
Help site development and operation
Something bugging you about site behavior? Have experience in running a web site? Know how to run an IRC server? Know your way around a Wiki? Can code Perl in your sleep? Have experience doing QA and/or test? Don't have this knowledge but would like to learn? Join our development and/or operations team.
Support the site
Your financial contributions are critical to our continued operation. Subscribe to SoylentNews or buy SoylentNews swag.
See something else where you'd like to help out? Let us know — the more the merrier! There's a lot of fun and camaraderie in our team... in large part it's why I continue to contribute to the site. Join in on the fun!

There are many rewards for contributing. Just to be a part of such a diverse and knowledgeable team is indescribable. I have learned so much from some amazingly helpful people. So join up as an editor, submit stories and comments, moderate, or help the site to keep running.

Lastly, spread the word. Share a link to the main page, to a particular story, or even to a single comment.


posted by cmn32480 on Friday December 23 2016, @02:38PM   Printer-friendly
from the we-appreciate-the-help! dept.

Hi Guys, Soylent's Editors do a lot behind the scenes to keep the community going. As a gift idea for them this year, please consider submitting lots of stories over the next two days to get the queue nice and full. Then they'll be able to schedule in their appearance on the home page ahead of time and take Christmas (or Hanukkah) off to spend time with their friends and families.

If you've never submitted a story before, here are guidelines for composing a story submission. You submit it here.

My own method is to find tech/science articles from SN's RSS-bot or a dozen other sources like the BBC or, grab the title, and a couple of paragraphs that communicate the gist. Often I'll add a quip, question, or note of my own, but that's up to your personal taste. It's easy and takes under 5 minutes per story.

Thanks for reading, and have a happy holiday!

[Ed Note: The week between Christmas and New Years is always slow for submissions and time is a precious commodity for all of us. The more subs in the queue, the further out we can get the story queue, and the more time we have to spend with our loved ones. Any help you can give would be appreciated!]

Original Submission