2018-07-01 00:00:00 ..
2018-07-15 21:33:37 UTC
2018-07-18 12:36:56 UTC
We always have a place for talented people, visit the Get Involved section on the wiki to see how you can make SoylentNews better.
Two of SoylentNews' staff submitted stories noting our three-year anniversary; one a site summary of where we are and a summary of what we've done, and the other a detailed presentation of the very early days and how SoylentNews got started.
Three years ago, today, SoylentNews announced its presence to the world. Much has happened along the way of our providing a place for a community to grow and to engage in discussion.
It started as a fork of five-year-old, open-sourced code which had suffered under benign neglect. Perl, Apache, MySQL, and other products had continued on. So we had to deal with dependencies on unsupported and back-level versions of code. A great deal of effort went into bringing the site up-to-date with current versions of that base. See below for mechanicjay's illuminating first-hand account of how that all got started.
Those of you who were with us then can attest to the fact that site outages were a regular occurrence. Bugs were found and eradicated. New bugs were made, and found as well. We invited the community to vote to name the site. We created documents of incorporation and had them dutifully filed. On July 4th, 2014 we received notice of officially becoming SoylentNews PBC. But I get ahead of myself.
Not content with just running a clone of the old code, the staff embarked on a large number of improvements to the site. Support for Unicode characters (via UTF-8) was an early improvement. Refinements to moderation took place — you could now moderate and comment in the same story. Moderation points were issued to every registered user every single day. An API was written and made available. We have our own Folding@Home team (currently ranked 314 of 226132 teams in the world) which contributes spare compute cycles to help find cures to maladies such as Huntington's Disease. (See the Main F@H site and our team page.) We sent out a call for new editors to help our beleaguered editing team which was approaching burnout; several of you answered the call and we are greatly enriched by their viewpoints and their questioning of the status quo.
And what have we wrought? Our own place on the world-wide web, supported and run entirely by the community. For the numerate in our midst, here are some statistics for the site. As of the time of this writing (20170217_002919 UTC), SoylentNews has:
But that's not all! Unwilling to rest on their laurels, our development team has been hard at work bringing improvements to the site — along with some bug fixes. If you want to play with the current, in-development, subject-to-change-without-notice version of the site, hop on over to our development server. Do be aware several specially-crafted stories were created and posted there so as to evoke certain test conditions, so please respect the admonitions stated on those stories. Have an observation, question, or found a bug? We'd love to hear your feedback in the #dev channel on our IRC server.
We could not have done it alone — a great many of you have contributed to the site. There is the administrative tasks of paying the bills and handling legal obligations. Sysops support to keep our boxes up and running. Writing code and patching bugs (while minimizing the bug writing). Suggesting and testing new code/features and providing constructive feedback. Making financial contributions by signing up for subscriptions. Submitting story submissions for the editors to poke and prod at. All of this in support of a goal to provide a place where people can submit comments and engage in discussions with other interesting and intelligent people on the 'net. As with any community, there have been some 'heated' discussions. And most refreshing of all, are those discussions where nuggets of wisdom and brilliance appear — and make the whole effort worthwhile.
So, on behalf of the rest of the all-volunteer staff here at SoylentNews, let me say thank you. For your support, engagement, and questioning — we are a better site because of you. May we continue to earn your trust and support for many years to come.
In the comments, please feel free to mention anything significant that happened over these years which were inadvertently omitted as well as to tell us what we can do better.
So, to wind this up, I have one last question: "emacs or vi?" =)
For our third year, I have some Reflections on our third day.
In some of the pre-history of SoylentNews, here is some of the stuff that gets lost in the mists of time around the first coordinated development effort -- running on a VM, on a laptop in my basement under the slashcott.org domain. The slashcott had been announced and was to commence in some number of days. A bunch of folks thought it would be an awesome idea to get an independent version of slash running in time for the slashcott -- what could go wrong?
3 years and ton of life changes for me, makes some of this a little fuzzy, but I'll do my best to put things together. I've relied heavily on my email archive of that time which helped spur a bunch of memories. Hopefully this will be a coherent tale. (Maybe for next year I'll mine my personal IRC logs from when we were still on freenode).
At first there was a bunch of coordination in the ##slashcode channel on freenode, a bunch of emails were also buzzing around trying to coordinate some things and ideas. My first email to Barrabas was on 02/06/2014 [6 Feb 2014 for our non-US readers]. The issue at hand was that "slashcode" had been hastily open sourced 5 years prior, then pretty well abandoned. Not only did you need to build the perl modules from scratch, but it would only build against Apache 1.x. Once you managed to run that gauntlet, even compiled and installed, things barely ran and were pretty horribly broken. Anyway, it soon became apparent that robinld, NCommander and myself were making the most progress on getting something running, as I recall Robin was the first to success in getting an installed running site, but his VM was stuck behind a corporate firewall.
In the meantime, I had gotten the domain slashcott.org registered while trying to build things myself. At some point, a bunch of us decided to combine forces, Robin shipped me his VM, I got it running on my laptop (as it was the only 64-bit thing I had at the time), we got myself and Ncommander ssh'ed in and we started hacking. For some reason, RedHat vm's were horribly laggy on my openSuse VirtualBox host and work was slow and painful, but progress started to be made.
The only bug I've ever fixed in the code base was a critical piece of the new account email/password generation stuff, as I recall the generated password wasn't actually getting written to the DB. (sadly the evidence of my contribution has been lost, I think I shipped the fix to either robin or ncommander, so they have credit in the git history). Regardless, it was a critical piece - I have an email dated 02/08/2014 with my new account/password, which worked -- it was a huge boon and let us start to let a couple people in to start hammering away to find front-end bugs (of which there were countless). The next big thing I see from mining my email is the first "Nightly stories email", which came out on 02/11/2014 (from the slashcott.org domain). I think we ended up with about 50ish users on slashcott.org (gosh I hope I still have that vmdk stashed somewhere).
On the night of 02/11/2014 (or very early morning of 02/12/2014), after giving up and going to bed (I had a new born and was teaching an undergrad class on the side in addition to my regular 9-5 -- I was beyond toasted after a week). The VM locked up hard (it had done this a couple times, but I was always available to poke it with a stick and bring it back. As I was unavailable and no one had exchanged important things like phone numbers yet, NCommander made the executive decision to spin up a linode, which was great. The laggy VM on the laptop wasn't meant to last forever, though I admit I had visions (delusions?) of hosting the site myself on some real hardware at some point. In retrospect, Linode has been an amazing way to run this site and absolutely the right decision.
I got my new account on the li694-22 domain, on the 02/12/2014, that new account email was for mechanicjay, UID 7 -- which is where I live on the site to this day. I kept the slashcott.org server in sync with code changes for a bit, and was a pretty handy testing platform, until the "official" dev box came online on 02/14/2014. At some point during this week, we had landed on the soylentnews.org domain and that's where we went live on 02/17/2014.
So there you have it, we went from a group of independent pissed off people with no organization and an abandoned broken codebase to launching an honest-to-goodness site in ELEVEN fucking days.
So, in previous posts, I've talked about the fact that SoylentNews currently is powered on Ubuntu 14.04 + a single CentOS 6 box. Right now, the sysops have been somewhat deadlocked on what we should do going forward for our underlying operating system, and I am hoping to get community advice. Right now, the "obvious" choice of what to do is simply do-release-upgrade to Ubuntu 16.04. We've done in-place upgrades before without major issue, and I'm relatively certain we could upgrade without breaking the world. However, from my personal experience, 16.04 introduces systemd support into the stack and is not easily removable. Furthermore, at least in my personal experience, working with journalctl and such has caused me considerable headaches which I detailed in a comment awhile ago.
Discounting systemd itself, I've also found that Ubuntu 16.04 seems less "polished", for want of a better word. I've found I've had to do considerably more fiddling and tweaking to get it to work as a server distro than I had to do with previous releases, as well as had weird issues with LDAP. The same was also true when I worked with recent versions with Debian. As such, there's been a general feeling with the sysops that it's time to go somewhere else.
Below the fold are basically the options as we see them, and I hope if the community can provide some interesting insight or guidance.
Right now, we have about three years before security updates for 14.04 stop, and we are absolutely forced to migrate or upgrade. However, we're already hitting pain due to outdated software; I managed to briefly hose the DNS setup over the weekend trying to deploy CAA records for SN due to our version of BIND being outdated. When TLS 1.3 gets standardized, we're going to have a similar problem with our frontend load balancers. As such, I want to get a plan in place for migration so we can start upgrading over the next year instead of panicking and having to do something at the last moment
As with any discussion for server operating system, knowing what our workloads and such is an important consideration. In short, this is what we use for SN, and the software we have to support
In addition, we use mandatory application controls (AppArmor) to limit the amount of stuff a given process can access for critical services to try and help harden security. We'd like to maintain support for this feature to whatever we migrate, either continuing with AppArmor, switching to SELinux, or using jails/zones if we switch operating systems entirely.
Right now, we've floated a few options, but we're willing to hear more.
The first choice is simply migrate over to a distribution where systemd is not present or completely optional. As of writing, Arch Linux, Gentoo, and Slackware are three such options. Our requirements for a Linux distribution is a good record of updates and security support as I don't wish to be upgrading the system once a week to a new release.
I'm aware of the Devuan project, and at first glance, it would seem like an obvious choice; Debian without systemd is the de-facto tagline. However, I've got concerns about the long-term suitability of the distribution, as well as an intentional choice to replace much of the time-tested Debian infrastructure such as the testing archive with a git-powered Jenkins instance in it's place. Another option would be slackware, but Slackware has made no indication that they won't adapt systemd, and is historically very weak with in-place upgrading and package management in general. Most of the other distributions on without-systemd.org are either LiveCDs, or are very small minority distros that I would be hesitant to bet the farm on with.
On the other side of the coin, and an option favored by at least some of the staff is to migrate to Gentoo or Arch, which are rolling-release. For those unaware, a rolling release distribution basically always has the latest version of everything. Security updates are handled simply by updating to the latest upstream package for the most part. I'm not a huge fan of this option, as we're dependent on self-built software, and it's not unheard of for "emerge world" to break things during upgrades due to feature changes and such. It would essentially require us to manually be checking release notes, and crossing our fingers every time we did a major upgrade. We could reduce some of this pain by simply migrating all our infrastructure to the form of ebuilds so that at least they would get rebuild as part of upgrading, but I'm very very hesitant about this option as a whole, especially for multiple machines.
Another way we could handle the problem is simply jump off the Linux ship entirely. From a personal perspective, I'm not exactly thrilled on the way Linux as a collective whole has gone for several years, and I see the situation only getting worse with time. As an additional benefit, switching off Linux gives us the possiblity of using real containers and ZFS, which would allow us to further isolate components of the stack, and give us the option to do rollbacks if ever necessary on a blocked upgrade; something that is difficult to impossible with most Linux distributions. As such, I've been favoring this option personally, though I'm not sold enough to make the jump. Two major options attract me of these two:
FreeBSD has been around a long time, and has both considerable developer support, and support for a lot of features we'd like such as ZFS, jails, and a sane upstream. FreeBSD is split into two components, the core stack which is what constitutes a release, and the ports collection which is add-on software. Both can be upgraded (somewhat) independently of each other, so we won't have as much pain with outdated server components. We'd also have the ability to easy create jails for things like rehash, MySQL, and such and easily isolate these components from each other in a way that's more iron-clad than AppArmor or SELinux.
illumos is descended from OpenSolaris, and forked after Oracle closed up the source code for Solaris 11. Development has continued on it (at a, granted, slower place). Being the originator of ZFS, it has class A support for it, as well as zones which are functionally equivalent to FreeBSD jails. illumos also has support for SMF, which is essentially advanced service management and tracking without all the baggage systemd creates and tendrils throughout the stack. Zones can also be branded to run Linux binaries to some extent so we can handle migrating the core system over by simply installing illumos, restoring a backup into a branded zone, and then piecemeal decommissioning of said zone. As such, as an upgrade choice, this is fairly attractive. If we migrate to illumos, we'll either use the SmartOS distribution, or OpenIndiana.
Right now, we're basically on the fence with all options, so hopefully the community can provide their own input, or suggest other options we're not aware of. I look forward to your comments below!
Earlier today, we ran an article detailing that Oracle released 270 critical security updates for many of its products, including MySQL cluster which we use here to provide high uptime and reliability for SoylentNews. Needless to say, it was time to upgrade both NDB backends, and the four MySQLd frontends. While the upgrade did not go completely smoothly due to the fact that MySQL strict mode got enabled, and broke the site briefly, our total downtime was less than five minutes or so. Right now, we had to do a full flush and purge of all caches, which means the site is running a bit larky until they can repopulate but I'm pleased to announce we're up to date and secure!
ndb_mgm> show Cluster Configuration --------------------- [ndbd(NDB)] 2 node(s) id=2 @redacted (mysql-5.7.17 ndb-7.5.5, Nodegroup: 0) id=3 @redacted (mysql-5.7.17 ndb-7.5.5, Nodegroup: 0, *) [ndb_mgmd(MGM)] 2 node(s) id=101 @redacted (mysql-5.7.17 ndb-7.5.5) id=102 @redacted (mysql-5.7.17 ndb-7.5.5) [mysqld(API)] 4 node(s) id=11 @redacted (mysql-5.7.17 ndb-7.5.5) id=12 @redacted (mysql-5.7.17 ndb-7.5.5) id=13 @redacted (mysql-5.7.17 ndb-7.5.5) id=14 @redacted (mysql-5.7.17 ndb-7.5.5)
If you notice any unusual breakages or slowdowns, please let me know in the comments. Otherwise, keep calm and carry on!
A couple months ago we ran a story asking the SoylentNews community for volunteers to help with editing and the community did not let us down; we received a full dozen inquiries! You've probably noticed a few new names art the top of the stories and quite frankly, their contributions made it possible for the staff to survive the holiday season — many, many thanks!
If, for whatever reason, you did not want to be an Editor, but still wish to contribute, there are many other areas:
There are many rewards for contributing. Just to be a part of such a diverse and knowledgeable team is indescribable. I have learned so much from some amazingly helpful people. So join up as an editor, submit stories and comments, moderate, or help the site to keep running.
Lastly, spread the word. Share a link to the main page, to a particular story, or even to a single comment.
Hi Guys, Soylent's Editors do a lot behind the scenes to keep the community going. As a gift idea for them this year, please consider submitting lots of stories over the next two days to get the queue nice and full. Then they'll be able to schedule in their appearance on the home page ahead of time and take Christmas (or Hanukkah) off to spend time with their friends and families.
My own method is to find tech/science articles from SN's RSS-bot or a dozen other sources like the BBC or sciencenews.org, grab the title, and a couple of paragraphs that communicate the gist. Often I'll add a quip, question, or note of my own, but that's up to your personal taste. It's easy and takes under 5 minutes per story.
Thanks for reading, and have a happy holiday!
[Ed Note: The week between Christmas and New Years is always slow for submissions and time is a precious commodity for all of us. The more subs in the queue, the further out we can get the story queue, and the more time we have to spend with our loved ones. Any help you can give would be appreciated!]
Hopefully you will have noticed a number of new editors that have appeared to help keep this site running. They have been active for over a week but you might not have noticed them if you have been enjoying the Thanksgiving Day holiday, or just spending money during Black Friday (which seems to last longer each year!)
Snow, Charon, FatPhil, Fnord666, and GreatOutdoors have completed their training and are busy making their contributions to the team, and there are several more volunteers who will begin training in the near future. I hope that you will welcome them and keep them busy by providing more and varied submissions for them to battle with. They have already significantly reduced the strain on the editorial team and we are all breathing a collective sigh of relief back here. Thank you for volunteering guys!