SoylentNews is people
SoylentNews
So, in previous posts, I've talked about the fact that SoylentNews currently is powered on Ubuntu 14.04 + a single CentOS 6 box. Right now, the sysops have been somewhat deadlocked on what we should do going forward for our underlying operating system, and I am hoping to get community advice. Right now, the "obvious" choice of what to do is simply do-release-upgrade to Ubuntu 16.04. We've done in-place upgrades before without major issue, and I'm relatively certain we could upgrade without breaking the world. However, from my personal experience, 16.04 introduces systemd support into the stack and is not easily removable. Furthermore, at least in my personal experience, working with journalctl and such has caused me considerable headaches which I detailed in a comment awhile ago.
Discounting systemd itself, I've also found that Ubuntu 16.04 seems less "polished", for want of a better word. I've found I've had to do considerably more fiddling and tweaking to get it to work as a server distro than I had to do with previous releases, as well as had weird issues with LDAP. The same was also true when I worked with recent versions with Debian. As such, there's been a general feeling with the sysops that it's time to go somewhere else.
Below the fold are basically the options as we see them, and I hope if the community can provide some interesting insight or guidance.
Right now, we have about three years before security updates for 14.04 stop, and we are absolutely forced to migrate or upgrade. However, we're already hitting pain due to outdated software; I managed to briefly hose the DNS setup over the weekend trying to deploy CAA records for SN due to our version of BIND being outdated. When TLS 1.3 gets standardized, we're going to have a similar problem with our frontend load balancers. As such, I want to get a plan in place for migration so we can start upgrading over the next year instead of panicking and having to do something at the last moment
The SN Software Stack
As with any discussion for server operating system, knowing what our workloads and such is an important consideration. In short, this is what we use for SN, and the software we have to support
- nginx - Loadbalancing/SSL Termination
- Apache 2.2 + mod_perl - rehash (we run it with a separate instance of Apache and Perl, and not the system copy)
- MySQL Cluster for production
- MySQL standard for secondary services
- Kerberos + Hesiod - single-signon/authetication
- Postfix+Squirrelmail - ... mail
In addition, we use mandatory application controls (AppArmor) to limit the amount of stuff a given process can access for critical services to try and help harden security. We'd like to maintain support for this feature to whatever we migrate, either continuing with AppArmor, switching to SELinux, or using jails/zones if we switch operating systems entirely.
The Options
Right now, we've floated a few options, but we're willing to hear more.
A non-systemd Linux distro
The first choice is simply migrate over to a distribution where systemd is not present or completely optional. As of writing, Arch Linux, Gentoo, and Slackware are three such options. Our requirements for a Linux distribution is a good record of updates and security support as I don't wish to be upgrading the system once a week to a new release.
Release-based distributions
I'm aware of the Devuan project, and at first glance, it would seem like an obvious choice; Debian without systemd is the de-facto tagline. However, I've got concerns about the long-term suitability of the distribution, as well as an intentional choice to replace much of the time-tested Debian infrastructure such as the testing archive with a git-powered Jenkins instance in it's place. Another option would be slackware, but Slackware has made no indication that they won't adapt systemd, and is historically very weak with in-place upgrading and package management in general. Most of the other distributions on without-systemd.org are either LiveCDs, or are very small minority distros that I would be hesitant to bet the farm on with.
Rolling-release distributions
On the other side of the coin, and an option favored by at least some of the staff is to migrate to Gentoo or Arch, which are rolling-release. For those unaware, a rolling release distribution basically always has the latest version of everything. Security updates are handled simply by updating to the latest upstream package for the most part. I'm not a huge fan of this option, as we're dependent on self-built software, and it's not unheard of for "emerge world" to break things during upgrades due to feature changes and such. It would essentially require us to manually be checking release notes, and crossing our fingers every time we did a major upgrade. We could reduce some of this pain by simply migrating all our infrastructure to the form of ebuilds so that at least they would get rebuild as part of upgrading, but I'm very very hesitant about this option as a whole, especially for multiple machines.
Switch to FreeBSD/illumos/Other
Another way we could handle the problem is simply jump off the Linux ship entirely. From a personal perspective, I'm not exactly thrilled on the way Linux as a collective whole has gone for several years, and I see the situation only getting worse with time. As an additional benefit, switching off Linux gives us the possiblity of using real containers and ZFS, which would allow us to further isolate components of the stack, and give us the option to do rollbacks if ever necessary on a blocked upgrade; something that is difficult to impossible with most Linux distributions. As such, I've been favoring this option personally, though I'm not sold enough to make the jump. Two major options attract me of these two:
FreeBSD
FreeBSD has been around a long time, and has both considerable developer support, and support for a lot of features we'd like such as ZFS, jails, and a sane upstream. FreeBSD is split into two components, the core stack which is what constitutes a release, and the ports collection which is add-on software. Both can be upgraded (somewhat) independently of each other, so we won't have as much pain with outdated server components. We'd also have the ability to easy create jails for things like rehash, MySQL, and such and easily isolate these components from each other in a way that's more iron-clad than AppArmor or SELinux.
illumos
illumos is descended from OpenSolaris, and forked after Oracle closed up the source code for Solaris 11. Development has continued on it (at a, granted, slower place). Being the originator of ZFS, it has class A support for it, as well as zones which are functionally equivalent to FreeBSD jails. illumos also has support for SMF, which is essentially advanced service management and tracking without all the baggage systemd creates and tendrils throughout the stack. Zones can also be branded to run Linux binaries to some extent so we can handle migrating the core system over by simply installing illumos, restoring a backup into a branded zone, and then piecemeal decommissioning of said zone. As such, as an upgrade choice, this is fairly attractive. If we migrate to illumos, we'll either use the SmartOS distribution, or OpenIndiana.
Final Notes
Right now, we're basically on the fence with all options, so hopefully the community can provide their own input, or suggest other options we're not aware of. I look forward to your comments below!
~ NCommander
(Score: 3, Interesting) by stack on Tuesday February 07 2017, @05:24PM
As an admin for a couple hundred systems set up for High Performance Computing + Webservers + Docker + storage (Ceph) + a handful of desktops who has been working with Linux since the 90's - here are my thoughts. We have had many conversations and debates internally on exactly your situation. We are a bit further out as we have actually begun implementing a lot of our changes. We are probably only about 10-15% through our migration right now, but we are migrating.
1) BSD has its place and is awesome. But I struggle to find good admins to hire that know Linux well enough to do what we need to do. Give them BSD and things are just different enough that it is a no-go unless I want to be on call 24x7 or spend a ton of time getting them up to speed. It is just little differences, but it seriously threw a wrench in our plans for migration when so many of our scripts broke on BSD and most of my team stalled out. I really don't want to re-write everything myself and catch them up to speed. Personally, I have enough experience that when those little changes bite me I just mutter a curse, fix it, and move on. I can't carry the full team and just the thought of what it would do to my large user base scares me.
2) We use a LOT of Red Hat (and Scientific Linux). We REALLY have had issues with systemd and weird issues surrounding it. We would prefer to stay on RH6, but it is incredibly annoying how fast companies have dropped support for it. For example, Rstudio no longer supports RH6 for literally no good reason. We are finding it harder and harder to support application updates on RH6 because other vendors are dropping it. Even things like Gitlab which support RH6 have dependencies that we have to use IUS or Software Collections for updates. Even Git (!) is so old that our user base has had issues and we have had to update/replace it internally. We had so many workarounds in place for our owncloud, that when we migrated to Nextcloud 11 we couldn't get it running right on RH6 and had to migrate it to a newer OS. While we have proper RH support and we utilize it for systemd bugs, they can be slow and it can be painful trying to get things replicated and fixed. Sometimes it is just "well that is a systemd problem upstream" to which my response is "Well then fix it. That is why I am paying for your support!". Don't get me wrong, they are usually pretty good, but we've had a few issues. We are /slowly/ migrating to 7. Honestly, the biggest thing about RH7 that causes cursing among myself and my team:
$ syscontrol $APP stop && syscontrol $APP start
admin: "!@#*&!@#!$%*$&@#!$^%^!$^!@*&$!!"
$ syscontrol stop $APP && syscontrol start $APP
Under initd of course it was "service $APP start". Not only is it that order ingrained into our memory and muscles, but it is a pain in the @$$ when you can't simply hit the up arrow and backspace the last argument to run a different command. Whoever did this in systemd is a first class @$$hole who must seriously hate admins...It is really bad right now since we are supporting a mixed environment of both initd and systemd systems. Maybe it won't be so bad when we fully transition.
3) We are using Lubuntu 16.04 on our desktops and Ubuntu 16.04 Server on several of our systems. I have found the Ubuntu team FAR more skilled and adept at getting systemd problems fixed. We still find hardware issues from time to time which is just freaking annoying. The Systemd crowd says "Oh it isn't systemd! It's your hardware!" to which we call BS. We have routinely proven that other non-systemd OS's use the hardware just fine and in many cases an different version of systemd works. Seriously, the biggest issue I have with systemd these days is that the developers all seem to be smug jerks who think they know more about my hardware setup then I do. I routinely prove them wrong. They also have the problem of saying brand new laptops are "broken" because systemd doesn't support them properly (laptop lid switch reporting on or off is a common problem that systemd deals with quite poorly). I have come to have a GREAT deal of respect for the Ubuntu systemd devs as they have always been helpful and have worked well with me to replicate and fix the problem.
4) The BIGGEST problem I have with Ubuntu 16.04? It freaking updates all the damn time! I'm in an environment where I have to have security patches applied regularly but I also can't reboot at will! I have users running jobs that may run months at a time! Yet Ubuntu kicks out kernel updates nearly weekly. My RH and 14.04 boxes are stable and get a few updates from time to time, but the 16.04 boxes are CONSTANTLY complaining about needing to reboot for patches. Knock it off, Canonical, and stabilize 16.04 already! Jeez.
It hasn't been easy, it is slow going, and we've reported TONS of bugs so far upstream...but we believe that moving forward with 16.04 and RH7/SL7 is the best way forward at the moment.
Hope that helps.
~Stack~
(Score: 0) by Anonymous Coward on Tuesday February 07 2017, @08:53PM
I'm convinced the frequent kernel updates on 16.04 (even when there aren't security issues from upstream) are to inconvenience enough so that people buy their live patching service.