So, in previous posts, I've talked about the fact that SoylentNews currently is powered on Ubuntu 14.04 + a single CentOS 6 box. Right now, the sysops have been somewhat deadlocked on what we should do going forward for our underlying operating system, and I am hoping to get community advice. Right now, the "obvious" choice of what to do is simply do-release-upgrade to Ubuntu 16.04. We've done in-place upgrades before without major issue, and I'm relatively certain we could upgrade without breaking the world. However, from my personal experience, 16.04 introduces systemd support into the stack and is not easily removable. Furthermore, at least in my personal experience, working with journalctl and such has caused me considerable headaches which I detailed in a comment awhile ago.
Discounting systemd itself, I've also found that Ubuntu 16.04 seems less "polished", for want of a better word. I've found I've had to do considerably more fiddling and tweaking to get it to work as a server distro than I had to do with previous releases, as well as had weird issues with LDAP. The same was also true when I worked with recent versions with Debian. As such, there's been a general feeling with the sysops that it's time to go somewhere else.
Below the fold are basically the options as we see them, and I hope if the community can provide some interesting insight or guidance.
Right now, we have about three years before security updates for 14.04 stop, and we are absolutely forced to migrate or upgrade. However, we're already hitting pain due to outdated software; I managed to briefly hose the DNS setup over the weekend trying to deploy CAA records for SN due to our version of BIND being outdated. When TLS 1.3 gets standardized, we're going to have a similar problem with our frontend load balancers. As such, I want to get a plan in place for migration so we can start upgrading over the next year instead of panicking and having to do something at the last moment
As with any discussion for server operating system, knowing what our workloads and such is an important consideration. In short, this is what we use for SN, and the software we have to support
In addition, we use mandatory application controls (AppArmor) to limit the amount of stuff a given process can access for critical services to try and help harden security. We'd like to maintain support for this feature to whatever we migrate, either continuing with AppArmor, switching to SELinux, or using jails/zones if we switch operating systems entirely.
Right now, we've floated a few options, but we're willing to hear more.
The first choice is simply migrate over to a distribution where systemd is not present or completely optional. As of writing, Arch Linux, Gentoo, and Slackware are three such options. Our requirements for a Linux distribution is a good record of updates and security support as I don't wish to be upgrading the system once a week to a new release.
I'm aware of the Devuan project, and at first glance, it would seem like an obvious choice; Debian without systemd is the de-facto tagline. However, I've got concerns about the long-term suitability of the distribution, as well as an intentional choice to replace much of the time-tested Debian infrastructure such as the testing archive with a git-powered Jenkins instance in it's place. Another option would be slackware, but Slackware has made no indication that they won't adapt systemd, and is historically very weak with in-place upgrading and package management in general. Most of the other distributions on without-systemd.org are either LiveCDs, or are very small minority distros that I would be hesitant to bet the farm on with.
On the other side of the coin, and an option favored by at least some of the staff is to migrate to Gentoo or Arch, which are rolling-release. For those unaware, a rolling release distribution basically always has the latest version of everything. Security updates are handled simply by updating to the latest upstream package for the most part. I'm not a huge fan of this option, as we're dependent on self-built software, and it's not unheard of for "emerge world" to break things during upgrades due to feature changes and such. It would essentially require us to manually be checking release notes, and crossing our fingers every time we did a major upgrade. We could reduce some of this pain by simply migrating all our infrastructure to the form of ebuilds so that at least they would get rebuild as part of upgrading, but I'm very very hesitant about this option as a whole, especially for multiple machines.
Another way we could handle the problem is simply jump off the Linux ship entirely. From a personal perspective, I'm not exactly thrilled on the way Linux as a collective whole has gone for several years, and I see the situation only getting worse with time. As an additional benefit, switching off Linux gives us the possiblity of using real containers and ZFS, which would allow us to further isolate components of the stack, and give us the option to do rollbacks if ever necessary on a blocked upgrade; something that is difficult to impossible with most Linux distributions. As such, I've been favoring this option personally, though I'm not sold enough to make the jump. Two major options attract me of these two:
FreeBSD has been around a long time, and has both considerable developer support, and support for a lot of features we'd like such as ZFS, jails, and a sane upstream. FreeBSD is split into two components, the core stack which is what constitutes a release, and the ports collection which is add-on software. Both can be upgraded (somewhat) independently of each other, so we won't have as much pain with outdated server components. We'd also have the ability to easy create jails for things like rehash, MySQL, and such and easily isolate these components from each other in a way that's more iron-clad than AppArmor or SELinux.
illumos is descended from OpenSolaris, and forked after Oracle closed up the source code for Solaris 11. Development has continued on it (at a, granted, slower place). Being the originator of ZFS, it has class A support for it, as well as zones which are functionally equivalent to FreeBSD jails. illumos also has support for SMF, which is essentially advanced service management and tracking without all the baggage systemd creates and tendrils throughout the stack. Zones can also be branded to run Linux binaries to some extent so we can handle migrating the core system over by simply installing illumos, restoring a backup into a branded zone, and then piecemeal decommissioning of said zone. As such, as an upgrade choice, this is fairly attractive. If we migrate to illumos, we'll either use the SmartOS distribution, or OpenIndiana.
Right now, we're basically on the fence with all options, so hopefully the community can provide their own input, or suggest other options we're not aware of. I look forward to your comments below!
~ NCommander
(Score: 1) by krait6 on Friday February 10 2017, @08:35AM
I don't blame anyone for wanting to avoid systemd; I deal with systems both with and withtout it and there are benefits and drawbacks to each. My goal in this reply is to respond to the OP's query to discuss distro choices without systemd.
A starting recommendation is "try before you commit"; you could load various free distros in VMs to try 'em out to evaluate which ones will fit your needs. I once loaded the top 25 free software distros listed at Distrowatch.com into VM's to run (simple) tests on some software and completed all that in 3 days. [Except for Gentoo -- that took days to compile the GUI within the VM.]
Debian: defaults to systemd since Jessie, but systemd can be removed and sysvinit put in it's place and kept that way with an APT PIN:
https://www.debian.org/releases/jessie/amd64/release-notes/ch-information.en.html#systemd-upgrade-default-init-system [debian.org]
http://without-systemd.org/wiki/index.php/How_to_remove_systemd_from_a_Debian_jessie/sid_installation [without-systemd.org]
http://people.skolelinux.org/pere/blog/How_to_stay_with_sysvinit_in_Debian_Jessie.html [skolelinux.org]
Debian supports running it without systemd for the time being, but there's no guarantee that this is sustainable, because several upstream projects expect systemd to be on the target (KDE, Gnome, Network Manager, etc). For more info on those details, see: https://bugs.debian.org/727708 [debian.org]
The main benefit of Debian is that it supports upgrade-in-place. Few distros support this as well as Debian does.
Debian boxes can be upgraded over Tor after installing apt-transport-tor. All packages require GPG signatures from developers as well as package checksums which get checked as the package is installed.
Distribution releases have 3-year support + 2-year "LTS" support (from another team) after that.
Debian supoprts AppArmor and I use it on some Debian systems, and I find I definitely like AppArmor over SELinux for a lot of reasons.
Debian now has support for ZFS as an add-on driver: (zfs-dkms): https://tracker.debian.org/pkg/zfs-linux [debian.org]
This means ZFS is usuable in Debian but (AFAIK) isn't supoprted in the installer, but can be used otherwse. There are a couple of Debian Developers using ZFS as the root filesystem who have written about it:
http://changelog.complete.org/archives/9152-why-and-how-to-run-zfs-on-linux [complete.org]
Disclaimer: I'm involved in Debian development and Debian-based distros are primarily what I run.
Devuan: I haven't tried it but I like the idea; I looked at the development model of depending on Git repos and wasn't thrilled with that. Others in the comments seem to like running it and being that it's based on Debian it's at least worth investigating; hopefully it has apt-transport-tor, GPG signatures from developers for uploads, checkums on packages and checked on installation, etc. I haven't investigated how long releases are supported for or how.
CentOS: 10-year release support is IMHO its distinguishing feature. It is possible to upgrade-in-place for servers but the procedure does not look pleasant and a "fresh install" is recommended over using it:
https://www.namhuy.net/3253/upgrade-centos-6-7.html [namhuy.net]
I run a CentOS 6 system so I know what that's like (works but not pleasant due to aging packages, uses Upstart which is dead upstream and has some issues), and of course CentOS 7 uses systemd. *shrug*.
Arch Linux: IMHO rolling-based distros are fine for desktops, but I wouldn't want to run a rolling-based distro on a server, and friends of mine running Arch and Gentoo generally say the same.
Arch in particular has a lesser-known mechanism available to downgrade to a prior date, and this can be limited to a specified set of packages to be held back. This comes up often for a friend of mine dealing with incompatibilities between X and the prorpietary Nvidia driver. For servers I really want the ability to downgrade a particular package to a particular version, and wih Arch that can be done but not as easily as I'd want.
I (sometimes) run Arch myself, and I love the distro in a lot of ways. Arch + the Arch AUR has even more packages than Debian Unstable does, and that's saying a lot.
Gentoo: everyone I knew in the Linux User Groups that used to run Gentoo (including a few prior Gentoo developers) have moved on and are running something else. I briefly ran it on a box back when the install started with a "stage1" compile from a Live environment, I've tried it several times since then when testing things, and I have not been happy with the results. Basically "your milage will vary" -- there are Gentoo experts that can keep Gentoo systems running well, but for others it seems easy to get into trouble.
Slackware: I started Slackware back in the 90's. Back then the recommended upgrade method was "wipe and reinstall" and that's why I was compelled to switch away from it. Today there are some mecanisms to upgrade Slackware machines, and if this distro is chosen I would recommend setting up a test system to understand the upgrade mechanism first. I don't know what security features (GPG sigs or checksums) Slackware might have, if any -- last I knew Slackware was all based on simple tarballs without any of that. Last I recall Slackware was a minimalistic distro, so upgrading certain packages are likely to involve compiling from source.
FreeBSD: I've run it occasionally but need (a lot) more experience with it before I think I'd have an informed opinion of it.
Good to ask this question now so that there's plenty of time for evaluation of the options.