So, in previous posts, I've talked about the fact that SoylentNews currently is powered on Ubuntu 14.04 + a single CentOS 6 box. Right now, the sysops have been somewhat deadlocked on what we should do going forward for our underlying operating system, and I am hoping to get community advice. Right now, the "obvious" choice of what to do is simply do-release-upgrade to Ubuntu 16.04. We've done in-place upgrades before without major issue, and I'm relatively certain we could upgrade without breaking the world. However, from my personal experience, 16.04 introduces systemd support into the stack and is not easily removable. Furthermore, at least in my personal experience, working with journalctl and such has caused me considerable headaches which I detailed in a comment awhile ago.
Discounting systemd itself, I've also found that Ubuntu 16.04 seems less "polished", for want of a better word. I've found I've had to do considerably more fiddling and tweaking to get it to work as a server distro than I had to do with previous releases, as well as had weird issues with LDAP. The same was also true when I worked with recent versions with Debian. As such, there's been a general feeling with the sysops that it's time to go somewhere else.
Below the fold are basically the options as we see them, and I hope if the community can provide some interesting insight or guidance.
Right now, we have about three years before security updates for 14.04 stop, and we are absolutely forced to migrate or upgrade. However, we're already hitting pain due to outdated software; I managed to briefly hose the DNS setup over the weekend trying to deploy CAA records for SN due to our version of BIND being outdated. When TLS 1.3 gets standardized, we're going to have a similar problem with our frontend load balancers. As such, I want to get a plan in place for migration so we can start upgrading over the next year instead of panicking and having to do something at the last moment
As with any discussion for server operating system, knowing what our workloads and such is an important consideration. In short, this is what we use for SN, and the software we have to support
In addition, we use mandatory application controls (AppArmor) to limit the amount of stuff a given process can access for critical services to try and help harden security. We'd like to maintain support for this feature to whatever we migrate, either continuing with AppArmor, switching to SELinux, or using jails/zones if we switch operating systems entirely.
Right now, we've floated a few options, but we're willing to hear more.
The first choice is simply migrate over to a distribution where systemd is not present or completely optional. As of writing, Arch Linux, Gentoo, and Slackware are three such options. Our requirements for a Linux distribution is a good record of updates and security support as I don't wish to be upgrading the system once a week to a new release.
I'm aware of the Devuan project, and at first glance, it would seem like an obvious choice; Debian without systemd is the de-facto tagline. However, I've got concerns about the long-term suitability of the distribution, as well as an intentional choice to replace much of the time-tested Debian infrastructure such as the testing archive with a git-powered Jenkins instance in it's place. Another option would be slackware, but Slackware has made no indication that they won't adapt systemd, and is historically very weak with in-place upgrading and package management in general. Most of the other distributions on without-systemd.org are either LiveCDs, or are very small minority distros that I would be hesitant to bet the farm on with.
On the other side of the coin, and an option favored by at least some of the staff is to migrate to Gentoo or Arch, which are rolling-release. For those unaware, a rolling release distribution basically always has the latest version of everything. Security updates are handled simply by updating to the latest upstream package for the most part. I'm not a huge fan of this option, as we're dependent on self-built software, and it's not unheard of for "emerge world" to break things during upgrades due to feature changes and such. It would essentially require us to manually be checking release notes, and crossing our fingers every time we did a major upgrade. We could reduce some of this pain by simply migrating all our infrastructure to the form of ebuilds so that at least they would get rebuild as part of upgrading, but I'm very very hesitant about this option as a whole, especially for multiple machines.
Another way we could handle the problem is simply jump off the Linux ship entirely. From a personal perspective, I'm not exactly thrilled on the way Linux as a collective whole has gone for several years, and I see the situation only getting worse with time. As an additional benefit, switching off Linux gives us the possiblity of using real containers and ZFS, which would allow us to further isolate components of the stack, and give us the option to do rollbacks if ever necessary on a blocked upgrade; something that is difficult to impossible with most Linux distributions. As such, I've been favoring this option personally, though I'm not sold enough to make the jump. Two major options attract me of these two:
FreeBSD has been around a long time, and has both considerable developer support, and support for a lot of features we'd like such as ZFS, jails, and a sane upstream. FreeBSD is split into two components, the core stack which is what constitutes a release, and the ports collection which is add-on software. Both can be upgraded (somewhat) independently of each other, so we won't have as much pain with outdated server components. We'd also have the ability to easy create jails for things like rehash, MySQL, and such and easily isolate these components from each other in a way that's more iron-clad than AppArmor or SELinux.
illumos is descended from OpenSolaris, and forked after Oracle closed up the source code for Solaris 11. Development has continued on it (at a, granted, slower place). Being the originator of ZFS, it has class A support for it, as well as zones which are functionally equivalent to FreeBSD jails. illumos also has support for SMF, which is essentially advanced service management and tracking without all the baggage systemd creates and tendrils throughout the stack. Zones can also be branded to run Linux binaries to some extent so we can handle migrating the core system over by simply installing illumos, restoring a backup into a branded zone, and then piecemeal decommissioning of said zone. As such, as an upgrade choice, this is fairly attractive. If we migrate to illumos, we'll either use the SmartOS distribution, or OpenIndiana.
Right now, we're basically on the fence with all options, so hopefully the community can provide their own input, or suggest other options we're not aware of. I look forward to your comments below!
~ NCommander
(Score: 3, Insightful) by MadTinfoilHatter on Tuesday February 07 2017, @12:30PM
Personally I've been using Gentoo on the desktop for a long time, and it's been remarkably stable, as long as you don't use the ~${arch} packages. The biggest issues tend to arise when there is a major update of some key graphical components, e.g. KDE, or Xorg. This would of course not be a problem on a server.
Using Debian without systemd (which was mentioned) has the problem of non-sysremd-users effectively being second class citizens with that distro, so I wouldn't recommend that. Also since systemd digs its tentacles so deeply into everything, you'll still need to deal with some systemd brain-damage, even if you don't let it act as the init system. Devuan would clearly be the better path if what one wants is Debian sans poetterix.
(Score: 2) by NCommander on Tuesday February 07 2017, @12:44PM
My biggest problem with Gentoo is you occasionally need to emerge world when you have an ABI bump in portage at a low level; maybe once every few months. That can take hours or days to finish depending on number of ports installed. Multiply that by a bunch of machines and life gets painful since each will likely update to a slightly different version of Gentoo unless I setup a portage mirror internally and use that to keep everything lock-step.
As far as rolling releases go though, its probably the top choice though; I prefer it much more to Arch.
Still always moving
(Score: 2) by tynin on Tuesday February 07 2017, @03:49PM
It becomes less of a hassle if you setup a PXE booted environment and then make a cloned base OS the latest Gentoo emerge. Then it is just a matter of pointing your PXE server to boot the new OS and reboot the servers into it. It'll mean maintaining a single OS image, and all of the servers are a reboot away from being a "clean" and updated install again.
(Score: 2) by NCommander on Tuesday February 07 2017, @04:13PM
Because almost all our machines run different configurations and software loads, we don't do ansible/puppet/StackScript deployments. The one exception are the frontend boxes. In the entirety SN has been up, we've only ever done a single nuke and pave, and that was when the original web-frontend (hydrogen) developed weird issues and was blown away and copied fluorine.
Still always moving
(Score: 2) by tynin on Thursday February 09 2017, @07:59PM
Thanks for the explanation!
(Score: 0) by Anonymous Coward on Wednesday February 08 2017, @04:52AM
I just got through rebuilding a system with it and libc++abi+libunwind as the abi backend. Works fine, and apparently according to the libcxx website, it is also possible to compile against it with gcc (no specific version mentioned, although it may be 4.8+).
Requires setting up environment flags for clang, but outside some cornercases seems to work just fine (notable failures were openscenegraph 3.5.x due to addition of templates which broken on libc++, and some weird build issues over memory types in recent firefox versions.) Outside of that it seems to compile slightly faster than libstdc++, hasn't show any of the annoying cross-version compiler errors that tend to crop up with libstdc++, and with a bit of work could be combined with the hardened musl profile on gentoo to provide a significantly cleaner and less finicky userspace, assuming your build dependencies compile cleanly against it (the number of assumptions in the average linux focused software project nowadays is obscene.)
(Score: 2) by Techwolf on Wednesday February 08 2017, @10:37PM
I have been using Gentoo for many years. One thing I have notice is portage has vastelly improve over the past years. Its rare to to b0rk a system with emerge -avuD world. One reason is a new portage feature that presearve libs when upgrading libs. If a package is linked to the old lib, portage will marked it presearve and will not delete untill the all packages that dedpends on it is rebuilt against the new lib.
I use a chroot to build the packages then emerge -k on main. This allows me to emerge -e world while still useing the desktop. Another advange is when a package fails to build, still have desktop to browse the gentoo forums and bugs to see if someone else posted a fix/workaround.
To speed things up, I bought my first SSD and mounted the chroot directory on it. I did use btfs on it to test it out. After a power failure, UPS died, I started to get emerge falures that did not happen on main system. Digging into some btfs tools, discovered the file system had a problem. Tried to fix only to become un-mountiable. Had to nuke start over using ext4 and backup. So btfs is a failure and I will stick to ext4.
My recominadation is gentoo is you have spare server/VM to build on and emerge -k to production. Otherwise, FreeBSD all the way. :-)
(Score: 2) by The Mighty Buzzard on Tuesday February 07 2017, @12:45PM
Me, I'm currently digging on Calculate. Essentially Gentoo with a well stocked binary overlay. Allows me to tweak the packages I want tweaked for speed with USE/make/env flags and get binaries of the ones I fail to give a shit about. Plus setup is way, way easier.
My rights don't end where your fear begins.
(Score: 2) by NCommander on Tuesday February 07 2017, @12:56PM
If we do jump onto the Gentoo ship, we're probably going to setup a build master on neon since the only thing that box is doing is running a backup DB node. That way, we can emerge world once, then get all those binaries published with our USE flags for everything else downstream to grab it.
Alternatively, we could use oxygen, as it has a lot of spare disk space ...
Still always moving
(Score: 0) by Anonymous Coward on Thursday February 09 2017, @11:59PM
(Score: 2) by The Mighty Buzzard on Saturday February 11 2017, @01:35AM
Actually, oxygen is dead full to the point that we've been making partial and nonexistent backups for who knows how long. I freed up a bunch of space getting rid of "temporary" full db backups on several boxes that'd been being backed up as part of our usual backup routine. We'll see how much is left tomorrow after the scheduled backup.
My rights don't end where your fear begins.
(Score: 0) by Anonymous Coward on Tuesday February 07 2017, @02:05PM
I've come to love Gentoo after jumping away from Debian.
Ignore the naysayers telling you that you don't get any performance benefit from recompiling from source, because that's not the point.
The USE flags really makes Gentoo stand out from traditional binary distros. You can use it to cut out all the unnecessary features.
Hate systemd? Use -systemd
Hate pulseaudio? Use -pulseaudio
Want to disable only certain GTK3* features? x11-libs/gtk+:3 -introspection
It gets complicated, but it is pretty well documented in portage(5). Check out equery(1) too.
(Score: 0) by Anonymous Coward on Tuesday February 07 2017, @05:36PM
(Score: 0) by Anonymous Coward on Wednesday February 08 2017, @04:57AM
I don't remember exactly what it is supposed to do, but it provides standardized ways to poke at certain glib/gtk/etc objects in a consistent manner.
That said, it has been a huge PITA for me in the past since some packages only worked with one particular version of glib or introspection or what have you thanks to the gimp/gnome projects not bothering to fucking change the version number and allow slotted includes/libraries when forcefully deprecating features actual software uses.