Stories
Slash Boxes
Comments

SoylentNews is people

Meta
posted by NCommander on Tuesday February 07 2017, @11:45AM   Printer-friendly
from the insert-systemd-rant-here dept.

So, in previous posts, I've talked about the fact that SoylentNews currently is powered on Ubuntu 14.04 + a single CentOS 6 box. Right now, the sysops have been somewhat deadlocked on what we should do going forward for our underlying operating system, and I am hoping to get community advice. Right now, the "obvious" choice of what to do is simply do-release-upgrade to Ubuntu 16.04. We've done in-place upgrades before without major issue, and I'm relatively certain we could upgrade without breaking the world. However, from my personal experience, 16.04 introduces systemd support into the stack and is not easily removable. Furthermore, at least in my personal experience, working with journalctl and such has caused me considerable headaches which I detailed in a comment awhile ago.

Discounting systemd itself, I've also found that Ubuntu 16.04 seems less "polished", for want of a better word. I've found I've had to do considerably more fiddling and tweaking to get it to work as a server distro than I had to do with previous releases, as well as had weird issues with LDAP. The same was also true when I worked with recent versions with Debian. As such, there's been a general feeling with the sysops that it's time to go somewhere else.

Below the fold are basically the options as we see them, and I hope if the community can provide some interesting insight or guidance.

Right now, we have about three years before security updates for 14.04 stop, and we are absolutely forced to migrate or upgrade. However, we're already hitting pain due to outdated software; I managed to briefly hose the DNS setup over the weekend trying to deploy CAA records for SN due to our version of BIND being outdated. When TLS 1.3 gets standardized, we're going to have a similar problem with our frontend load balancers. As such, I want to get a plan in place for migration so we can start upgrading over the next year instead of panicking and having to do something at the last moment

The SN Software Stack

As with any discussion for server operating system, knowing what our workloads and such is an important consideration. In short, this is what we use for SN, and the software we have to support

  • nginx - Loadbalancing/SSL Termination
  • Apache 2.2 + mod_perl - rehash (we run it with a separate instance of Apache and Perl, and not the system copy)
  • MySQL Cluster for production
  • MySQL standard for secondary services
  • Kerberos + Hesiod - single-signon/authetication
  • Postfix+Squirrelmail - ... mail

In addition, we use mandatory application controls (AppArmor) to limit the amount of stuff a given process can access for critical services to try and help harden security. We'd like to maintain support for this feature to whatever we migrate, either continuing with AppArmor, switching to SELinux, or using jails/zones if we switch operating systems entirely.

The Options

Right now, we've floated a few options, but we're willing to hear more.

A non-systemd Linux distro

The first choice is simply migrate over to a distribution where systemd is not present or completely optional. As of writing, Arch Linux, Gentoo, and Slackware are three such options. Our requirements for a Linux distribution is a good record of updates and security support as I don't wish to be upgrading the system once a week to a new release.

Release-based distributions

I'm aware of the Devuan project, and at first glance, it would seem like an obvious choice; Debian without systemd is the de-facto tagline. However, I've got concerns about the long-term suitability of the distribution, as well as an intentional choice to replace much of the time-tested Debian infrastructure such as the testing archive with a git-powered Jenkins instance in it's place. Another option would be slackware, but Slackware has made no indication that they won't adapt systemd, and is historically very weak with in-place upgrading and package management in general. Most of the other distributions on without-systemd.org are either LiveCDs, or are very small minority distros that I would be hesitant to bet the farm on with.

Rolling-release distributions

On the other side of the coin, and an option favored by at least some of the staff is to migrate to Gentoo or Arch, which are rolling-release. For those unaware, a rolling release distribution basically always has the latest version of everything. Security updates are handled simply by updating to the latest upstream package for the most part. I'm not a huge fan of this option, as we're dependent on self-built software, and it's not unheard of for "emerge world" to break things during upgrades due to feature changes and such. It would essentially require us to manually be checking release notes, and crossing our fingers every time we did a major upgrade. We could reduce some of this pain by simply migrating all our infrastructure to the form of ebuilds so that at least they would get rebuild as part of upgrading, but I'm very very hesitant about this option as a whole, especially for multiple machines.

Switch to FreeBSD/illumos/Other

Another way we could handle the problem is simply jump off the Linux ship entirely. From a personal perspective, I'm not exactly thrilled on the way Linux as a collective whole has gone for several years, and I see the situation only getting worse with time. As an additional benefit, switching off Linux gives us the possiblity of using real containers and ZFS, which would allow us to further isolate components of the stack, and give us the option to do rollbacks if ever necessary on a blocked upgrade; something that is difficult to impossible with most Linux distributions. As such, I've been favoring this option personally, though I'm not sold enough to make the jump. Two major options attract me of these two:

FreeBSD

FreeBSD has been around a long time, and has both considerable developer support, and support for a lot of features we'd like such as ZFS, jails, and a sane upstream. FreeBSD is split into two components, the core stack which is what constitutes a release, and the ports collection which is add-on software. Both can be upgraded (somewhat) independently of each other, so we won't have as much pain with outdated server components. We'd also have the ability to easy create jails for things like rehash, MySQL, and such and easily isolate these components from each other in a way that's more iron-clad than AppArmor or SELinux.

illumos

illumos is descended from OpenSolaris, and forked after Oracle closed up the source code for Solaris 11. Development has continued on it (at a, granted, slower place). Being the originator of ZFS, it has class A support for it, as well as zones which are functionally equivalent to FreeBSD jails. illumos also has support for SMF, which is essentially advanced service management and tracking without all the baggage systemd creates and tendrils throughout the stack. Zones can also be branded to run Linux binaries to some extent so we can handle migrating the core system over by simply installing illumos, restoring a backup into a branded zone, and then piecemeal decommissioning of said zone. As such, as an upgrade choice, this is fairly attractive. If we migrate to illumos, we'll either use the SmartOS distribution, or OpenIndiana.

Final Notes

Right now, we're basically on the fence with all options, so hopefully the community can provide their own input, or suggest other options we're not aware of. I look forward to your comments below!

~ NCommander

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 0) by Anonymous Coward on Tuesday February 07 2017, @08:02PM

    by Anonymous Coward on Tuesday February 07 2017, @08:02PM (#464237)

    yes, really.

    systemd: i haven't experienced any/most of the issues you mentioned in your previous comment about systemd. journald corruption? that's either old ass versions or distro/hardware/config specific, i would guess. every once in a while i run into something where i don't know how to get the info i want but i'm sure it's just my ignorance as it has been in the past. i'm not discounting others worries about the unix philosophy, etc. but all that's above my pay grade. for my use case(different types of servers, firewalls, desktops, etc) it's easy and nice 98% of the time. for the 2% output to syslog and use the tools/methods you're used to. read the arch wiki again.

    systemd2: or not. whatever works.

    arch: almost completely problem free. a somewhat competent sysadmin that pays attention to terminal output will have minimal issues. everything i manage is arch (for a few years now) and i'm probably much less experienced overall than soylent staff. gentoo is probably cool too but my *guess* (no real experience with gentoo) is that you will have more problems with their ebuilds than you will with arch packages. People think of arch as bleeding edge but it's really just current stable upstream versions with minimal changes. not a big deal. Distros that bastardize the hell out of everything are way more issue prone, IMHO.
    use linux-grsec kernel. arch is supposed to be adding PIE but it's currently stalled. I think we need to get organized and pay some of the devs...have staging servers or lxc containers if you're worried about something breaking.

    bsd? i have nothing against bsd (except i don't love the license) but i think you're kidding yourself and/or taking the vast power of the GNU+linox ecosystem for granted. bsd is fine if it meets your requirements and you want to jack with it, but chances are it's missing a few somethings you will need. you just may not find out until you're in too deep for comfort.

    good luck.

  • (Score: 2) by bzipitidoo on Tuesday February 07 2017, @09:28PM

    by bzipitidoo (4388) on Tuesday February 07 2017, @09:28PM (#464299) Journal

    When did Arch make systemd optional? I quit Arch some years ago when they tried to push systemd through their rolling update process, and talked like systemd was going to become a requirement for the rest of eternity. The update to systemd was long and complicated, and somewhere towards the end it went wrong, leaving me with an unbootable mess of an installation.

    One problem I ran into with systemd was that the default setting for the system logs was compressed. I forget which distro that was, think it was OpenSuse. To view the most recent lines, couldn't do "tail /var/log/messages" anymore and get results instantly, had to wait half a minute for journalctl to decompress the current log. Was really annoying when all I wanted was to view the last few lines over and over to check on whether this or that fix had resolved some issue the system was having.

    • (Score: 1) by TheSage on Wednesday February 08 2017, @06:03AM

      by TheSage (133) on Wednesday February 08 2017, @06:03AM (#464458) Journal

      https://sourceforge.net/projects/archopenrc/files/arch-openrc/ [sourceforge.net]
      works for me. For more details see
      http://systemd-free.org/ [systemd-free.org]

    • (Score: 0) by Anonymous Coward on Wednesday February 08 2017, @03:33PM

      by Anonymous Coward on Wednesday February 08 2017, @03:33PM (#464564)

      i'm not saying journalctl is perfect, but the reason journalctl was taking so long is because you were not restricting output to current boot. that means you were opening all of the journal since the beginning of time. especially large if you didn't set up a low size limit for the journal. if you specify the current boot "journalctl -b" and then hit "end" you'll be at the last few lines reasonably quickly. all in the arch wiki. maybe not when you were having issues though. i also didn't move to systemd until it had been in use in arch for a little while, since i knew nothing about it except that it was an important part of the system that i didn't want to have to try and fix.

      • (Score: 2) by bzipitidoo on Wednesday February 08 2017, @07:09PM

        by bzipitidoo (4388) on Wednesday February 08 2017, @07:09PM (#464686) Journal

        Thanks for the tip. And yet, -b seems a hackish workaround. I vaguely recall journalctl has another flag to get the most recent messages. I did not know of the Arch wiki at the time I was wrestling with systemd. I do remember that the maintainers of the distro had not troubled to document things. I had to find out the hard way where the system logs had been moved and how to access them. Read a lot of man pages and did a lot of Internet searches to learn that the system had been changed to systemd, that systemd was an init replacement, then learn of the existence of journalctl and that it was the new way to read the logs. A simple document explaining these basic facts and listing common Sys V commands with systemd equivalents would have saved me a lot of time.

        One thing that MS doesn't get is that gratuitously changing the interface is bad. Thought the Linux world was a bit more mindful of that, until this systemd fiasco. What could possibly justify breaking "tail /var/log/syslog" and "tail /var/log/messages" as ways to view the logs? They could have at least put in a short text file in their place, just have /var/log/messages contain something like the text "journalctl is the new command to view the logs. See the man page for details." then I would have seen at least that when I tried to view the logs the old way, instead of being left to wonder why the log files were missing and what was wrong with the system. But no, they didn't even do that. The move to systemd was badly done. Now, /var/log/syslog has come back in Ubuntu at least, if it was ever removed. Even the ipv4 to ipv6 move didn't uproot the old ways, you can still type "ifconfig", don't have to use the newer "ip addr".