SoylentNews is people
SoylentNews
So, in previous posts, I've talked about the fact that SoylentNews currently is powered on Ubuntu 14.04 + a single CentOS 6 box. Right now, the sysops have been somewhat deadlocked on what we should do going forward for our underlying operating system, and I am hoping to get community advice. Right now, the "obvious" choice of what to do is simply do-release-upgrade to Ubuntu 16.04. We've done in-place upgrades before without major issue, and I'm relatively certain we could upgrade without breaking the world. However, from my personal experience, 16.04 introduces systemd support into the stack and is not easily removable. Furthermore, at least in my personal experience, working with journalctl and such has caused me considerable headaches which I detailed in a comment awhile ago.
Discounting systemd itself, I've also found that Ubuntu 16.04 seems less "polished", for want of a better word. I've found I've had to do considerably more fiddling and tweaking to get it to work as a server distro than I had to do with previous releases, as well as had weird issues with LDAP. The same was also true when I worked with recent versions with Debian. As such, there's been a general feeling with the sysops that it's time to go somewhere else.
Below the fold are basically the options as we see them, and I hope if the community can provide some interesting insight or guidance.
Right now, we have about three years before security updates for 14.04 stop, and we are absolutely forced to migrate or upgrade. However, we're already hitting pain due to outdated software; I managed to briefly hose the DNS setup over the weekend trying to deploy CAA records for SN due to our version of BIND being outdated. When TLS 1.3 gets standardized, we're going to have a similar problem with our frontend load balancers. As such, I want to get a plan in place for migration so we can start upgrading over the next year instead of panicking and having to do something at the last moment
The SN Software Stack
As with any discussion for server operating system, knowing what our workloads and such is an important consideration. In short, this is what we use for SN, and the software we have to support
- nginx - Loadbalancing/SSL Termination
- Apache 2.2 + mod_perl - rehash (we run it with a separate instance of Apache and Perl, and not the system copy)
- MySQL Cluster for production
- MySQL standard for secondary services
- Kerberos + Hesiod - single-signon/authetication
- Postfix+Squirrelmail - ... mail
In addition, we use mandatory application controls (AppArmor) to limit the amount of stuff a given process can access for critical services to try and help harden security. We'd like to maintain support for this feature to whatever we migrate, either continuing with AppArmor, switching to SELinux, or using jails/zones if we switch operating systems entirely.
The Options
Right now, we've floated a few options, but we're willing to hear more.
A non-systemd Linux distro
The first choice is simply migrate over to a distribution where systemd is not present or completely optional. As of writing, Arch Linux, Gentoo, and Slackware are three such options. Our requirements for a Linux distribution is a good record of updates and security support as I don't wish to be upgrading the system once a week to a new release.
Release-based distributions
I'm aware of the Devuan project, and at first glance, it would seem like an obvious choice; Debian without systemd is the de-facto tagline. However, I've got concerns about the long-term suitability of the distribution, as well as an intentional choice to replace much of the time-tested Debian infrastructure such as the testing archive with a git-powered Jenkins instance in it's place. Another option would be slackware, but Slackware has made no indication that they won't adapt systemd, and is historically very weak with in-place upgrading and package management in general. Most of the other distributions on without-systemd.org are either LiveCDs, or are very small minority distros that I would be hesitant to bet the farm on with.
Rolling-release distributions
On the other side of the coin, and an option favored by at least some of the staff is to migrate to Gentoo or Arch, which are rolling-release. For those unaware, a rolling release distribution basically always has the latest version of everything. Security updates are handled simply by updating to the latest upstream package for the most part. I'm not a huge fan of this option, as we're dependent on self-built software, and it's not unheard of for "emerge world" to break things during upgrades due to feature changes and such. It would essentially require us to manually be checking release notes, and crossing our fingers every time we did a major upgrade. We could reduce some of this pain by simply migrating all our infrastructure to the form of ebuilds so that at least they would get rebuild as part of upgrading, but I'm very very hesitant about this option as a whole, especially for multiple machines.
Switch to FreeBSD/illumos/Other
Another way we could handle the problem is simply jump off the Linux ship entirely. From a personal perspective, I'm not exactly thrilled on the way Linux as a collective whole has gone for several years, and I see the situation only getting worse with time. As an additional benefit, switching off Linux gives us the possiblity of using real containers and ZFS, which would allow us to further isolate components of the stack, and give us the option to do rollbacks if ever necessary on a blocked upgrade; something that is difficult to impossible with most Linux distributions. As such, I've been favoring this option personally, though I'm not sold enough to make the jump. Two major options attract me of these two:
FreeBSD
FreeBSD has been around a long time, and has both considerable developer support, and support for a lot of features we'd like such as ZFS, jails, and a sane upstream. FreeBSD is split into two components, the core stack which is what constitutes a release, and the ports collection which is add-on software. Both can be upgraded (somewhat) independently of each other, so we won't have as much pain with outdated server components. We'd also have the ability to easy create jails for things like rehash, MySQL, and such and easily isolate these components from each other in a way that's more iron-clad than AppArmor or SELinux.
illumos
illumos is descended from OpenSolaris, and forked after Oracle closed up the source code for Solaris 11. Development has continued on it (at a, granted, slower place). Being the originator of ZFS, it has class A support for it, as well as zones which are functionally equivalent to FreeBSD jails. illumos also has support for SMF, which is essentially advanced service management and tracking without all the baggage systemd creates and tendrils throughout the stack. Zones can also be branded to run Linux binaries to some extent so we can handle migrating the core system over by simply installing illumos, restoring a backup into a branded zone, and then piecemeal decommissioning of said zone. As such, as an upgrade choice, this is fairly attractive. If we migrate to illumos, we'll either use the SmartOS distribution, or OpenIndiana.
Final Notes
Right now, we're basically on the fence with all options, so hopefully the community can provide their own input, or suggest other options we're not aware of. I look forward to your comments below!
~ NCommander
(Score: 2) by VLM on Tuesday February 07 2017, @01:50PM
If you stay on linode there's a nice extremely long and hyper detailed document explaining the linode install process and everything is "as you'd expect" with two exceptions, shut off Lassie temporarily or she'll drive you crazy during the rebooting process (woof woof) and you have to convince freebsd to serial console boot to make the linode admin tools talk to the console correctly (no big deal). Other than that, it looks about as exciting as setting up under openstack was at work. Openstack setup was quite boring I assure you, not even worthy of comment.
As far as freebsd criticisms the coordination between packages is a little weaker than some linuxes so as an example of hilarity a couple weeks/months ago there was an exploit for mysql 5.6 that "couldn't be patched" or some damn thing (or maybe it was 5.5, who cares) that would appear in the daily report emails from my DB servers and that was seen as OK because 5.7 is in the system and you can install it, but the standard for every client of mysql in freebsd was 5.6 so you could install 5.7 which was patched and secure but nothing native to freebsd can talk to it, or you can install 5.6 which works with everything but outputs a security warning every day (or was it week?). In my application it was an intranet internal use only system at work and internal use only at home and you'd have to be pretty crazy to open up the mysql port to "the internet" so I didn't sweat it and ran 5.6 although someone trying to sell DB-aasS to the general public would likely WTF.
As for jails you can do that by hand (tedious) or write your own scripts (error prone) or try the pre-alpha-ish support for docker (maybe its improved since I heard about it; probably it has) or try the CBSD suite which works perfectly and seems straightforward and has been extremely reliable for me although apparently is only used and loved by the developer in Russia and myself and quite possibly no one else on the planet. Its really nice, really ignored, and gets no love at all on the internet. Too bad.
I think given your situation (public internet site) you'd like PF. My interactions with PF have been somewhat faster simpler and more logical than old-days ipchains or iptables. I've never heard anyone complain about PF.
There's sort of three parts, the core system as you mention, /usr/ports manually compiling which is a slow PITA but highly versatile, and pkg-ng which is essentially apt-get for freebsd. I mean literally where you type apt-get update you type pkg update instead, etc. Mixing hand compiled stuff from /usr/ports with pkgng packages can be ... unusually exciting in the Gentoo sense. I mean if you're doing something weird and have to compile your own, then do it, but usually you won't have to and things can get weird (um, do I upgrade xyz in /usr/ports or via pkg or what if I get it "wrong" and do the opposite or)
you didn't mention your config management system. For like a decade I used puppet and it was so much fun having it randomly crash days or weeks of operation on Debian, it was the only unreliable part of the system so I know it wasn't hardware, so its theoretically pull but in practice debug and push constantly, what a PITA. Also on an "internal only system" I greatly enjoyed (sarcasm) running multiple authentication systems so I have kerberos, ssh keys, web "real" SSL, and fake puppet only SSL, what an ungodly PITA it is to keep puppet's SSL running. Which is why I scrapped it all and eventually settled on ansible. And the topic of this paragraph is dealing with ansible on freebsd. First your hosts files will be full of some_hostname ansible_python_interpreter=/usr/local/bin/python or legacy_linux_hostname ansible_python_interpreter=/usr/bin/python. Then in your roles, tasks, you're gonna have lots of stuff of the form " when: ansible_distribution == "FreeBSD" " or VERY similar. Unless someone else has a better idea.
Just like my linux boxes the outta the box ssh config is not that strong. Just like my linux boxes fixing it is pretty uneventful WITH the sole exception of yet another path issue. If you copy a sshd config from Ubuntu you'll see sftp lives in /usr/lib/sftp-server and on freebsd it lives in /usr/libexec/sftp-server. And you'll be like WTF I don't use SFTP so I don't give a F but it turns out ssh won't work AT ALL (or maybe it was just scp that failed, or just console, regardless it was Fed up). Unless sshd can find the referenced module it doesn't work. So you end up with two sshd configs, one for legacy linux and one for freebsd differing hopefully only in paths and nothing else. You really need to replace ssh_config and sshd_config on all linux or freebsd boxes. Seriously, don't some still permit like MD1 MACs and "PermitRootLogin yes" and dumb stuff like that, that sounded good in 1993 but not so much in 2017?
Another fun almost flamewar almost as much fun as emacs vs vi is dealing with 3rd party or homemade scripts running under bash and linux-only-noobs have lines like #!/bin/bash but under freebsd bash lives in /usr/local/bin so its an unholy flamewar if root should say fuck it and symlink /usr/local/bin/bash to /bin/bash on each machine (a simple ansible config to be sure) or if the script should be forcibly converted permanently to freebsd with a like like #!/usr/local/bin/bash or if the script author is a script-kiddie and should file a bug because the script should be starting with #!/usr/bin/env bash which works PERFECTLY cross platform.
ZFS is awesome. AFS on freebsd was not so awesome, I ran that for 10, 15, maybe 20 years, I'd have to think about it, but the big freebsd conversion made me dump it, kernel crashes under heavy load not funny. ZFS survived the crashed and didn't care.
I'm not mentioning stuff that just works, which is pretty much everything else.
(Score: 2) by NCommander on Tuesday February 07 2017, @04:02PM
I hate iptables with the passion of a thousand suns, and am somewhat horrorified that I know it was better than ipchains which it replaced :/. We use Ubuntu's UFW to frontend the firewall which took it from *horrid* to relatively managable. Our firewall rules are actually fairly simple overall so as long as it utter braindamage, it's usable. I'd be fine with plain old ipfw.
Still always moving
(Score: 2) by mrpg on Tuesday February 07 2017, @07:03PM
If anyone here uses Debian:
"After years of heavy development, the nftables framework is ready to use in production environments. You are encouraged to migrate from iptables to nftables. "
https://wiki.debian.org/nftables [debian.org]
(Score: 3, Funny) by TheGratefulNet on Tuesday February 07 2017, @11:43PM
I tried switching from iptables to little bobby tables, but somehow, my system was not usable after I tried it.
(lol)
"It is now safe to switch off your computer."
(Score: 2) by VLM on Wednesday February 08 2017, @02:54PM
You can find nftables in jessie-backports
ah thats why I was like "what is that"
Nice dual stack ipv4/v6
I see its not infected with systemd but is systemd compatible in that people have made files that'll run it from systemd
If its truly superior the *BSDs will just "steal" it so I look forward to that possibility. I don't mind PF though.
(Score: 0) by Anonymous Coward on Friday February 10 2017, @07:55AM
FreeBSD has a superior firewall to pf already: ipfw.
(Score: 0) by Anonymous Coward on Tuesday February 07 2017, @11:09PM
Would you mind providing a short statement on the major problem points of iptables from your point of view?
I primarily just do personal stuff with iptables, and while I can get things to work, I have noticed that it seems clunky to manage from a configuration perspective. (I just use tons of commented-out lines in my config file/script and remove/add comment-marks manually since I'm paranoid, lazy, and like to have the feeling that I know exactly what I just did because I just looked directly at the only iptables config file in use.)
(Score: 2) by The Mighty Buzzard on Saturday February 11 2017, @01:44AM
Actually, we don't anymore. Audioguy hand rolled us an iptables firewall after several hours of us arguing over rule specifics. Details should be on the tech wiki.
My rights don't end where your fear begins.
(Score: 2) by NCommander on Tuesday February 07 2017, @04:06PM
Second follow-up; we don't use centralized configuration management with the exception of production, which we rsync the /srv/soylentnews.org folder which has all the server software and configs in one package.
With the exception of Hesiod+Kerberos, everything is running different software.
Still always moving
(Score: 2) by TheRaven on Wednesday February 08 2017, @01:36AM
but the standard for every client of mysql in freebsd was 5.6 so you could install 5.7 which was patched and secure but nothing native to freebsd can talk to it, or you can install 5.6 which works with everything but outputs a security warning every day
That sucks, and is something that you should poke ports-secteam about if you see it. That said, there's a third option of setting up Poudriere (which is very easy) and running your own package build with the default MySQL versions set to 5.7. It takes about 5 minutes to set up and then the build time obviously depends on the machine performance. On my AMD E-350, building the installed package set takes a few days, on the beefy machines at work we can build the entire ports collection in under 24 hours.
As for jails you can do that by hand (tedious) or write your own scripts (error prone) or try the pre-alpha-ish support for docker (maybe its improved since I heard about it; probably it has) or try the CBSD suite which works perfectly and seems straightforward and has been extremely reliable for me although apparently is only used and loved by the developer in Russia and myself and quite possibly no one else on the planet.
I've not come across CBSD, but it looks pretty good. iocage [readthedocs.io] has replaced ezjail as the popular tool for jail management (unlike ezjail, it's ZFS only, but that does mean that when you are using ZFS it is better positioned to take advantage of all of the shiny features than ezjail).
There's sort of three parts, the core system as you mention, /usr/ports manually compiling which is a slow PITA but highly versatile, and pkg-ng which is essentially apt-get for freebsd. I mean literally where you type apt-get update you type pkg update instead, etc. Mixing hand compiled stuff from /usr/ports with pkgng packages can be ... unusually exciting in the Gentoo sense
End users should never build anything from /usr/ports (do we even install it anymore?). If you want to build your own packages, then use Poudriere. It will handle all of the dependencies for you, builds in a jail and ensures that packages have all of their dependencies rebuilt. You can then just specify the generated package repo in your config and get packages from there if they exist or from upstream otherwise.
sudo mod me up