SoylentNews is people
SoylentNews
So, in previous posts, I've talked about the fact that SoylentNews currently is powered on Ubuntu 14.04 + a single CentOS 6 box. Right now, the sysops have been somewhat deadlocked on what we should do going forward for our underlying operating system, and I am hoping to get community advice. Right now, the "obvious" choice of what to do is simply do-release-upgrade to Ubuntu 16.04. We've done in-place upgrades before without major issue, and I'm relatively certain we could upgrade without breaking the world. However, from my personal experience, 16.04 introduces systemd support into the stack and is not easily removable. Furthermore, at least in my personal experience, working with journalctl and such has caused me considerable headaches which I detailed in a comment awhile ago.
Discounting systemd itself, I've also found that Ubuntu 16.04 seems less "polished", for want of a better word. I've found I've had to do considerably more fiddling and tweaking to get it to work as a server distro than I had to do with previous releases, as well as had weird issues with LDAP. The same was also true when I worked with recent versions with Debian. As such, there's been a general feeling with the sysops that it's time to go somewhere else.
Below the fold are basically the options as we see them, and I hope if the community can provide some interesting insight or guidance.
Right now, we have about three years before security updates for 14.04 stop, and we are absolutely forced to migrate or upgrade. However, we're already hitting pain due to outdated software; I managed to briefly hose the DNS setup over the weekend trying to deploy CAA records for SN due to our version of BIND being outdated. When TLS 1.3 gets standardized, we're going to have a similar problem with our frontend load balancers. As such, I want to get a plan in place for migration so we can start upgrading over the next year instead of panicking and having to do something at the last moment
The SN Software Stack
As with any discussion for server operating system, knowing what our workloads and such is an important consideration. In short, this is what we use for SN, and the software we have to support
- nginx - Loadbalancing/SSL Termination
- Apache 2.2 + mod_perl - rehash (we run it with a separate instance of Apache and Perl, and not the system copy)
- MySQL Cluster for production
- MySQL standard for secondary services
- Kerberos + Hesiod - single-signon/authetication
- Postfix+Squirrelmail - ... mail
In addition, we use mandatory application controls (AppArmor) to limit the amount of stuff a given process can access for critical services to try and help harden security. We'd like to maintain support for this feature to whatever we migrate, either continuing with AppArmor, switching to SELinux, or using jails/zones if we switch operating systems entirely.
The Options
Right now, we've floated a few options, but we're willing to hear more.
A non-systemd Linux distro
The first choice is simply migrate over to a distribution where systemd is not present or completely optional. As of writing, Arch Linux, Gentoo, and Slackware are three such options. Our requirements for a Linux distribution is a good record of updates and security support as I don't wish to be upgrading the system once a week to a new release.
Release-based distributions
I'm aware of the Devuan project, and at first glance, it would seem like an obvious choice; Debian without systemd is the de-facto tagline. However, I've got concerns about the long-term suitability of the distribution, as well as an intentional choice to replace much of the time-tested Debian infrastructure such as the testing archive with a git-powered Jenkins instance in it's place. Another option would be slackware, but Slackware has made no indication that they won't adapt systemd, and is historically very weak with in-place upgrading and package management in general. Most of the other distributions on without-systemd.org are either LiveCDs, or are very small minority distros that I would be hesitant to bet the farm on with.
Rolling-release distributions
On the other side of the coin, and an option favored by at least some of the staff is to migrate to Gentoo or Arch, which are rolling-release. For those unaware, a rolling release distribution basically always has the latest version of everything. Security updates are handled simply by updating to the latest upstream package for the most part. I'm not a huge fan of this option, as we're dependent on self-built software, and it's not unheard of for "emerge world" to break things during upgrades due to feature changes and such. It would essentially require us to manually be checking release notes, and crossing our fingers every time we did a major upgrade. We could reduce some of this pain by simply migrating all our infrastructure to the form of ebuilds so that at least they would get rebuild as part of upgrading, but I'm very very hesitant about this option as a whole, especially for multiple machines.
Switch to FreeBSD/illumos/Other
Another way we could handle the problem is simply jump off the Linux ship entirely. From a personal perspective, I'm not exactly thrilled on the way Linux as a collective whole has gone for several years, and I see the situation only getting worse with time. As an additional benefit, switching off Linux gives us the possiblity of using real containers and ZFS, which would allow us to further isolate components of the stack, and give us the option to do rollbacks if ever necessary on a blocked upgrade; something that is difficult to impossible with most Linux distributions. As such, I've been favoring this option personally, though I'm not sold enough to make the jump. Two major options attract me of these two:
FreeBSD
FreeBSD has been around a long time, and has both considerable developer support, and support for a lot of features we'd like such as ZFS, jails, and a sane upstream. FreeBSD is split into two components, the core stack which is what constitutes a release, and the ports collection which is add-on software. Both can be upgraded (somewhat) independently of each other, so we won't have as much pain with outdated server components. We'd also have the ability to easy create jails for things like rehash, MySQL, and such and easily isolate these components from each other in a way that's more iron-clad than AppArmor or SELinux.
illumos
illumos is descended from OpenSolaris, and forked after Oracle closed up the source code for Solaris 11. Development has continued on it (at a, granted, slower place). Being the originator of ZFS, it has class A support for it, as well as zones which are functionally equivalent to FreeBSD jails. illumos also has support for SMF, which is essentially advanced service management and tracking without all the baggage systemd creates and tendrils throughout the stack. Zones can also be branded to run Linux binaries to some extent so we can handle migrating the core system over by simply installing illumos, restoring a backup into a branded zone, and then piecemeal decommissioning of said zone. As such, as an upgrade choice, this is fairly attractive. If we migrate to illumos, we'll either use the SmartOS distribution, or OpenIndiana.
Final Notes
Right now, we're basically on the fence with all options, so hopefully the community can provide their own input, or suggest other options we're not aware of. I look forward to your comments below!
~ NCommander
(Score: 4, Informative) by canopic jug on Tuesday February 07 2017, @12:41PM
You can't uninstall systemd from Debian since many years ago.
Server-side, Devuan GNU/Linux is quite good. It really is a drop-in replacement. But if you don't trust it yet and really want plain Debian, then there is Debian GNU/kFreeBSD. Only the kernel and a few kernel related user-land tools are different. Nothing that should be noticeable except for PF vs iptables -- unless you run into a hardware compatibility issue.
I think the main question is how interested you are in ZFS. There is some ZFS action in both Devuan GNU/Linux and Debian GNU/kFreeBSD, but the real ZFS action is happening on plain FreeBSD. I myself don't enjoy FreeBSD so much and, again, there is the hardware support question. FreeBSD is not as organized as OpenBSD, but OpenBSD means committing to doing serious OS upgrades at least every 12 months, unless you shell out for the M:Tier options. Some releases of FreeBSD have a long support cycle [freebsd.org] and I think 11 is good until 2022.
Money is not free speech. Elections should not be auctions.
(Score: 3, Insightful) by NCommander on Tuesday February 07 2017, @12:54PM
My problem with Debian/kFreeBSD is they changed out the entire userland to GNU+glibc. This causes a hilarious amount of breakage for anything that hasn't been patched to recognize that (uname == FreeBSD) != (libc == FreeBSD). We have to compile a lot of CPAN modules for rehash, and I can see that just blowing up in hilarious ways if we tried it on kFreeBSD; I have a better chance getting it to work on HURD.
It also failed to meet release qualifications for jessie, and as such was only released to the side, not as an official Debian release (similar to etch-m68k).
Still always moving
(Score: 2) by canopic jug on Tuesday February 07 2017, @01:03PM
It also failed to meet release qualifications for jessie, and as such was only released to the side, not as an official Debian release (similar to etch-m68k).
That was mostly due to systemd though.
How deep into the OS do you need to delve? I got the impression that you were relying on Perl 5, shell scripts, and some pre-packaged databases. Most of that is pretty remote from the kernel.
If FreeBSD were as easy to work with as OpenBSD, then it'd be the hands-down winner. However, once it's set up it is rather low maintenance. Additionally, FreeBSD has jails and ZFS, even if its PF is out of date.
Money is not free speech. Elections should not be auctions.
(Score: 2) by NCommander on Tuesday February 07 2017, @01:31PM
The original notes for Slashcode had a fairly large caveat section on running on BSD due to some of the underlying Perl modules taking a shit. They did note it ran perfectly fine on Solaris. As far as I know, we cleaned out most of that breakage when we migrated the entire stack forward into 2015. For shits and giggles last year for April fools, I tried to setup rehash on Hurd, but failed about half way through it as it fell over due to missing Perl modules. I've since "improved" the installation instructions so I can test it in a VM and fix breakages relatively easy if it came do that.
Still always moving
(Score: 0) by Anonymous Coward on Tuesday February 07 2017, @02:22PM
I am not surprised.
I have the same problem with compiling software that uses assembler or specifies a CPU architecture ever since putting a 64-bit kernel on my 32-bit Slackware install. They check uname, notices a 64-bit CPU and tries to compile a 64-bit version, resulting in an error from the 32-bit compiler.
Asking GCC for the architecture is one line of shell script, and gives all the information (i686-Linux in my case) in one go.
(Score: 2) by schad on Tuesday February 07 2017, @01:06PM
If I were the only one affected by the decision, I'd switch to FreeBSD. I love Solaris, but it's over and done with. FreeBSD is the next best thing.
If others were affected too, I'd seriously consider whether my philosophical objection to systemd is actually worth the practical efforts of avoiding it. This is why, at work, we're going to CentOS 7. None of us are happy about it, but using a niche OS like FreeBSD means that we're committing ourselves to 100% ownership of every aspect of these servers' lives for as long as we have them. That's something we very much don't want to do -- we've got this entire huge IT department with follow-the-sun 24/7/365 coverage; why not use it? -- so we suck it up and deal.
It's weird to me that SVR4 is actually turning out to be the loser of the Unix wars. The proprietary Unixes are basically deprecated even by their owners now, and Linux is doing everything it can to strip away its Unix-clone origins. Meanwhile, the BSDs just keep doing their thing... while slowly absorbing the interesting bits of the last (and first!) SVR4 derivative. Even five years ago, I don't think I would have seen this coming.
(Score: 3, Insightful) by canopic jug on Tuesday February 07 2017, @01:13PM
FreeBSD hasn't been niche for ages. It runs Netflix, is on the PS4, used to run HotMail during its growth phase, ran the now defunct Yahoo, runs WhatsApp, is one of the systems used by Verisign, used in Juniper, and Experts-Exchange (mind the dash). It's just a bit weird to set up and needs reading its handbook to make that possible, though it does have a really good handbook.
Money is not free speech. Elections should not be auctions.
(Score: 2) by Pino P on Tuesday February 07 2017, @02:19PM
I thought that because of the paywall, the user base abandoned Expert S-ex Change in favor of Stack Overflow and the rest of the Stack Exchange network, which runs Windows Server [stackexchange.com].
(Score: 2) by canopic jug on Tuesday February 07 2017, @02:24PM
I've never even looked at it, but do recall the noise about the hyphen. I guess that was a while ago. However, they do have a testimonial up about FreeBSD [freebsdfoundation.org].
Money is not free speech. Elections should not be auctions.
(Score: 0) by Anonymous Coward on Tuesday February 07 2017, @03:27PM
Well, without the hyphen they would be inaccessible from quite a few places with stupid filters. I guess they learned that the hard way.
(Score: 0) by Anonymous Coward on Tuesday February 07 2017, @08:06PM
that explains why the rules/mods are so old school douchetastic.
(Score: 2) by TheGratefulNet on Tuesday February 07 2017, @02:57PM
I worked at juniper back in 2000 or so and their whole eng dep ran on freebsd (desktop, mail, etc). the router itself ran freebsd!
at that time, other networking companies I was at were using netbsd (for power-pc or some other non-intel chip).
now, linux is all the rage ,but linux is not as stable as it once was (oddly enough). I'd go with bsd.
"It is now safe to switch off your computer."
(Score: 2, Disagree) by schad on Wednesday February 08 2017, @12:56AM
When you can rattle off a near-complete list of every major deployment, present or recent past, you're kind of making my case for me.
I love FreeBSD. I like it better than Linux, and I always have. It fits better with my sense for how computers ought to operate. But... I'm sick of fighting with Linux people. They know only one way: The Linux Way. They won't learn FreeBSD not because they can't, but because they don't want to. They get pissy at having to type "netstat -r" instead of "route" (never mind that the former works in Linux just fine). They bitch about "ps -ef" not working the way they expect. And God help you if they have to write a shell script. They'll come to you in a blind fury about how /bin/sh isn't bash, and what kind of idiotic system doesn't even have bash, and so they installed it from "this 'ports' thing, which by the way took forever," but it went into /usr/local/bin/bash instead of replacing /bin/sh like it should, so they made /bin/sh a symlink to /usr/local/bin/bash and now the system won't even boot, and what kind of idiot designed this shit, and why the fuck can't we just use Linux which just works and you don't have to fight with it all the time?
(You may be able to tell that I've been in this situation with a coworker.)
(Score: 5, Informative) by TheRaven on Wednesday February 08 2017, @01:18AM
sudo mod me up
(Score: 2) by NCommander on Tuesday February 07 2017, @01:14PM
If it was just me, we'd upgraded probably have upgraded already. Our CentOS 6 box has driven me mad since GoLive but its still here.
Still always moving
(Score: 3, Insightful) by VLM on Tuesday February 07 2017, @02:32PM
None of us are happy about it, but using a niche OS like FreeBSD means that we're committing ourselves to 100% ownership of every aspect of these servers' lives for as long as we have them. That's something we very much don't want to do -- we've got this entire huge IT department with follow-the-sun 24/7/365 coverage; why not use it? -- so we suck it up and deal.
I've found in practice at "giant Fortune 50" megacorps over a couple decades of linux and now freebsd, that unless you're doing something really niche, there is no niche support required, so if you're doing something weird no one on the planet can help you as much as yourself, and if you're not doing something weird its so easy to help yourself it doesn't matter. Secondly, people who want to work with you, will work with you, and people who don't want to work with you, will not work with you, and the mere topic of OS is more or less irrelevant because if "Mordac the Preventer" in IT land wants to stop you, merely using a supported OS isn't going to provide much armor against him. Or if you have and are using a big club to get past Mordac the Preventer then using your choice of OS is mere icing on the cake.
From a telecom background, isolation and demarcation are the key points. Never tell IT that you're running freebsd and its slow and they need to help you debug freebsd, only tell them the problem is NAS share named wtf-1234 measured multiple times only has 70 K/sec of measured bandwidth which seems a bit low unless the NAS is running on a 360K 5.25 floppy drive. Make sure no one discusses what you're doing with 70 K/sec of bandwidth or your astrological sign or your OS version or any of that, they're just optimistically going to fix the NAS or the network or whatever.
(Score: 2) by Yog-Yogguth on Wednesday February 08 2017, @03:07AM
Very good advice except the very last part which I'm not sure I understood because sometimes wild geese will lead you to the answer/solution, especially if it's a really hard problem way above your "paygrade".
But I still think I didn't get what you meant to say in your last sentence.
Bite harder Ouroboros, bite! tails.boum.org/ linux USB CD secure desktop IRC *crypt tor (not endorsements (XKeyScore))
(Score: 2) by VLM on Wednesday February 08 2017, @02:52PM
I've worked "with" Mordac the Preventer on a couple occasions over the past decades and say you want a dude to add a DNS record and dude wants to play Quake or WoW or pokemon go or Facebook instead, then rather than spending 60 seconds adding your DNS A record you'll get enormous feedback asking if that device has a barcode in the new inventory system or has it been added to the centralized ping monitoring system. You know how 3rd world roadblocks are just random garbage piled up to stop people from getting what they want? "So you want a DNS A record, just to verify does that new windows box have a corporate install of Avast anti virus on it and who paid for that license?" and the F-ing idiot doesn't realize that IP address in the DNS request is the management port of a Cisco ethernet switch or its an arduino ethernet shield or its a SCADA controller that runs VXworks. I've actually had conversations like this.
I've found over the decades that the most personally rewarding way to blow past a human obstacle like that is flame him and his boss until the smell of his burning flesh makes him repent, but the fastest way to actually accomplish the goal is just to politely persistently focus on the task until its clear to Mordac the Preventer that the easiest way to get back to playing WoW is to just add the A record. "Why don't I stay on hold until I verify this is done" "I'll call you back in ten to see how its going" "Your boss told me it wouldn't be a problem as long as I filled out the request form correctly". Tempting as it is as an alternative to discuss something more in the style of "The best in life is to crush your enemies, see them driven before you, and hear the lamentations of their women"
The only thing less effective is going evangelical on the poor MCSE, the last thing he wants to hear is both a request for a simple DNS "A" record PLUS a free rant about the superiority of unix over windows and BTW emacs is superior to visual studio (admittedly all true, but irrelevant to the topic...)
If you know Mordac the Preventer is just going to throw up roadblocks make sure he obtains nothing extra from you to pile up on the roadblock.
Its actually kinda fun blowing past those folks once you get good at it. "The more you tighten your hand, the more star systems slip thru your fingers...."
(Score: 2) by TheGratefulNet on Tuesday February 07 2017, @02:54PM
freebsd is my vote, too.
production headless server that runs ip stuff?
freebsd.
not linux.
only issue: most admins know linux better than bsd. but that's not a big problem, its still unix.
"It is now safe to switch off your computer."
(Score: 1) by animal on Tuesday February 07 2017, @04:35PM
You can install Debian wheezy which is systemd free. Prior to upgrading to jessie, there are steps that can be taking to keep sysvinit and skip the whole systemd installation and removing using apt-pinning.
https://www.debian.org/releases/jessie/amd64/release-notes/ch-information.en.html#systemd-upgrade-default-init-system. [debian.org] I didn't want systemd either, I recently upgraded to jessie after I was show that link on irc.debian.org. The people there are VERY helpful.
(Score: 2) by butthurt on Tuesday February 07 2017, @08:49PM
You can't uninstall systemd from Debian since many years ago.
It worked for me.
(Score: 2) by butthurt on Wednesday February 08 2017, @12:44PM
You can't uninstall systemd from Debian since many years ago.
I've been able to do so, apart from leaving the libsystemd0 package installed; it hasn't caused problems.