SoylentNews is people
SoylentNews
from the backporting-is-not-a-maintainable-process dept.
Two months ago, I polled the community for advice on the underlying operating system that should power SoylentNews (SN). After reading comments, and some recent experiences in my personal and professional life, we are migrating to Gentoo as the operating system of choice. As of right now, we've already migrated our development box, lithium, over, and using it as a shakedown to see how painful the overall migration will be. I'm pleased to report that, aside from varnish (an HTTP accelerator), the process went relatively smoothly.
Recap
For those who weren't here for the original article querying the community (linked above), let me recap the situation. At the time that I wrote that article, SN was mostly standardized on Ubuntu 14.04, with a single CentOS 6.7 box lurking in our midst. In the course of testing updates and other projects, the staff and myself felt that Ubuntu (and Debian) had lost a lot of the advantages that had made it a rock solid choice for the last three years of powering SN, combined with the fact that the upgrade process would not have been trivial due to the systemd migration.
Though greatly disliked by all of us, systemd being part of Ubuntu 16.04 LTS (Long Term Stable) was not a deal breaker. More importantly was the perception that the release lacked stability and we had a serious sense that the upgrade would be problematic. I felt it was time to reopen the scenario to see if we were better off migrating to a different distro, or abandoning Linux entirely. As such the original article was penned to see what the community's feelings on the subject were. The overwhelming consensus was that I was not alone with my feelings on the latest LTS, and many thought FreeBSD would be a good choice for us. Ultimately, we decided to trial Gentoo over FreeBSD for four reasons
- FreeBSD's upgrade system with ports/core isn't anywhere as robust as I would like
- We would lose Linode's backup services, and be fully dependent on our own self-rolled backup system
- Trying to backport security features is ultimately a losing battle
- Gentoo had overwhelming support from the staff
I'll break these first three item by item
FreeBSD Upgrades
FreeBSD is divided into two parts, the core system which has basic utilities, and the ports collection which has all the add on software like Apache and such. In theory, these two components can be updated independently of each other allowing a stable base while migrating to newer software versions with relative ease. Ports can be installed from binaries, or manually compiled to suit one's taste in a relatively automated fashion bringing together the best of a binary and source based distribution. On paper, it looks perfect.
In further research, I've found that port upgrades are fragile at the best of times. Unlike Debian's APT which has strict package dependence and shared library management, port upgrades are very much upgrade and pray and its possible to hose a system in this way. The situation is similar to using EPEL on CentOS, or using Slackware that port upgrades can leave artifacts, and there's often considerable manual intervention to keep things chugging around. This is compounded by the fact that the version of Kerberos we need is in the ports collection due to incompatibility between MIT Kerberos V (which we use) and Heimdal Kerberos which ships out of the box. For those of you familiar with Active Directory, this is roughly on par with the effort required to rebuild AD from scratch along side a pre-existing forest. This meant unless we rebuilt the entire Kerberos domain (a drastic and painful option to say the least) that we could easily break a node because a ports upgrade went sideways.
Furthermore, mixing binary and source ports also have several ways it can go wrong which is problematic. To ease our system maintenance burden, its long been a goal of the admin team to have rehash and its dependencies built and deployed through package management instead of the rather horrorifying script+rsync that we use now. While we could have technically achieved this with Ubuntu by running our own buildd (or using a PPA), the sheer amount of dependencies combined with the pain of rebuilding the world ultimately doomed this to the "would be nice" pile list of ideas.
On top of this, the split architecture of FreeBSD would also mean that upgrades are no longer "one command and done" as they are with Ubuntu and Debian. Instead it becomes a matter of determining what, if any, core system upgrades are available, deploy them, then deploy/rebuild ports as needed. None of this by itself would be deal-breaking, but when compounded with the other reasons it tipped me away from this option.
Linode Backups
For any production website, having backups is the thing you must have, not the thing you wish you had. With the exception of our development box, all our systems are backed up to off-site storage on a machine called oxygen via rsnapshot nightly (and yes, we do test our backups). However, due to the way SoylentNews is situated, there is the possibility that if an attacker ever successfully breached SN, its possible they may be able to gain access to oxygen, and rm -rf / everything.
For this reason alone, we used two separate sets of backups in case of system failure or node compromise. As mentioned many times before, SN is hosted on a number of VPSes by Linode who I continue to highly recommend for anyone's VPS needs. One very useful and handy feature of Linode is that they offer snapshotting and node backups as part of their hosting services for reasonable prices, and critical system boxes are backed up with them as a second-level of defense. Unfortunately, Linode's backup services require that their system understand the underlying filesystem format used by the OS so they can snapshot it easily. As of writing, they do not support FreeBSD's UFS or ZFS. A migration would mean we'd have to sink additional costs in a new backup system to supplement oxygen.
Backporting Security Features is a Losing Battle
I'm going to get flamed for this reason, but recent events have sort of drilled this home for me, both at SoylentNews and as my work as a freelancer. During the last round of security updates, I've been fighting to get several of CentOS's security issues fixed. Red Hat (and CentOS) offer ten year support for their products but in many ways it is the wrong approach to system stability and security. A real-world issue I ran into with CentOS's support is that they ship rather old issues of dovecot, a relatively popular IMAP server.
Now, in theory, as long as security patches are backported, this shouldn't be a problem. In practice however, it means you're essentially tethered to the security features as offered at the time of the release. For example, a good number of our users are likely familiar with the Logjam attack. The mitigation for Logjam is to regenerate DH parameters to larger sizes, and change to a non-common prime. Relatively straightforward, right?
Well, not so much. Dovecot 2.0 (which is what CentOS 6.7 ships with) doesn't allow for setting of custom DH parameters, or even tweaking anything beyond the most basic TLS settings. To a lesser extent, we also had this problem with Postfix (we can't disable client side negotiation). The solution in both cases is to upgrade. That would be great, if we could in-place upgrade CentOS, or reliably upgrade the RPMs without hosing YUM at a later date. In practice, we've been forced into doing a number of arcane hacks to get most of the survey tools to report anything better than a "C" grade, with the situation worsening as time goes on. Before people say "well that's a problem with dovecot", and not CentOS, you can't get OCSP stapling (which is an important security feature to help fix SSL's revocation system) with Apache out of the box. You need to either patch Apache 2.2 in place, or upgrade to 2.4.
This problem also has shown its head on Ubuntu. To Canonical's credit, their security team actually has gone through the work of mainlining newer security features in popular products; Ubuntu 14.04's Apache 2.2 supports OCSP stapling because they patch Apache in their binaries. However this practice only goes so far. Deploying CAA records to SoylentNews in the last round of tweaks was an exercise in frustration because only the most recent versions of BIND knows how to handle the CAA record type. Once again, we're in serious voodoo territory if we tried to upgrade BIND outside of a distro release.
This brings me to my final point: trying to follow industry best-practices falls apart if you can't easily update your stack. Release based distros at best (with Ubuntu) update once every six months, or once every year or so for longer term support from other distros. That's a very long time in the security world. Furthermore, each major upgrade is an event and a large time sink in and of itself. As such I've (grudgingly) come to the conclusion that if you actually want to have real security, you need to update frequently. Furthermore, by having smaller upgrades at a given time instead of them in one large pile, you have a better chance of not getting overwhelmed at those release points.
Why Gentoo
Gentoo ultimately won by being both rolling-release based, and source based. It meant that we could easily upgrade the entire stack (including rehash's special dependencies) as a single emerge world, and then deploy. It also edged out the other options by not forcing systemd on us (and OpenRC is an absolute pleasure to debug and maintain in comparison). We've also discussed the issue at length and have determined how we're going to approach the rather daunting task ahead of us.
The Steps Ahead
The first step, which was already completed, was to migrate our development system over to Gentoo to get an idea of how much pain we're going to be in. This was accomplished by booting the system in rescue mode, moving "/" (i.e. the root of our filesystem) to "/old-rootfs", extracting a stage3, cooking the kernel, and rebooting. audioguy and TheMightyBuzzard worked out the correct set of USE flags for our environment, and I used the serial console to do the actual changeover. Aside from Varnish breaking, the migration was actually relatively smooth if time consuming. Right now, we're still wrestling with varnish, but after kicking MySQL cluster's init scripts and copying configs, it sputtered to life and dev.soylentnews.org popped back onto the internet.
The next steps is to create ebuilds for hesinfo (a Hesiod support tool that Gentoo doesn't ship in their hesiod package), and then to create a custom stage3 with our kernel config and base system with catalyst. We're going to work out the set of packages we need and configure lithium to work as a binary package source for portage. In effect, every package we need will be compiled once on lithium, then published via a private portage repository. Other machines will simply be able to emerge world and download the pre-tested and compiled binaries in one fell swoop keeping the software stack across SoylentNews consistent across the organization. As an added bonus, we can now easily migrate our custom set of compilation scripts to ebuilds and have sane package and dependency tracking for the entire site infrastructure.
Since most of the site infrastructure is fully redundant, we don't expect too much downtime or breakage as we begin migrating other boxes from Ubuntu. As usual, we'll keep the community apprised of our status, as well as if we need to schedule actual site downtime during this period. While some of us might thing we're insane, I will just note for the record we took a similarly drastic step of migrating to a IPv6-only backend two years ago in the name of administration sanity, and serving SN needs best. As always, I'll be reading and commenting below.
~ NCommander
(Score: 3, Funny) by Anonymous Coward on Thursday April 20 2017, @06:09PM (4 children)
Ok, but you will edit configuration files using ed :)
(Score: 2, Funny) by Anonymous Coward on Thursday April 20 2017, @06:45PM (2 children)
Who is ed? My config file editor is called Jim and he does a great job using vim. Useless relational db admin though...
(Score: 4, Funny) by Anonymous Coward on Thursday April 20 2017, @06:51PM (1 child)
The name's Joe, and he uses Joe's Own Editor.
(Score: 0) by Anonymous Coward on Wednesday May 03 2017, @08:55PM
No no no, it's Jonathan, and he uses JOVE, Jonathan's Own Version of Emacs. https://en.wikipedia.org/wiki/JOVE [wikipedia.org]
(Score: 0) by Anonymous Coward on Friday April 21 2017, @01:42PM
Did you say ed? https://www.gnu.org/fun/jokes/ed-msg.html#content [gnu.org]
(Score: 5, Insightful) by DarkMorph on Thursday April 20 2017, @06:10PM (34 children)
And no systemd whatsoever.
(Score: 4, Insightful) by goodie on Thursday April 20 2017, @06:20PM
And no systemd whatsoever
That sold me right there.
i understand the FreeBSD issue. I love FreeBSD but I don't run it on a production system in an "enterprise" setting so I can see how the dual ports/pkg system can be touchy (although I never use the pkg system myself and have always used ports). Right now, I'm putting together a VM for a course I am giving and ended up going for Lubuntu against my will, simply because it provided the best cde performance for the buck out of the box. but seriously fuck systemd...
(Score: 0, Troll) by bob_the_systemd_builder on Thursday April 20 2017, @08:13PM (29 children)
Are people still really scared of Systemd? If so you should just really switch over to Windows at this point because Linux systems are just going to be too hard for you to use. Learn to embrace change yo! This is what I see in my head when all the old people on their rascal scooters complain about Systemd.
https://www.youtube.com/watch?v=LHpdgHTINik [youtube.com]
(Score: 5, Insightful) by linuxrocks123 on Thursday April 20 2017, @08:52PM (1 child)
Your assertion is that, yes, there is a giant f*ing hole in the house, but it's unreasonable to be unhappy about that.
Additionally, disliking something is not the same thing as fearing it.
(Score: 0) by bob_the_systemd_builder on Friday April 21 2017, @02:30PM
My assertion is that everyone is scared of change and Stewie is funny. Man fears what he does not understand and tries to mask fear with hate.
(Score: 3, Informative) by Azuma Hazuki on Thursday April 20 2017, @08:55PM (5 children)
SystemD per se isn't the issue, it's 1) that it seems to be a vehicle for making RedHat money, i.e., "vendor lockin through obscurity" and 2) the way it's implemented is very monolithic and difficult to configure, thus breaking the shit out of The Unix Way (TM).
Those of us who've either been using *nix a long time, or like me, started with something that forces you to get down and dirty with the guts of the system, have a creeping, rotting feeling that "this is not beautiful, this is not just, this is not open" when dealing with SystemD as it is. SysVInit has issues, yes, but OpenRC does wonderfully and for the life of me I cannot understand why it didn't become standard. I always knew my devotion to Gentoo would pay off...
I am "that girl" your mother warned you about...
(Score: 1, Interesting) by Anonymous Coward on Thursday April 20 2017, @09:23PM (3 children)
Notice how redhat based systems are mentioned as easy to hack in the recent wikileaks?
Ever wonder why.
Defects by design.
(Score: 2) by kaszz on Friday April 21 2017, @08:56AM (1 child)
Hope someone checks out where Red Hat gets their money flow from. If they have done a "RSA deal" or if their board is way to cosy with certain elements of the government.
(Score: 0) by Anonymous Coward on Friday April 21 2017, @05:13PM
RedHat's largest single client is the US Department of Defense.
(Score: 0) by bob_the_systemd_builder on Friday April 21 2017, @02:36PM
Red Hat is only easy to hack if you turn off selinux. If you're not using selinux you're wrong. The more used an operating system is the more vulnerabilities are found. That's why you see tons of exploits for firefox and not lunascape.
(Score: -1, Troll) by bob_the_systemd_builder on Friday April 21 2017, @02:27PM
1) I'm very curious how you think Systemd makes Red Hat any money. Can you show any proof or are you just babbling about "vendor lock in" because you don't like enterprise operating systems? On that note you CAN force another init system if you really want to, but that's like sending back a fancy steak and asking for a happy meal. There's also no obscurity. https://github.com/systemd/systemd [github.com] It's open source so you can review all of the code and understand the inner workings. Systemd is incredibly straight forward once you take the time to learn it. How is it not open?
2) Systemd is far easier to configure than Upstart or any of the older sysv init systems. A unit file is VERY simple to build and adjust. What would take over 30 lines in an init script takes 10 lines in a unit file. The Unix way is old and needed to be changed.
Been using Linux since the early 90's and I started with Slackware. I've seen a variety of init systems and Systemd is the best so far. So don't try and pull the "Been using it longer" card.
(Score: 3, Insightful) by Scruffy Beard 2 on Thursday April 20 2017, @09:26PM
Um... Stewie is complaining about a broken house.... I think Stewie is being completely prudent in that clip.
(Score: 1, Insightful) by Anonymous Coward on Thursday April 20 2017, @10:30PM (2 children)
And you sound like a naive young jerk who doesn't even comprehend his own limitations. Go look back at your old school pictures, the horror you experience at seeing your young "fashion sense" will be similar to what happens when the clouds finally lift from your mind and you can see the bigger picture. So many techies can't comprehend that yes, the world is that fucked up, and they have some weird faith that corporations wouldn't do nasty stuff because they'd go out of business... hahahahahahaha
(Score: 0) by bob_the_systemd_builder on Friday April 21 2017, @02:39PM (1 child)
Depends on your definition of young. If people in their 50's are young then I'm a spring chicken baby! Also while the world is fucked up I don't see what anything you said has to do with systemd. Are you saying systemd is the trump of linux? Corporations are people dude. Don't bash people. It's just wrong. Trolllollollollollollollollol. Seriously though what big picture are you referring to?
(Score: 0) by Anonymous Coward on Sunday April 23 2017, @04:28PM
Not sure about the GP but the big picture on systemd, IMO, would be bad by design, i.e. the monolithic opaque design. I've certainly seen again and again that from an engineering perspective, it is a bad design. It's is prone to breakage and difficult/impossible to fix. The only slightly positive comment I could say about it is that, it can be a useful design for trying to stop competitors and for locking others out. But I'm not trying to 'compete' with my OS. I want something that works as stable as possible and is easy to fix when it doesn't work. I see systemd as technical change that's non-cooperative. This from my user perspective is undesired.
But no fear. We use purposely crappily designed tech and systems every day. I don't fear a crappily made screw driver. But I'm sure going to avoid using the piece of junk if I can get my hands on something better.
(Score: 0) by Anonymous Coward on Friday April 21 2017, @07:07AM (7 children)
> Are people still really scared of Systemd?
you mean, that piece of software that is behaving exactly as its detractors predicted?
(Score: 0) by bob_the_systemd_builder on Friday April 21 2017, @02:42PM (6 children)
Please explain how systemd is behaving badly. My complaint that old sys admins hate change is pretty evident and well documented. See your post above for proof. But what problems is systemd causing?
Just like any other piece of software. If it's properly setup it runs perfectly.
(Score: 2) by Scruffy Beard 2 on Saturday April 22 2017, @06:34AM (5 children)
The only weirdness I have personally seen so far is that the filesystem check did not run properly on a dirty boot. Can't have a 5 minute Fsck hold back parallel booting, now can we? The Knoppix live DVD got rid of systemd because parallel boot slows things down considerably when seeking costs up to 300ms.
There was that bug with systemd re-implementing the rm utility...poorly [github.com]
Systemd is behaving badly by trying to do everything: with a development team that apparently does not take serious bugs seriously.
The advantage of "the UNIX way" is that you can rip out and re-implement troublesome components. With systemd, it is largely an "all or nothing" proposition.
(Score: 2) by linuxrocks123 on Sunday April 23 2017, @03:39AM (4 children)
Wow ... hadn't heard of THIS one lol. I guess bricking the machines of people who did rm -rf --no-preserve-root / wasn't stupid enough for them.
Do you know what exactly "R!" is, why it exists, etc.? I can't find too much documentation on this command -- if it is a command -- not unusual for SystemD's documentation to be poor, of course.
(Score: 2) by Scruffy Beard 2 on Sunday April 23 2017, @06:14AM (3 children)
Good question. I had to go back to the bug report to find it.
R is used in configuration files to tell systemd what directories to routinely delete.
systemd-tmpfiles [freedesktop.org]
(Score: 2) by Scruffy Beard 2 on Sunday April 23 2017, @06:21AM (2 children)
It is actually documented here: tmpfiles.d(5) [freedesktop.org]
(Score: 2) by linuxrocks123 on Sunday April 23 2017, @08:15AM (1 child)
Good find. Thanks for the info.
The bug you linked is an excellent example of the problem with their style and attitude. Reimplementing "rm" is NIH to an absurd and terrifying extreme.
(Score: 1) by Scruffy Beard 2 on Sunday April 23 2017, @03:03PM
They apparently needed a way to clean up files without using simple shell scripts....
(Score: 3, Insightful) by Runaway1956 on Friday April 21 2017, @02:05PM (7 children)
Well, bob_the_systemd_troll - I don't think anyone "fears" systemd. We distrust it, generally. Except for some of the totally clueless, who have no idea WTF systemd is. "WTF is an init system, anyway?" Those of us who don't actually distrust it, dislike it. What we all want to know is, who is paying whom to make systemd default in all the distros? To date, there has been no compelling reason to choose one init system over another. But, systemd is growing in popularity - among distro maintainers. Why?
(Score: 0, Informative) by bob_the_systemd_builder on Friday April 21 2017, @02:49PM (4 children)
Since systemd is fully open source I'm wonder what there is to distrust. Also I know fully well what an init system is and have probably been writing init scripts as long as if you have (if not longer). The reason why systemd is replacing upstart and other init systems is because it's better. Unit files allow more flexibility and simplicity of setup. With cgroups being so heavily used I have full control over resource consumption as it's being launched. I have the ability to closely monitor every single process under a service. Lets look at the legacy systems. If I run 'service httpd status' I get a single PID, but very little information as to how apache is actually running. I also can define dependencies in a simple straight forward manner. This makes it so I don't have to rely on a number based order.
TLDR, systemd has tons of benefits and is easy to use. What don't you "trust" when you can read over the source code for yourself?
(Score: 2) by Runaway1956 on Friday April 21 2017, @03:33PM (3 children)
I'll grant that AFTER you understand WTF systemd is doing, it is relatively "simple", in that, everything can be found in more or less one place.
What's to distrust? Monoculture is always untrustworthy, in and of itself. Why has Windows always been so easy to hack? Well, because damned near every Windows system in the world responds to the same outsite stimuli as every other Windows system in the world. If you can get into one Windows system, you can get into damned near all Windows system by doing the same thing again and again. I believe the term "script kiddie" specifically targeted Windows "hackers" who knew how to use a script, but had little if any idea how it worked. With Unix-likes, the script kiddie thing never got the traction that Windows users got. Pretty much, each system had to be targeted, and the majority of them didn't respond to the same outside poking and prodding that any other system responded to.
Give it a couple more years, and anyone who can exploit a systemd system, will be able to get into half or more of the Linux systems in the world.
I think the *nix world is losing something valuable, when all of the distros make the same init system default.
Unlike others, I don't "hate" systemd. Not yet, anyway. But, I distrust it. And, I have little faith in the future of systemd. It threatens to make Linux more Windows-like than Unix-like.
(Score: 2) by Gaaark on Friday April 21 2017, @06:05PM (1 child)
I've started using Arch (Manjaro, really, because of the easy install --my free time seems to be so unavailable anymore) because of the speed: started hating the sloooooowness of Ubuntu.
If only Gentoo had an 'easy' install, i'd probably switch (from what i can see, aren't arch and gentoo 'basically' the same thing?)
If i get a patch of free time, i might just install Gentoo, though. At least on a separate partition, dual boot with arch/manjaro.
Will look into removing systemd from arch, but it looks like a difficult time, from what i'm reading.... :(
--- Please remind me if I haven't been civil to you: I'm channeling MDC. ---Gaaark 2.0 ---
(Score: 2) by The Mighty Buzzard on Friday April 21 2017, @08:33PM
Have a look at Calculate Linux. It's essentially gentoo with a binary overlay and some handy admin type scripts thrown in. If you like you can remove all things Calculate after install and just have a Gentoo system. Plus it's quite a lot easier to install.
My rights don't end where your fear begins.
(Score: 2) by Gaaark on Friday April 21 2017, @06:06PM
oh, yeah... main reason for posting here:
Bob? Runaway put his balls in your court.... whack away! :)
--- Please remind me if I haven't been civil to you: I'm channeling MDC. ---Gaaark 2.0 ---
(Score: 0) by Anonymous Coward on Friday April 21 2017, @03:07PM (1 child)
> But, systemd is growing in popularity - among distro maintainers. Why?
Because it is so much better than alternatives that it even worth switching the default.
> who is paying whom to make systemd default in all the distros?
Must be a consortium of NSA, FSB, Mossad and the Chinese to haxxor all of the internet. Isn't it obvious?
(Score: 0) by bob_the_systemd_builder on Friday April 21 2017, @03:20PM
That just sounds crazy. It's clearly our lizard overlords trying to keep us in check. Linux admins were getting too close to the truth.
(Score: 0) by Anonymous Coward on Friday May 05 2017, @08:21AM
People are not scared of systemd, just see it as wrong minded and going in a direction they care not to follow.
It's true that many popular distros have become redhat derivatives and for those that don't notice or care that will be their linux world, they've been captured.
But those people don't contribute in any great measure, only users have been captured.
For the people who do care, do contribute the only choice is to move away from systemd/linux in exactly the same way as we moved away from windows.
This is not a zero sum game, systemd people seem to think they have 'won' others just see them as damage and are routing around them and following linus' advice, to paraphrase:
"linux is a big project with lots of people, find people you can and like work with, doing things you care about"
(Score: 0) by Anonymous Coward on Friday April 21 2017, @05:07AM (2 children)
Are you sure? Distrowatch says gentoo included systemd in 2014. https://distrowatch.com/table.php?distribution=gentoo [distrowatch.com]
(Score: 2) by The Mighty Buzzard on Friday April 21 2017, @10:25AM (1 child)
True, it is, however, an optional package and can be avoided entirely by making sure the -systemd flag is set in your USE flags in make.conf.
My rights don't end where your fear begins.
(Score: 0) by Anonymous Coward on Friday April 21 2017, @03:03PM
Thank you! I didn't know about that flag. I've been wanting to try GenToo and now I will. Tired of the "end of support" crap with others (like CentOS). Been a longtime SlackWare nut and even though it has versions, you can "keep it rolling" well too.
(Score: 2, Interesting) by Anonymous Coward on Thursday April 20 2017, @06:34PM
Run musl if you can. While there is no promise it is 100 percent security issue free, it is a *MUCH* smaller codebase than glibc, supports utf-8 natively (check the website for support/errata for other encodings.), supports gcc 3-6+clang for compilation, has no symbol versioning restrictions to force you to upgrade/recompile software, and is competitive with glibc on code size and operation speed in most circumstances.
You would need to benchmark with identical loads to 100 prove it equivalent or better performance to glibc for your workloads, but the long term maintenance savings and compilation speed from its much more minimalist headers/code size, etc could help both with the long term security and maintainability of your servers. Plus with libcxx-abi, libcxx, and clang, you could have a fully legacy-free C++ platform for any C++ based applications you are running on the system. I highly recommend it (again!)
(Score: 0) by Anonymous Coward on Thursday April 20 2017, @06:44PM (8 children)
n/t
(Score: 0) by Anonymous Coward on Thursday April 20 2017, @06:53PM (7 children)
(Score: 3, Funny) by turgid on Thursday April 20 2017, @06:55PM (5 children)
-fomit-instructions
I refuse to engage in a battle of wits with an unarmed opponent [wikipedia.org].
(Score: 3, Funny) by The Mighty Buzzard on Thursday April 20 2017, @08:05PM (4 children)
-fomit-omissions
My rights don't end where your fear begins.
(Score: 3, Funny) by NCommander on Thursday April 20 2017, @10:05PM (2 children)
-fomit-trolls -ffeed-buzzards
Still always moving
(Score: 4, Insightful) by LoRdTAW on Thursday April 20 2017, @11:39PM
Those two options are mutually exclusive and cause GCC to exit.
(Score: 2) by jasassin on Friday April 21 2017, @01:19AM
-fpeephole2
jasassin@gmail.com GPG Key ID: 0xE6462C68A9A3DB5A
(Score: 2) by c0lo on Friday April 21 2017, @02:16PM
-finstruct-ommisions
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 2) by kazzie on Thursday April 20 2017, @06:57PM
Variant Reader, is that you?
(Score: 3, Funny) by EvilSS on Thursday April 20 2017, @07:09PM
(Score: 3, Interesting) by its_gonna_be_yuge! on Thursday April 20 2017, @07:12PM (4 children)
15 years and running. My company is all Gentoo with the exception of some Windows boxes which act as machine controllers that can't be migrated, and some 3D design boxes running Rhino3D. Now that Rhino3D is ported to the MAC, I'm hopeful they put out a linux version so that I can get rid of a few more Windows machines.
There have been a couple of hiccups with Gentoo - most notably with the slip in admin management and re-attempt by Daniel Robbins to take over again. And the Gentoo charter was revoked at one point. But it blew over fairly quickly and in hindsight seems to be a lot less of an issue than what Ubuntu is going through recently.
Overall, Gentoo just seems to be logical. Things are where you expect them, and options are left open to users as much as possible.
(Score: 3, Insightful) by rleigh on Thursday April 20 2017, @07:58PM (2 children)
> "options are left open to users"
There's the defining difference. For some inexplicable reason, the rest of the world seems to have taken it into their heads that users aren't to be given any options at all. Because it's "bad", or something. The rot started with GNOME, but it's spread since.
What I can't understand is that free software, open source software, whatever, was intended to empower the end user, to give them the ability to write software, adapt existing software to their needs, to let them have the freedom to get their stuff done. And for a long time, this was absolutely the case, and many projects and distributions intended and encouraged their users to do exactly that. And we did, and it was great fun for a long time. But for a decade or more, a number of high-profile people and projects have insisted that this is awful and to be discouraged. It's taken control from us users and distribution maintainers and put it firmly into corporate control. I have as much control on a "modern" systemd system than I do on a Windows system; it's a black box which I hope works, because when it doesn't I just hit the reset button and hope it works the second or third attempt. You can't get too enthused about that sort of system.
(Score: 4, Insightful) by Azuma Hazuki on Thursday April 20 2017, @08:52PM
Options require time and learning. Sane defaults are a good thing, especially if we want Linux mainstream...problem is, I have yet to see any truly sane set of defaults in any distro's DE aside from Mint.
Watching most people, they're as clueless on Windows as they would be on a sane Linux, and if anything more likely to harm themselves. I got my parents onto a Debian/Xfce setup years ago and i haven't heard a single complaint yet.
I am "that girl" your mother warned you about...
(Score: 2) by kaszz on Friday April 21 2017, @09:05AM
Corporations are inclined to control and dumbing down. The leverage that corporations have is hired developer time and marketing. Being able to provide a ready-to-go-box in the shop etc. Oh and someone for the boss to yell at when things doesn't work.
So make them do the effort and then reuse the effort to open the options and freedom again.
(Score: 2) by kaszz on Friday April 21 2017, @09:00AM
Maybe you can throw Microsoft Win and Apple Mac into a virtualization environment and mmap any critical devices you need?
(Score: 3, Insightful) by rleigh on Thursday April 20 2017, @07:49PM (8 children)
Given the requirements you had, it seems like a pretty sensible and rational choice. In a world which has gone mad, gentoo is at least maintaining some sanity. Thanks to all the OpenRC developers! I only wish I'd had the opportunity to integrate it in Debian back when I was a sysvinit maintainer; it would have been a fine upgrade from sysv-rc. Unfortunately, that wasn't possible as I'm sure we're all painfully aware.
I have gone the FreeBSD route on some systems, but none are public facing like soylentnews, and I am able to use the stock pkg binaries. So far I've not had any big problems with upgrades of the base system or the ports, and I've taken one system from 10.0 to 10.1, 10.2, 10.3 and 11.0 and it's all been great. I would say that the pkg tool has made great strides over the last few years. Just a year and a half ago, it was failing often when upgrading a desktop system with thousands of installed packages; the improved SAT solver has meant I've not had a single issue in well over a year. For me it's now as reliable as apt-get, and I do regular updates of work systems without a hitch. Since last week I have automatic provisioning and deployment of FreeBSD11 nodes with openstack and ansible; it's working fine as a replacement for Debian in practice for me, though I can accept that won't be true for everyone. As it continues to improve, it might become increasingly viable though.
Just as an FYI, poudriere exists to do automated custom ports builds. Had you wished to go this route, you could have set up your own package repository this way.
Out of interest, what's the percentage of IPv6 traffic being handled? I'm connecting via native v6, but I'm just curious how your observations match the general stats from e.g. https://www.google.com/intl/en/ipv6/statistics.html [google.com]
(Score: 2) by The Mighty Buzzard on Thursday April 20 2017, @08:16PM (1 child)
It's really difficult to tell. We don't run access logs on our Nginx load balancer (or the Apache frontends) unless there's an issue that needs debugging or a ddos that needs mitigated. And we can't trust linode's traffic numbers because we do daily backups of nine systems over IPv6 to our remote backup box, oxygen, which until recently included backing up everything in /usr, /home, and /var.
If NCommander can figure out a way to check accurately, I'll punt the question off to him.
My rights don't end where your fear begins.
(Score: 4, Interesting) by NCommander on Friday April 21 2017, @10:46AM
You can actually seperate Linode's graphs into inbound and outbound. Last I checked, it was roughly about 10-12% of all inbound traffic was IPv6.
Still always moving
(Score: 0) by Anonymous Coward on Friday April 21 2017, @02:06AM (1 child)
Debian seems to have OpenRC avaliable now. And there is Devuan too in case Debian did not have it (and probably will have to keep it once the "you can use something else" becomes "bend over, but we are not admiting we should just also bend over and everyone use Fedora, we will keep with the non-bending-over farce").
Maybe talking about the far past?
(Score: 0) by Anonymous Coward on Friday April 21 2017, @06:20AM
Doesn't Devuan remove OpenRC from Debian to provide an even higher level of init freedom?
(Score: 0) by Anonymous Coward on Friday April 21 2017, @09:09AM (3 children)
Have you tried to setup a FreeBSD machine as a soylentnews webserver just to see how it works out?
And yeah the package system with "pkg" is a breeze.
(Score: 2) by NCommander on Friday April 21 2017, @10:45AM (2 children)
It was mostly loosing Linode's backup abilities that were the final nail in the coffin combined with the fact the migration would have considerably more difficult and painful.
Still always moving
(Score: 2) by srobert on Friday April 21 2017, @04:27PM (1 child)
I'm ignorant about what's involved in running a site like this, and moreso about Linode. Hence my question is, are ZFS Snapshots something that could handle the backups that the site uses Linode for?
(Score: 3, Interesting) by NCommander on Sunday April 23 2017, @09:31AM
ZFS snapshots can be deleted if you get root. Linode backups can only be deleted if the master account for Linode is breached. To successfully hose the site, you'd have to compromise the Linode panel *and* our offsite backup system which requires very specific permissions to access.
Still always moving
(Score: 2) by Azuma Hazuki on Thursday April 20 2017, @07:51PM (3 children)
Gentoo was my very first Linux way back in 2004, and I'm *still* running it! What most people don't get is that once you've done the initial setup, it's really not much more, if any, work than a binary distro. And the control it gives you is completely unprecedented, even better than FreeBSD's Ports system (on which Portage is based).
I've run Gentoo in production environments before, though of course being careful not to use ~arch unless absolutely necessary. It was amazing.
I am "that girl" your mother warned you about...
(Score: 2) by tibman on Thursday April 20 2017, @08:26PM (2 children)
Similar here. Currently have a linode running Gentoo. The only gripe i could have is sometimes emerge Blocks aren't easy to resolve without visiting the forums.
SN won't survive on lurkers alone. Write comments.
(Score: 2) by Azuma Hazuki on Thursday April 20 2017, @08:50PM (1 child)
Hah. I have to teach myself. Last night I got LXPanelX running--that's a fork of LXPanel that IMO should have been merged into the project, it's way better and looks nice. Had to edit the ebuild to change the source, the eclass/inherit lines, and force it to use pkgconf vs pkgconfig (they were blocking). Never heard of any of these before but by Cthulhu I got it to work.
Gentoo makes you learn like nothing else.
I am "that girl" your mother warned you about...
(Score: 2) by tibman on Friday April 21 2017, @03:05PM
You know you're spoiled when using a different distro and think: "Where do i set the flags to drop fortran support? It's taking up disk space!", lol
SN won't survive on lurkers alone. Write comments.
(Score: 2) by wonkey_monkey on Thursday April 20 2017, @09:10PM
Soylentnews.org is Moving to Gentoo...
For some reason the ellipsis makes you sound slightly embarrassed about this.
systemd is Roko's Basilisk
(Score: 0) by Anonymous Coward on Thursday April 20 2017, @09:21PM (2 children)
Fuck SystemD
We've been saying for years that it is an intentional obfuscation of security holes.
Wikileaks confirmed this recently.
(Score: 2) by kaszz on Friday April 21 2017, @09:36AM (1 child)
Any specific pointers where this is explicit?
(Score: 0) by Anonymous Coward on Friday April 21 2017, @02:15PM
Of course not. This, and the Snowden files, are used as blanket statements to support whatever anti-government nonsense one wants to spout. It is like all the claims of what is said in the Bible or the Constitution. You can make those claims because 95% of the people have never read those documents, and it isn't worth their time to dig through them to prove you incorrect, AND, if you already believe the nonsense, the very statement that it is supported by those documents is entirely believable to you and makes you believe it even more.
(Score: 2) by RedBear on Thursday April 20 2017, @10:49PM
I was one of those pushing for FreeBSD. My comment from last time seemed to be well-received, about how it's been a pretty stable Unix community for decades. I still like the overall more conservative approaches they tend to use in the BSD world.
But... every potential issue presented in the summary makes sense, so I'm totally sold on the end decision not to go with FreeBSD on a public-facing production web server that needs to rapidly roll with new security threats. Ultimately it is the people doing the actual work of maintaining the site who need to be happy with the system, its stability, security, adaptability and compatibility with other systems. Sounds like Gentoo balances all those things quite well. Being able to keep the hell away from systemd for another few years is a huge bonus.
The examination of the decision-making process presented here in the summary shows once again that people like NCommander have an order of magnitude more knowledge and experience in this area than I ever will, so I feel confident that the results of this decision will be good for the site. Now to google "OCSP stapling".
Great work as always. Your efforts are not in vain. Have fun on the grand migration adventure.
¯\_ʕ◔.◔ʔ_/¯ LOL. I dunno. I'm just a bear.
... Peace out. Got bear stuff to do. 彡ʕ⌐■.■ʔ
(Score: 3, Interesting) by requerdanos on Thursday April 20 2017, @10:56PM (12 children)
we could easily upgrade the entire stack... as a single emerge world, and then deploy.
Out of curiosity, how long does something like
time emerge world
take?
(Score: 2) by Techwolf on Friday April 21 2017, @12:19AM (4 children)
On my one gentoo server, about an hour or less. On my desktop, about 20 to 24 hours.
(Score: 2) by requerdanos on Friday April 21 2017, @01:09AM (1 child)
24 hours seems like an inconveniently long wait, but an hour is statistically pretty close to no time at all.
Thanks.
(Score: 0) by Anonymous Coward on Friday April 21 2017, @11:32AM
You're not going to wait for that. As soon as parts are installed, you can already use them. The advantages of a multitasking system.
(Score: 2) by kaszz on Friday April 21 2017, @09:39AM
What are the hardware specs for those? (CPU, frequency, memory subsystem, disks, motherboard bridge..)
(Score: 0) by Anonymous Coward on Friday April 21 2017, @02:17PM
If it goes well. Years ago I did that on a new server and it wouldn't boot any longer. Then you lose a day fixing it (my fix eventually ended up being installing CentOS instead).
(Score: 2) by The Mighty Buzzard on Friday April 21 2017, @01:08AM (6 children)
Well, if you're doing it straight up standard, it can take quite a long time. If you're doing like we will be and building everything we use on one box and setting that box as PORTAGE_BINHOST for the others, it'll take a while for the box doing the building and a couple minutes for all the rest of them.
My rights don't end where your fear begins.
(Score: 2) by requerdanos on Friday April 21 2017, @01:11AM (1 child)
building everything we use on one box
Even better if said box is...
- very very fast
- used for pretty much nothing else
(Score: 2) by The Mighty Buzzard on Friday April 21 2017, @01:58AM
Dev box, so sees almost no use when we're not actively testing something and has the benefit of us not caring if we break it once in a while.
My rights don't end where your fear begins.
(Score: 0) by Anonymous Coward on Friday April 21 2017, @03:40AM (3 children)
Are you using distcc?
(Score: 0) by Anonymous Coward on Friday April 21 2017, @04:46AM
I thought distcc was unmaintained. Perhaps I was wrong. But what we do here is use ccache with a shared cache. The machine with the lowest usage (usually the hot spare) is scheduled to go first. That helps with a shared cache because the other production machines can update with minimal interruption by using the cache generated by the spare machine.
(Score: 3, Informative) by The Mighty Buzzard on Friday April 21 2017, @10:32AM (1 child)
Nah, we don't want the web frontends, load balancer, db nodes, and such like having to peg out even one core each for compiling updates. Dev, however, isn't doing anything useful at all 99% of the time so if it takes half a day to build all the packages that need updating it's no big whoop.
My rights don't end where your fear begins.
(Score: 0) by Anonymous Coward on Friday April 21 2017, @02:13PM
Oh, thanks. I thought you could restrict the CPU utilization with distcc, but it's been a few years since I've used it. You could always "nice" the slave tasks low and the good stuff high, but then a slow slave task might bottleneck the whole compile so nevermind...
(Score: 2) by Techwolf on Friday April 21 2017, @12:15AM
I have been using gentoo for many years and have my own overlay. https://github.com/Techwolf/techwolf-overlay [github.com]
I use a chroot on a SSD drive to do my emerges and then exit the desktop to the command line and do an emerge -k upgrade using said binary that was built in the chroot. Another advantage of doing this is one can do an gcc upgrade by emerge gcc and then emerge world while still using the desktop for other things while the 20+ hours build runs in the background. This is for desktop, servers usually emerge world in an hour or two, depending on how many packages there is.
Another beneficent, one only emerges what is needed, no crapware install as depends to just that one program you need.
(Score: 2) by butthurt on Friday April 21 2017, @02:31AM (1 child)
Linode's backup services require that their system understand the underlying filesystem format used by the OS so they can snapshot it easily. As of writing, they do not support FreeBSD's UFS or ZFS.
It's ages since I tried to use it and I found it buggy when I did, but FreeBSD does support the Ext2 file-system.
https://wiki.freebsd.org/Ext2fs [freebsd.org]
From a glance at Linode's page about its backup service, I couldn't tell what file-systems it supports, but I assume that Ext2 is among them.
https://www.linode.com/backups [linode.com]
(Score: 2) by NCommander on Sunday April 23 2017, @09:34AM
It *only* supports ext2 and last I checked, not as the root filesystem. As such we can easily corrupt something in hard crash scenario or powerdown especially w.r.t. to the database servers/
Still always moving
(Score: 2) by canopic jug on Friday April 21 2017, @03:45AM (5 children)
Gentoo is a fine choice. I'm wondering why Devuan was not even mentioned in the evaluation, however. Was it looked at and found wanting? Or was it skipped? If so, why?
Money is not free speech. Elections should not be auctions.
(Score: 0) by Anonymous Coward on Friday April 21 2017, @06:13AM (2 children)
Because Devuan is dead.
Even if they finally manage to finish a release after 3(?) years, do you trust developers that take sometimes month, but at best a week to refresh their expired SSL certificate?
Really makes one wonder if it is not just "veterans" that hate learning (how to use SSL certs, how to use systemd, ...) and firmly believe "ignorance is strength", "less choice is more choice" (Debian offers more "init freedom" after all than Devuan).
(Score: 2) by Bot on Friday April 21 2017, @07:15AM (1 child)
> Debian offers more "init freedom"
Q: why?
troll: because it offers systemd
Q: I, too, once went to a bakery, had the baker prepare me a cake with no eggs, then asked him to put an egg on top.
troll: brother!!!
Account abandoned.
(Score: 2) by Bot on Friday April 21 2017, @07:18AM
BTW antix is without systemd and available now, cousin mx linux is with as few systemd as possible and available now.
Account abandoned.
(Score: 2) by The Mighty Buzzard on Friday April 21 2017, @10:42AM
Primarily because they don't have the manpower to be seriously considered on world-facing systems. Waiting a week or a month for a package update you need today is out of the question. I have nothing against hobbyist distros but they're just that and should not be used or even considered for critical production boxes.
My rights don't end where your fear begins.
(Score: 2) by NCommander on Friday April 21 2017, @10:44AM
It was discussed in the previous thread, but basically they're hacking up Debian and ignoring many of the systems in place that made Debian as stupidly rock solid as it was. One does not build a sane distribution out of a jenkin's instance. See my comments in the last thread for more details.
Still always moving
(Score: 0) by Anonymous Coward on Friday April 21 2017, @01:31PM (1 child)
You should've posted this on the 1st day..
(Score: 0) by Anonymous Coward on Friday April 21 2017, @01:54PM
AAHAHAhahaha--haha--ha-ah ... hey wait that's not funny.
(Score: 0) by Anonymous Coward on Saturday April 22 2017, @06:17AM (4 children)
Currently in the sidebar under Most Recent Journal Entries there is an entry that reads
[a-zA-Z0-9$_.+!*'(),-] hacking tools booter metaspolit gateways
However clicking that link only produces
The user you requested does not exist, no matter how much you wish this might be the case.
So, is he a deleted spammer or does his handle break the system? If former, then perhaps we could have a message telling that is the case instead of his journal and instead of this current 404 behavior.
(Score: 2) by The Mighty Buzzard on Saturday April 22 2017, @08:19PM (3 children)
That's our bad. We should not have allowed '+' in nicknames because of how browsers/servers handle it nowadays. Patch for it might be out as early as Monday since it's simple to reencode it as %2B.
My rights don't end where your fear begins.
(Score: 0) by Anonymous Coward on Sunday April 23 2017, @04:41PM (2 children)
Good to hear that, thanks for all the work! Love this amazing place.
(Score: 2) by The Mighty Buzzard on Monday April 24 2017, @04:18PM (1 child)
Fixed. It'll be in the next site update. Pretty soonish as there's only a minor math error to fix in subs left to code up for the next one and we'll be pushing it out.
My rights don't end where your fear begins.
(Score: 0) by Anonymous Coward on Wednesday April 26 2017, @06:56PM
And BAM it's alive. Great service! ++