Stories
Slash Boxes
Comments

SoylentNews is people

posted by NCommander on Monday August 08 2016, @12:00PM   Printer-friendly
from the now-with-a+-scores dept.

So after an extended period of inactivity, I've finally decided to jump back into working on SoylentNews and rehash (the code that powers the site). As such, I've decided to scratch some long-standing itches. The first (and easiest) to deploy was HSTS to SoylentNews. What is HSTS you may ask?

HSTS stands for HTTP Strict Transport Security and is a special HTTP header that signifies that a site should only be connected to over HTTPS and causes the browser to automatically load encrypted versions of a website should it see a regular URL. We've forbid non-SSL connections to SN for over a year, but without HSTS in place, a man-in-the-middle downgrade attack was possible by intercepting the initial insecure page load.

One of the big views I have towards SoylentNews is we should be representative of "best practices" on the internet. To that end, we deployed IPv6 publicly last year, and went HTTPS-by-default not long after that. Deploying HSTS continues this trend, and I'm working towards implementing other good ideas that rarely seem to see the light of day.

Check past the break for more technical details.

[Continues...]

As part of prepping for HSTS deployment, I went through every site in our public DNS records, and made sure they all have valid SSL certificates, and are redirecting to HTTPS by default. Much to my embarrassment, I found that several of our public facing sites lacked SSL support at all, or had self-signed certificates and broken SSL configurations. This has been rectified.

Let this be a lesson to everyone. While protecting your "main site" is always a good idea, make sure when going through and securing your infrastructure that you check every public IP and public hostname to make sure something didn't slip through the gaps. If you're running SSLLabs against your website, I highly recommend you scan all the subjectAlternativeNames listed in your certificate. Apache and nginx can provide different SSL options for different VHosts, and its very important to make sure all of them have a sane and consistent configuration.

Right now, HSTS is deployed only on the main site, without "includeSubdomains". The reason for this is I wanted to make sure I didn't miss any non-SSL capable sites, and I'm still working on getting our CentOS 6.7 box up to best-practices (unfortunately, the version of Apache it ships with is rather dated and doesn't support OSCP stapling. I'll be fixing this, but just haven't gotten around to it yet).

Once I've fixed that, and am happy with the state of the site, SN, and her subdomains will be submitted for inclusion into browser preload lists. I'll run an article when that submission happens and when we're accepted. I hope to have another article this week on backend tinkering and proposed site updates.

Until then, happy hacking!
~ NCommander

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 2) by Marand on Tuesday August 09 2016, @04:39AM

    by Marand (1081) on Tuesday August 09 2016, @04:39AM (#385629) Journal

    I don't think you can do it with Ubuntu, because they went 100% anti-sysvinit way back with upstart, but you can still stick with Debian and run it systemd-free. sysvinit is still there, along with some other alternate init systems, and you can choose which one you want just like you could before. They just changed the default to systemd, but there are still people in Debian that care about supporting other inits (like this guy [simonrichter.eu]), so hopefully the support will continue for a long time.

    Rather than switch to an entirely different OS, this would probably be a better route. Like another user said, there should be little (if anything) on a server that will depend on systemd, so switching OSes over it is a bit excessive. It's more of an issue for desktop use because of how much of the desktop is getting caught up in it, even outside of GNOME.

    I have a question, though: why did you even use Ubuntu for the server in the first place? I keep seeing people use it and I just don't understand that, what exactly does Ubuntu bring to the table for a server that Debian doesn't? Its release cycle is silly, it's more likely to have dist-upgrade problems (though it IS still pretty solid for it, I've found Debian to be more reliable), and most of the work Canonical does seems to be aimed at the desktop user. This is a real question; I've never understood the appeal of Ubuntu for server-use and I'm curious what the logic behind it is.

    Though, on a more ranty note, if you have a problem with systemd, why the hell would you have used Ubuntu at all? Canonical was doing the exact same shit pushing upstart on people before it was production ready, and trying to replace battle-tested init systems with their own in-house NIH bullshit. Their devs also were just as involved in the anti-sysvinit push that got Debian to change its default; both the upstart and systemd camps were shouting down anybody that attempted to suggest other alternatives (including keeping sysvinit). After both camps successfully shut out everything else, then upstart lost, but Canonical and upstart is as much at fault for Debian's systemd status as anybody else. Sure, I'd have preferred upstart over systemd based on what I know of the two, but the reality of it is they both pushed this shit too soon, and they did it together.

    Starting Score:    1  point
    Karma-Bonus Modifier   +1  

    Total Score:   2  
  • (Score: 2) by NCommander on Tuesday August 09 2016, @10:40AM

    by NCommander (2) Subscriber Badge <michael@casadevall.pro> on Tuesday August 09 2016, @10:40AM (#385709) Homepage Journal

    Um, sysvinit is old to the point of needing replacement. No one has seriously argued it was getting long in the tooth, it was an argument on how to replace it. SPecifically, here are the issues

      * No dependency tracking, which means you're only method of controlling startup are Sxx numbers
      * Can't paralizared startup of services
          * Related, one hung service can hang the boot
      * No standard; Debian, Red Hat, and other OSes had subtly incompatible sysvinit semantics making writing a cross platform init script a bitch

    Short list, but points 1+2 make reboot times minutes long. You can complain that "reboot times" don't matter, but end users disagree, and its nice not to take 30 minutes to reboot a server.

    I actually worked at Canonical and was involved in some of the upstart discussions as well as having drinks with Keybuk multiple times. Upstart was deployed in a non-LTS branch for testing a year and a half before the next LTS (it went in Intrepid if I remember correctly). Furthermore, upstart is an init system, and all it does is load its unit files, and start processes. It doesn't hijack most of the system startup, or replace logs or whatever. It was fairly drop in replacement to the point Debian had shipped it for quite awhile. You'll notice that on my rant list, the only rant I have about startup behavior with systemd is it doesn't fail safely on malformed unit files. It's everything else I can't stand.

    --
    Still always moving
    • (Score: 2) by Marand on Tuesday August 09 2016, @07:56PM

      by Marand (1081) on Tuesday August 09 2016, @07:56PM (#385928) Journal

      Um, sysvinit is old to the point of needing replacement. No one has seriously argued it was getting long in the tooth, it was an argument on how to replace it. SPecifically, here are the issues

      Not really inclined to agree that "it's old so we need to repalce it" is a good argument, especially when distros had still been managing to keep it usable. But that's just a difference of opinion.

      * No dependency tracking, which means you're only method of controlling startup are Sxx numbers

      This is an outdated claim. Dependency-based boot in Debian's sysvinit has been available nearly ten years, and the default for almost that long. OpenRC has been available longer than that.

      * Can't paralizared startup of services

      This is also an outdated claim. Parallelised boot (which I'm guessing you meant) has been available in Debian's sysvinit for 6-7 years. Not sure about OpenRC's status for it or other distros.

      * Related, one hung service can hang the boot

      Unless the rest of the boot process depends on it, this should not occur with the dependency-based bootup. Which means this, also, is an outdated claim.

      * No standard; Debian, Red Hat, and other OSes had subtly incompatible sysvinit semantics making writing a cross platform init script a bitch

      The only potentially valid complaint so far. More of an issue for package maintainers, though, so not really on-topic with regard to the rest of it. Also kind of a funny argument considering Canonical and Redhat both decided to add even more init systems instead of working with existing alternatives. Not really helping the cross-platform init problem there.

      Also, that argument actually favours systemd, because now there is a cross-platform standard, whereas nobody but Ubuntu used upstart.

      Short list, but points 1+2 make reboot times minutes long. You can complain that "reboot times" don't matter, but end users disagree, and its nice not to take 30 minutes to reboot a server.

      No, I'd argue (and did) that the first three points you made are extremely out of date and seem to indicate you haven't really looked at anything but Ubuntu in years, if ever. Which is fine, except that you're considering switching to a BSD based on Linux complaints while apparently not really knowing the landscape outside of Ubuntu.

      Furthermore, if you think sysvinit is outdated and that means it's bad, you're in for a rude awakening with FreeBSD [freebsd.org]. You may even hit some of the problems you (mistakenly) declared for sysvinit, though I'm not certain about that.

      I actually worked at Canonical and was involved in some of the upstart discussions as well as having drinks with Keybuk multiple times. Upstart was deployed in a non-LTS branch for testing a year and a half before the next LTS (it went in Intrepid if I remember correctly)

      You never did answer my question about what advantage there is to deploying Ubuntu servers instead of going with Debian -- which was a serious question, FYI, and I'd hoped for a real answer. Right now the lack of answer + mention of working at Canonical makes it look like a company loyalty thing more than any real decision based on merits. I'd love a real answer though. I'd guess support contracts would be a valid reason, but I doubt SN's paying for that.

      Furthermore, upstart is an init system, and all it does is load its unit files, and start processes. It doesn't hijack most of the system startup, or replace logs or whatever.

      Yep, like I said in the other comment, I'd have preferred to see upstart "win" rather than systemd, and that's a big part of why. I never really had any problems with upstart itself, other than Canonical completely removing alternative inits to force use of upstart. I'm kind of sad that Canonical abandoned it, because it's not like they're afraid of being incompatible...what with Mir, upstart, Unity, etc. Oh well.

      Still, my big issue wasn't with upstart, it was a social problem: I didn't like how upstart+systemd proponents both forced the issue in Debian, and argued down anybody suggesting something that wasn't upstart or systemd.

      It was fairly drop in replacement to the point Debian had shipped it for quite awhile.

      It's still available in Debian stable. I could switch to it right now if I wanted. It does looks like it's been dropped from unstable/testing, though, probably because Canonical abandoned it. Shame, because other inits are still there. Even with Upstart gone, there's still openrc, file-rc, runit, sysv-rc, and maybe more I'm not aware of, in addition to the systemd default. There are still maintainers that care about having a choice, and they're keeping those inits working and available.

      See, unlike Ubuntu, Debian actually kept maintaining packages for those init systems. In fact, Ubuntu should still have them just by virtue of being a Debian-derived distribution, but someone decided to purge non-upstart inits from Ubuntu. Just like Redhat, Canonical preferred to take away the choice and force their own in-house solution on people, which is unfortunate.

      So, again, I'll suggest taking a look at other options, such as Debian, before making a knee-jerk reaction against systemd and Linux as a whole and running off to FreeBSD. You're going to have an "old" type of script-based init there, too, and there's no requirement to run systemd in Debian, so it'd probably be a better transition. You just seem to be running off old knowledge and bad assumptions about the state of non-Ubuntu Linux.

      • (Score: 2) by NCommander on Wednesday August 10 2016, @02:00AM

        by NCommander (2) Subscriber Badge <michael@casadevall.pro> on Wednesday August 10 2016, @02:00AM (#386068) Homepage Journal

        You do realize I'm a Debian Developer in addition to an Ubuntu Core Dev, right?. Anyway, my point is sysvinit has issues, and was due for either update and replacement. I thought upstart good replacement for technical reasons. The primary reason it didn't get adopted by Debian was political most over Canonical's copyright assignment policy; I remember it came up several times on d-private.

        * Not really inclined to agree that "it's old so we need to repalce it" is a good argument, especially when distros had still been managing to keep it usable. But that's just a difference of opinion.

        Fine. Let's get into the deep technical flaws with sysvinit.

        This is an outdated claim. Dependency-based boot in Debian's sysvinit has been available nearly ten years, and the default for almost that long. OpenRC has been available longer than that.

        When you refer to dependency tracking, I assume you're referring to this block at the start of the file:

        ### BEGIN INIT INFO
        # Provides: scriptname
        # Required-Start: $remote_fs $syslog
        # Required-Stop: $remote_fs $syslog
        # Default-Start: 2 3 4 5
        # Default-Stop: 0 1 6
        # Short-Description: Start daemon at boot time
        # Description: Enable service provided by daemon.
        ### END INIT INFO

        Debian's dependency tracking (which is based on LSB) is a serious hack, and one I am aware of. When I was saying sysvinit has no concept of dependencies, I was referring to the daemon itself.

        update-rc.d de-facto works by dynamically ordering the scripts as part of update-rc.d. Boot dependencies are statically specified, and from the original thread where this was implemented, Tollef Fog Heen [debian.org] provides an example of why this is a bad thing. The system is incredibly brittle because it can't resolve dependency loops or the case of something failing to start. The reason it all appears to work fine is considerable effort is put in by DDs to keep sysvinit chugging along.

        This is also an outdated claim. Parallelised boot (which I'm guessing you meant) has been available in Debian's sysvinit for 6-7 years. Not sure about OpenRC's status for it or other distros.

        I actually had to go look this up and it turns out you're right. It's been in unstable for years, but it didn't actually ship until Debian wheezy (2013). Debian reference manual [debian.org]. I didn't switch back to Debian as my primary OS until jessie so I completely missed this.

        Unless the rest of the boot process depends on it, this should not occur with the dependency-based bootup. Which means this, also, is an outdated claim.

        See previous point on "dependency" based startup on sysvinit. The fact is sysvinit has no mechanism to detach a process on boot beyond standard shell control. This generally isn't a problem for servers which can wait for a slow process to get going (and generally want to), but it is problematic on desktops for getting short startup times. Basically, in a sysvinit world, you can either wait for a init script to finish entirely, or detach and clear that boot item. Once you detach, you have no way of if a service has fully come up, or is still starting in a generic way.

        A big focus is from kernel start to GDM in about 10 seconds. The system can keep booting beyond that, but you need enough up to get X11 fired up and you need a way to signal that its safe to run Xsession (which generally requires other system daemons be loaded).

        Upstart handed this via a socket and dbus. systemd has an entire event hook API (https://www.freedesktop.org/software/systemd/man/sd_notify.html)

        You never did answer my question about what advantage there is to deploying Ubuntu servers instead of going with Debian -- which was a serious question, FYI, and I'd hoped for a real answer. Right now the lack of answer + mention of working at Canonical makes it look like a company loyalty thing more than any real decision based on merits. I'd love a real answer though. I'd guess support contracts would be a valid reason, but I doubt SN's paying for that.

        I actually missed this point. I wanted a Debian-based OS because of bad experiences with Red Hat ones. Architecturally, Debian has very strict constrains on library dependencies and preventing skew which Ubuntu inherited. (man dpkg-shlibs) combined with a very well tested in-place upgrade system. I would have been perfectly fine if we had standardized on Debian stable at the time. I was against using CentOS due to the fact that its notorious for shipping dated software combined with a very high chance of breaking everything if you upgrade in place. Ultimately, we split the difference, that the rehash machines would run ubuntu, and we'd run CentOS elsewhere. In practice, after the one person really pushing for CentOS left, everything new became Ubuntu to standardize.

        In contrast, with one minor hiccup, the upgrade from Ubuntu 12.04->14.04 was a non-event. Beryllium (the CentOS box) will get wiped and reinstalled to match everything else. What we may be running on is unknown at this time.

        What ultimately made the decision was that when I setup the site, I installed Ubuntu because I done the core work on my laptop which also ran Ubuntu so I didn't have to worry about library skew or anything.

        Yep, like I said in the other comment, I'd have preferred to see upstart "win" rather than systemd, and that's a big part of why. I never really had any problems with upstart itself, other than Canonical completely removing alternative inits to force use of upstart. I'm kind of sad that Canonical abandoned it, because it's not like they're afraid of being incompatible...what with Mir, upstart, Unity, etc. Oh well.
        Still, my big issue wasn't with upstart, it was a social problem: I didn't like how upstart+systemd proponents both forced the issue in Debian, and argued down anybody suggesting something that wasn't upstart or systemd.

        I'm not sure how much I can actually say on this due to most of the Debian side being discussed on d-private. I can say the primarily reason upstart didn't go anywhere was political vs technical. What I can say, on the Ubuntu front, Mark Shuttleworth himself made the call we were going to systemd because Debian decided to do it and we couldn't justify the engineering resources.

        As for the rest you mentioned, well, I'll just say I left Canonical for many reasons.

        See, unlike Ubuntu, Debian actually kept maintaining packages for those init systems. In fact, Ubuntu should still have them just by virtue of being a Debian-derived distribution, but someone decided to purge non-upstart inits from Ubuntu. Just like Redhat, Canonical preferred to take away the choice and force their own in-house solution on people, which is unfortunate.

        Ubuntu has sysvinit, upstart fired it up at the end of boot processing to run the legacy scripts. You are correct that Ubuntu had a hard dependency on booting with upstart as I believe we had hooked it into our udev. I generally didn't have much issue with this as upstart did one thing, and did it well, vs. systemd's kitchen sink approach. Debian's support for other init systems to my knowledge is at best spotty in the real world though ...

        So, again, I'll suggest taking a look at other options, such as Debian, before making a knee-jerk reaction against systemd and Linux as a whole and running off to FreeBSD. You're going to have an "old" type of script-based init there, too, and there's no requirement to run systemd in Debian, so it'd probably be a better transition. You just seem to be running off old knowledge and bad assumptions about the state of non-Ubuntu Linux.

        sysvinit had valid technical reasons to be replaced or overhauled. I just think the new 'standard' is a pile of crap. I posted my long list of technical reasons why I hate systemd elsewhere in this thread. If systemd was content to stay on desktop Linux, I probably won't care much; sysvinit is fine for servers for the most part and I've never had a serious issue with it in real life in a server-role. If systemd did one thing and did one thing well (aka, if it had limited itself to being an upstart competitor), I'd also probably not care enough to work up a damn. It's the entire thing combined with the tendrils its run through the stack that makes me throw my hands up and say enough is enough and look at getting off this ship.

        I'm well aware that the boot situation on FreeBSD is extremely primitive in comparison. You have rc.conf and rc.local and that's about it. Everything is sanely logged. For a server, that's more than acceptable.

        • (Score: 2) by Marand on Wednesday August 10 2016, @05:37AM

          by Marand (1081) on Wednesday August 10 2016, @05:37AM (#386134) Journal

          You do realize I'm a Debian Developer in addition to an Ubuntu Core Dev, right?. Anyway, my point is sysvinit has issues, and was due for either update and replacement. I thought upstart good replacement for technical reasons. The primary reason it didn't get adopted by Debian was political most over Canonical's copyright assignment policy; I remember it came up several times on d-private.

          Had no idea about you being a DD, since I only saw mention of Canonical work in the other comment. You're as anonymous as anybody else here to me. :p

          And yeah, the copyright assignment thing sucked, I recall there being some grumpiness over it in the past, so that's not surprising at all. What I meant though was that in the open discussions about the future of Debian's default init, upstart+systemd people (including some devs on both sides) seemed keen on shutting down any attempt to suggest anything else. People were determined to make it a two-party vote, and once that happened Upstart lost to politics like you said.

          As for the sysvinit thing, I'm aware it's a script-based hack and has its issues, but it still does the job mentioned, so I took your statement as a literal "it can't do this thing it does". It's not hard to imagine someone would miss a change like that in a completely different distro, so I took the remark at face value. Just a communication error there: you were being more literal in referring to init itself while I was interpreting it more broadly.

          Also, thanks for the answer about the server selection. I'm with you on the general aversion to Redhat-derived distros, and preferring Debian-based distros for stability and the upgrade process. It's why I've stuck with Debian since around 2000, and why I keep ending up coming back to it after experimenting with other options occasionally. I've mentioned it here before, but my primary OS is a Debian install that started on 2.2 (potato) and has been dist-upgraded through the years, with migrations to new hardware, new disks, and even updated in-place from 32bit to 64bit. I don't know any other OS or distro that I could have done this with. I probably should just reinstall it one day, but it amuses me to keep it going instead of starting fresh at this point. :)

          Vaguely related distro talk: I keep intending to spend time with NixOS. I already get a lot of use out of Nix as a secondary package manager and really like its (admittely weird as hell) filesystem layout and philosophies, so it's on my radar but I haven't gotten around to it. It starts out with systemd though, bleck. Does seem to have other options (sysv, upstart, runit) but I have no idea how much trouble it would be to switch out.

          At this point, my take on systemd is I'm mostly avoiding it for now. sysvinit and the patchwork of hacks on top of it may not be the way to go long-term (I agree about it being due for update or replacement, I just think it all happened a bit prematurely, at least for Debian), but I'd still rather use it than systemd right now. Like you said, if it were just a desktop thing, it wouldn't be such a big deal. I haven't had any problems with the "higher" parts of systemd; it's not really any better or worse than dealing with, say, HAL was in that regard. (I remember HAL being a nightmare at times early on...)

          It's just that I don't want systemd as my init, so I stick to systemd-shim and running anything else underneath. The init just happens to be sysv because that's what Debian used before and it just worked, but it's not like I'm not open to alternatives. Just preferably alternatives that aren't systemd's init. Hopefully by the time I don't have a choice any more, Poettering and Sievers will have gotten bored and moved on so others can clean house. It mostly worked for Pulseaudio like that, at least...

          On a related note, have you ever used runit for anything? It sounds interesting but I've never gotten around to spending time with it. I've only heard of one distro (Void Linux) using it by default [voidlinux.eu], but their page on how it works is intriguing. It also appears you can even use it alongside another init in case you only want to manage certain services with it, which sounds like it could be appealing.

    • (Score: 2) by Justin Case on Wednesday August 10 2016, @02:26AM

      by Justin Case (4239) on Wednesday August 10 2016, @02:26AM (#386078) Journal

      No dependency tracking, which means you're only method of controlling startup are Sxx numbers

      Has never caused any problems for me. Which of course I'm not arguing it hasn't been a problem for others, but that's what I love about a non-proprietary OS: choice!

      Can't paralizared startup of services

      Don't want to.

      one hung service can hang the boot

      As it should. Fix it before proceeding.

      reboot times minutes long. You can complain that "reboot times" don't matter, but end users disagree, and its nice not to take 30 minutes to reboot a server.

      Again, not a problem for me. I'm not an "end user" and I want to barf every time someone forces me to use software written for the ignorant and lazy, because I am neither. Oh, and if something isn't right, I don't want it to take 30 minutes to boot a server, I want the server to not boot at all. That's an incredibly effective technique for making sure the problem gets noticed, prioritized, and fixed, instead of "oh sure let's proceed down the runway even though one jet engine didn't start..."

      BTW please don't take any of this as a personal attack. Love the work you're doing, and praise for doing "the right thing" with https etc.