SoylentNews is people
SoylentNews
So after an extended period of inactivity, I've finally decided to jump back into working on SoylentNews and rehash (the code that powers the site). As such, I've decided to scratch some long-standing itches. The first (and easiest) to deploy was HSTS to SoylentNews. What is HSTS you may ask?
HSTS stands for HTTP Strict Transport Security and is a special HTTP header that signifies that a site should only be connected to over HTTPS and causes the browser to automatically load encrypted versions of a website should it see a regular URL. We've forbid non-SSL connections to SN for over a year, but without HSTS in place, a man-in-the-middle downgrade attack was possible by intercepting the initial insecure page load.
One of the big views I have towards SoylentNews is we should be representative of "best practices" on the internet. To that end, we deployed IPv6 publicly last year, and went HTTPS-by-default not long after that. Deploying HSTS continues this trend, and I'm working towards implementing other good ideas that rarely seem to see the light of day.
Check past the break for more technical details.
[Continues...]
As part of prepping for HSTS deployment, I went through every site in our public DNS records, and made sure they all have valid SSL certificates, and are redirecting to HTTPS by default. Much to my embarrassment, I found that several of our public facing sites lacked SSL support at all, or had self-signed certificates and broken SSL configurations. This has been rectified.
Let this be a lesson to everyone. While protecting your "main site" is always a good idea, make sure when going through and securing your infrastructure that you check every public IP and public hostname to make sure something didn't slip through the gaps. If you're running SSLLabs against your website, I highly recommend you scan all the subjectAlternativeNames listed in your certificate. Apache and nginx can provide different SSL options for different VHosts, and its very important to make sure all of them have a sane and consistent configuration.
Right now, HSTS is deployed only on the main site, without "includeSubdomains". The reason for this is I wanted to make sure I didn't miss any non-SSL capable sites, and I'm still working on getting our CentOS 6.7 box up to best-practices (unfortunately, the version of Apache it ships with is rather dated and doesn't support OSCP stapling. I'll be fixing this, but just haven't gotten around to it yet).
Once I've fixed that, and am happy with the state of the site, SN, and her subdomains will be submitted for inclusion into browser preload lists. I'll run an article when that submission happens and when we're accepted. I hope to have another article this week on backend tinkering and proposed site updates.
Until then, happy hacking!
~ NCommander
(Score: 0) by Anonymous Coward on Monday August 08 2016, @12:23PM
NCommander,
All the security checking is great. However, when I clicked on this post (while logged in), to read your full text, it logged me out. And gave me the default red-bar color scheme.
While the login box displays, it wouldn't log me in as long as I'm viewing this article, "SoylentNews Deploys HSTS and Mandates HTTPS Everywhere". Tried twice to make sure,
As soon as I go back to the home page, I can log in again....
Update, after I clicked Preview (only option when not logged in), I appear to be logged in again and my nice VT100 green screen re-appeared! Now I have the normal choices (including check box for posting AC).
Weird?
(Score: 2) by NCommander on Monday August 08 2016, @12:30PM
That's extremely unusual esp. given your post popped up as AC. We haven't changed anything except the HSTS header. It's possible your login cookies are screwed up or you're getting kicked into the varnish cache.
Can you try doing a full log out and log in to see if it clears it, and post your SN username so I can check the logtokens table.
Still always moving
(Score: 2) by NCommander on Monday August 08 2016, @12:31PM
Update: did you intentionally post that as AC? I scanned your SUBID (hash of your IP address), and I see one account that has modded recently, and an AC that posted recently.
Still always moving
(Score: 2) by carguy on Monday August 08 2016, @12:44PM
I did intentionally post "side effects?" as AC -- although during the weird behavior, it was looking at first like I would have to be AC (not logged in).
Took your advice, logged out, cleared the Soylent cookies from Firefox (two of them...) and logged back in.
Appears to be working normally now. Thanks for the quick reply!
(Score: 0) by Anonymous Coward on Monday August 08 2016, @12:29PM
Thanks for your work!
(Score: 3, Informative) by Anonymous Coward on Monday August 08 2016, @12:40PM
Ever consider migrating to FreeBSD?
I have had very good luck with FreeBSD. You can set default versions and libraries via /etc/make.conf. ZFS clones + jails gives you the perfect testing environment with little overhead that can be spun up in a few minutes. The package management system will tell if you something is vulnerable as soon as it is known, rather than when an updated package is out. If you go the ports route, it even adds an extra hoop you have to jump through to install a package with known vulnerabilities. I have done a fair number of in place upgrades of RHEL 6-7 based OS's and found that it is usually a huge pain because a lot of packages from the old major release stick around and need manual removal and upgrading. FreeBSD's in place upgrades have always worked fairly well for me. Getting a gazillion prompts for how to handle each non-stock config file is tedious though. Reinstalling all packages is also a pain, but with ZFS, if you do the upgrade on a test system first, you will keep your downtime down to a reboot or two and however long it takes to do a zfs send | ssh zfs receive of /usr/local. This applies to databases as well. As long as your data files are on their own filesystem so they won't get clobbered, the zfs send/receive works just fine.
After administering Linux (Debian and RHEL based distros), HP-UX, Solaris, and FreeBSD, I would say my favorites are a tie between Solaris and FreeBSD, with Debian being a close second. SE Linux is nice, but it isn't a magic bullet and requires admins that understand it well, which isn't as common as it should be.
(Score: 5, Interesting) by NCommander on Monday August 08 2016, @01:04PM
We've discussed it. Most of the boxes are Ubuntu 14.04 with the single CentOS 6 box lurking. I liked Ubuntu (and Debian) as I've had in-place updates never go castrophically wrong before; my desktop was installed with Ubuntu 12.10, and had upgraded through every intervening release since without once requiring a reinstall. CentOS is something of a joke you can't in-place upgrade.
That being said, none of the staff have been very thrilled with Linux as of late, and my recent experiences with both systemd and Ubuntu 16.04 tell me to go "fuck it" and replace it with something not brain damaged. FreeBSD seems to be the popular option, though illumos is also been considered (though its service management system has made me kinda "eh" since its very similar to systemd though it hasn't tendralled through the entire stack yet).
Still always moving
(Score: 0) by Anonymous Coward on Monday August 08 2016, @01:29PM
I'm not real thrilled with Linux lately either. Debian with systemd has worked ok for me, although I did run into some braindead behavior wear a machine wouldn't boot into a usable state due to some nonsense with mdadm due to braindead target dependencies.
I really don't like what Red Hat did with 7. The upgrade package in the repo doesn't work. Rebuild the source rpm from: https://git.centos.org/summary/rpms!redhat-upgrade-tool.git. [centos.org] The one in the repo listed on: http://dev.centos.org/centos/6/upg/x86_64/Packages/ [centos.org] is terribly out of date. The upstream packaged was fixed to work good enough for a decent admin to get the job done. Make sure that /var/run ends up symlinked to /run or systemd will lose its mind if you try to restart some services. I have not done this on CentOS, just the upstream RHEL and Oracle Linux. Oracle Linux uses a dated update tool not quite as dated as the CentOS tool that doesn't resolve package versions correctly where it sees a package with a higher version ending with el6 as being newer than el7. It is possible to manually fix all of those, but it is a huge pain. For example if RHEL6 has grep 1.2.3.el6 and RHEL 7 has grep 1.2.2.el7, the tool will think the 1.2.3.el6 is newer and stick with that. You have to reinstall ALL packages after the upgrade with the dated tool because the scripts within the packages complain about missing libraries, but still somehow successfully upgrade. Don't even get me started on firewalld (I use the old style rules files). The only reason I am using 7 in my org is young projects that require RHEL and that will be hard to upgrade or migrate, so the longer availability of security patches is the sensible thing to do. I wish they would have left grub alone, the new config file style is convoluted in my opinion. It appears that Red Hat is trying to make an OS that is for the server and a laptop, and fails at both because it is like designing a vehicle to be a gas sipping commuter that can pull a big rig's trailer.
(Score: 4, Insightful) by NCommander on Monday August 08 2016, @02:58PM
Red Hat is trying to make Red Hat be a poor knock off of Windows. It was never a good OS compared to Debian in terms of cohensively or sanity, and now it just flat out sucks. Right up until Debian jumped on the systemd train, it, and its descendants were by far one of the best experiences you could get on Linux.
Still always moving
(Score: 0) by Anonymous Coward on Monday August 08 2016, @03:07PM
All it takes is several wasted hours to seriously hate systemd
Mine turned into several days worth of time in the end
(Score: 3, Interesting) by Scruffy Beard 2 on Monday August 08 2016, @03:24PM
So far I have seen weirdness, but have not been bitten in the *ss.
I finally installed systemd on my old debian machine. The disk check caused the boot to fail (with a scary looking error) instead of simply resuming after waiting.
I should be getting my new FreeBSD install working instead of posting on the Internet.
(Score: 2) by zeigerpuppy on Monday August 08 2016, @06:31PM
I've found that pinning systemd on Debian works pretty well. All
my Debian machines run headless so there doesn't appear to be much need for systemd. I'm mostly running a wheezy/unstable hybrid on most machines which works well when you need apache 2.4, reasonably sane versions of node etc.
I also use ZFS extensively, it's awesome but I'm sure there's a performance hit in ZFSonLinux vs native on BSD.
(Score: 2) by Dr Spin on Monday August 08 2016, @01:33PM
I have also tried Mint - same results.
Why do the distro merchants think removing features is an idea with any merits at all?
And why does Linux /Gnome have millions of settings which are cunningly hidden, or just not accessible from the GUI?
and why can you view the print queue, but not delete jobs without using the command line?
Why do you need to use the command line to edit permissions of things in /dev/ to install a printer, and why does the error message you get when the new install fails to work, not mention permissions or /dev?
I believe these problems may be connected to the fact that some people are actually willing to use Windows! (I am not one of those).
Maybe it will be FreeBSD for me too.
Warning: Opening your mouth may invalidate your brain!
(Score: 2) by Scruffy Beard 2 on Monday August 08 2016, @03:28PM
The most common source of problems is user error.
I speculate that they feel that by removing settings/options few people use, they remove opportunities for error.
Of course, that theory relies on a level of attention to detail not present in the software industry.
(Score: 3, Informative) by canopic jug on Monday August 08 2016, @01:34PM
If you're comfortable with Debian then you could stay close to it with Devuan. It is basically a port of Jessie, for the time being, without systemd. It is a drop-in replacement minus the systemd crap.
Or if you really want Debian then there is Debian GNU/kFreeBSD which is pretty much all the same as Debian GNU/Linux for all the user-land activity but, again, minus the systemd crap.
Money is not free speech. Elections should not be auctions.
(Score: 5, Interesting) by NCommander on Monday August 08 2016, @02:44PM
I've looked at Devuan in the past and its a joke. They're rebuilding the entire archive out of a Jenkins instance using a based up version of git and lots of other stuff. That breaks a lot of shit like shlibs tracking, change metadata, etc. It also means they can't reasonably track any packages that are in non-git VCS (which are most of them) If you're going to build Debian, do it fucking right, don't kitbash you way to something else.
I've talked with them about it but was completely blown off. I saw a few other DDs have poked their head in and got their nose cut off for it. Debian kFreeBSD is really not a direction I'd like to go in. They shipped glibc instead of BSD libc, which causes all sorts of application compat problems so interesting build failures are common, and you get all sorts of bloat you don't want or need on FreeBSD.
Still always moving
(Score: 2) by canopic jug on Monday August 08 2016, @03:49PM
Money is not free speech. Elections should not be auctions.
(Score: 2) by NCommander on Monday August 08 2016, @03:56PM
I'm leaning far more towards FreeBSD. OpenBSD doesn't have a MAC framework. I've also found of the two, OpenBSD tends to be more difficult to get to run under hosted virtualization (aka Linode), and while its network tools (3 pf) are great, everything else falls kinda flat. Great respect for Theo, but OBSD simply lacks what we need.
We run Apache under AppArmor to prevent unexpected stupidity dating back to when we ran Apache 1.3 due to slashcode being tied to that release before I took a nuclear blowtorch to it. If you could somehow manage to get a remote shell due to a flaw in rehash, you'd be limited to a very small amount of files that Apache+rehash need to access to run, and the inability to run anything else. Not great, but much better than having unrestricted access to the filesystem as the slash user and migates many attempts to elevate to root.
Still always moving
(Score: 2) by canopic jug on Monday August 08 2016, @04:05PM
Ok. Thanks. It will be very interesting to see which way you go in the future.
The team is doing a fantastic job running the site and the infrastructure. Plus, being able to follow along somewhat with these updates is worth gold and, in my opinion, keeps with the stated nature of the site.
Money is not free speech. Elections should not be auctions.
(Score: 1) by pTamok on Monday August 08 2016, @06:07PM
That is a shame. I was hoping Devuan was going to be a way for GNU/Linux to avoid the systemd mess. A BSD is beckoning. Debian voting to embrace systemd seems to have been a watershed.
(Score: 4, Interesting) by schad on Monday August 08 2016, @02:18PM
SMF isn't as bad as systemd at all. After a brief adjustment period, I viewed it as "really just init, but slightly better in many ways."
If you actually have to create SMF service manifests, though, you may hate your life for a while. Lots of XML boilerplate. Basically all the complexity of service management is front-loaded. While advanced services become much easier to develop because SMF takes care of a lot of it for you, simple services become a lot more cumbersome. Still, it's really only a chore you have to do once. After that, you're just tweaking a setting here or there, just like you would with any other system.
SMF is a lot like IPS (the packaging system used by OpenSolaris). It's better in some ways and worse in others. I personally like it, but even I don't consider it a huge improvement over existing technologies. That's in contrast to, for example, ZFS.
But there's a certain mentality common to everything introduced in Solaris 10. If you happen to share that mentality -- which I do -- then you'll like stuff just because it "makes sense," no matter its other pros and cons. If you don't, I'm told that Solaris can be an intensely frustrating experience. So take my opinion with a grain of salt.
(Score: 2) by darkfeline on Monday August 08 2016, @03:25PM
What issues specifically have you had with systemd?
Join the SDF Public Access UNIX System today!
(Score: 5, Informative) by NCommander on Monday August 08 2016, @03:51PM
Off-hand, here's what I've run into on multiple systems:
* journald corruption on a semi-regular basis
* Recovery is non-trivial and generally requires deletion of the db files
* Can happen almost without fail on some machines on a hard restart
* bad service start announcements; i.e. I can do systemctl *unit* start, and get a false start notice for example
* Which requires a trip to journald to see what happened
* journalctl sucks if you're trying to search for something and doesn't handle rotation cleanly.
* which means if you've got a service which is non-chatty, you could see weeks or older log messages
* No ease of seperation between standard out logs, and stderr.
* No way to filter on stderr output only.
* Unit files don't fail on typo. If you somehow manage to type Userr=nobody, it will silently run your process as root and not even pop a warning
* Related. I've seen systemd race with nss_{ldap|hesiod}, where it tries to enumerate users before the network goes up, and craps itself or simply ignores the User= settings. Unaware if its fixed.
* No easy way to parse in a unit-specific config. With both upstart and sysvinit, I could put a file in /etc/defaults, and load that into the unit to successfully seperate how I want to start a daemon and how to configure it. For example, varnish's setup has to go right into the unit file. This becomes problematic when ubuntu updates varnish and I have to manually merge the units together.
* (Ubuntu specific) - service X action doesn't print success/fail messages. Plenty of scripts still depend on the old symantics
* Historically: very very buggy, and I've had deadlocks.
* At least with recent CentOS, this hasn't been a real issue, but I don't like it when the "massive daemon everything needs" goes belly up
* cgroups are required (problematic in some virtualization scenarios; not an issue for KVM/Xen. Big issue with LXC).
* Actively hostile upstream with no concern for any usecases beyond theirs.
That's off the top of my head. I can go more indepth if need be.
Still always moving
(Score: 1, Informative) by Anonymous Coward on Monday August 08 2016, @07:27PM
Gentoo is still free of systemd. I've been using it for Linux based fileservers and routers sucessfully for years.
(Score: 2) by vux984 on Tuesday August 09 2016, @02:38AM
That being said, none of the staff have been very thrilled with Linux as of late
Heh. Feels like dumping the baby out with the bathwater to make the change for change sake.
(Score: 2) by NCommander on Tuesday August 09 2016, @10:45AM
It's more the straw that broke the camels back to be honest.
From a kernel perspective, I've never really liked Linux over the BSDs. The code is complicated, bloated inplaces, and has an absolutely nasty-to-work-with upstream. Going on LKML is like asking to be shot if you do slightly out of line. I get Linus's view, but most people I know only contribute to the tree because they're paid to do so, not because they want to.
Moving further down the stack, with journald and systemd corrupting the lower bits of the userland, system stabilize seems to have gotten a lot worse. Now yes, you can run Debian without systemd (Debuvan), and a few distros, like Gentoo or SLackware haven't caved. However, I won't run a rolling release on a production server, and I have seen no indication that Slackware has no intention to add systemd. In addition, Slack is historically fragile on upgrades, and its package management system leaves a lot to be desired ...
Still always moving
(Score: 2) by Marand on Tuesday August 09 2016, @04:39AM
I don't think you can do it with Ubuntu, because they went 100% anti-sysvinit way back with upstart, but you can still stick with Debian and run it systemd-free. sysvinit is still there, along with some other alternate init systems, and you can choose which one you want just like you could before. They just changed the default to systemd, but there are still people in Debian that care about supporting other inits (like this guy [simonrichter.eu]), so hopefully the support will continue for a long time.
Rather than switch to an entirely different OS, this would probably be a better route. Like another user said, there should be little (if anything) on a server that will depend on systemd, so switching OSes over it is a bit excessive. It's more of an issue for desktop use because of how much of the desktop is getting caught up in it, even outside of GNOME.
I have a question, though: why did you even use Ubuntu for the server in the first place? I keep seeing people use it and I just don't understand that, what exactly does Ubuntu bring to the table for a server that Debian doesn't? Its release cycle is silly, it's more likely to have dist-upgrade problems (though it IS still pretty solid for it, I've found Debian to be more reliable), and most of the work Canonical does seems to be aimed at the desktop user. This is a real question; I've never understood the appeal of Ubuntu for server-use and I'm curious what the logic behind it is.
Though, on a more ranty note, if you have a problem with systemd, why the hell would you have used Ubuntu at all? Canonical was doing the exact same shit pushing upstart on people before it was production ready, and trying to replace battle-tested init systems with their own in-house NIH bullshit. Their devs also were just as involved in the anti-sysvinit push that got Debian to change its default; both the upstart and systemd camps were shouting down anybody that attempted to suggest other alternatives (including keeping sysvinit). After both camps successfully shut out everything else, then upstart lost, but Canonical and upstart is as much at fault for Debian's systemd status as anybody else. Sure, I'd have preferred upstart over systemd based on what I know of the two, but the reality of it is they both pushed this shit too soon, and they did it together.
(Score: 2) by NCommander on Tuesday August 09 2016, @10:40AM
Um, sysvinit is old to the point of needing replacement. No one has seriously argued it was getting long in the tooth, it was an argument on how to replace it. SPecifically, here are the issues
* No dependency tracking, which means you're only method of controlling startup are Sxx numbers
* Can't paralizared startup of services
* Related, one hung service can hang the boot
* No standard; Debian, Red Hat, and other OSes had subtly incompatible sysvinit semantics making writing a cross platform init script a bitch
Short list, but points 1+2 make reboot times minutes long. You can complain that "reboot times" don't matter, but end users disagree, and its nice not to take 30 minutes to reboot a server.
I actually worked at Canonical and was involved in some of the upstart discussions as well as having drinks with Keybuk multiple times. Upstart was deployed in a non-LTS branch for testing a year and a half before the next LTS (it went in Intrepid if I remember correctly). Furthermore, upstart is an init system, and all it does is load its unit files, and start processes. It doesn't hijack most of the system startup, or replace logs or whatever. It was fairly drop in replacement to the point Debian had shipped it for quite awhile. You'll notice that on my rant list, the only rant I have about startup behavior with systemd is it doesn't fail safely on malformed unit files. It's everything else I can't stand.
Still always moving
(Score: 2) by Marand on Tuesday August 09 2016, @07:56PM
Um, sysvinit is old to the point of needing replacement. No one has seriously argued it was getting long in the tooth, it was an argument on how to replace it. SPecifically, here are the issues
Not really inclined to agree that "it's old so we need to repalce it" is a good argument, especially when distros had still been managing to keep it usable. But that's just a difference of opinion.
* No dependency tracking, which means you're only method of controlling startup are Sxx numbers
This is an outdated claim. Dependency-based boot in Debian's sysvinit has been available nearly ten years, and the default for almost that long. OpenRC has been available longer than that.
* Can't paralizared startup of services
This is also an outdated claim. Parallelised boot (which I'm guessing you meant) has been available in Debian's sysvinit for 6-7 years. Not sure about OpenRC's status for it or other distros.
* Related, one hung service can hang the boot
Unless the rest of the boot process depends on it, this should not occur with the dependency-based bootup. Which means this, also, is an outdated claim.
* No standard; Debian, Red Hat, and other OSes had subtly incompatible sysvinit semantics making writing a cross platform init script a bitch
The only potentially valid complaint so far. More of an issue for package maintainers, though, so not really on-topic with regard to the rest of it. Also kind of a funny argument considering Canonical and Redhat both decided to add even more init systems instead of working with existing alternatives. Not really helping the cross-platform init problem there.
Also, that argument actually favours systemd, because now there is a cross-platform standard, whereas nobody but Ubuntu used upstart.
Short list, but points 1+2 make reboot times minutes long. You can complain that "reboot times" don't matter, but end users disagree, and its nice not to take 30 minutes to reboot a server.
No, I'd argue (and did) that the first three points you made are extremely out of date and seem to indicate you haven't really looked at anything but Ubuntu in years, if ever. Which is fine, except that you're considering switching to a BSD based on Linux complaints while apparently not really knowing the landscape outside of Ubuntu.
Furthermore, if you think sysvinit is outdated and that means it's bad, you're in for a rude awakening with FreeBSD [freebsd.org]. You may even hit some of the problems you (mistakenly) declared for sysvinit, though I'm not certain about that.
I actually worked at Canonical and was involved in some of the upstart discussions as well as having drinks with Keybuk multiple times. Upstart was deployed in a non-LTS branch for testing a year and a half before the next LTS (it went in Intrepid if I remember correctly)
You never did answer my question about what advantage there is to deploying Ubuntu servers instead of going with Debian -- which was a serious question, FYI, and I'd hoped for a real answer. Right now the lack of answer + mention of working at Canonical makes it look like a company loyalty thing more than any real decision based on merits. I'd love a real answer though. I'd guess support contracts would be a valid reason, but I doubt SN's paying for that.
Furthermore, upstart is an init system, and all it does is load its unit files, and start processes. It doesn't hijack most of the system startup, or replace logs or whatever.
Yep, like I said in the other comment, I'd have preferred to see upstart "win" rather than systemd, and that's a big part of why. I never really had any problems with upstart itself, other than Canonical completely removing alternative inits to force use of upstart. I'm kind of sad that Canonical abandoned it, because it's not like they're afraid of being incompatible...what with Mir, upstart, Unity, etc. Oh well.
Still, my big issue wasn't with upstart, it was a social problem: I didn't like how upstart+systemd proponents both forced the issue in Debian, and argued down anybody suggesting something that wasn't upstart or systemd.
It was fairly drop in replacement to the point Debian had shipped it for quite awhile.
It's still available in Debian stable. I could switch to it right now if I wanted. It does looks like it's been dropped from unstable/testing, though, probably because Canonical abandoned it. Shame, because other inits are still there. Even with Upstart gone, there's still openrc, file-rc, runit, sysv-rc, and maybe more I'm not aware of, in addition to the systemd default. There are still maintainers that care about having a choice, and they're keeping those inits working and available.
See, unlike Ubuntu, Debian actually kept maintaining packages for those init systems. In fact, Ubuntu should still have them just by virtue of being a Debian-derived distribution, but someone decided to purge non-upstart inits from Ubuntu. Just like Redhat, Canonical preferred to take away the choice and force their own in-house solution on people, which is unfortunate.
So, again, I'll suggest taking a look at other options, such as Debian, before making a knee-jerk reaction against systemd and Linux as a whole and running off to FreeBSD. You're going to have an "old" type of script-based init there, too, and there's no requirement to run systemd in Debian, so it'd probably be a better transition. You just seem to be running off old knowledge and bad assumptions about the state of non-Ubuntu Linux.
(Score: 2) by NCommander on Wednesday August 10 2016, @02:00AM
You do realize I'm a Debian Developer in addition to an Ubuntu Core Dev, right?. Anyway, my point is sysvinit has issues, and was due for either update and replacement. I thought upstart good replacement for technical reasons. The primary reason it didn't get adopted by Debian was political most over Canonical's copyright assignment policy; I remember it came up several times on d-private.
Fine. Let's get into the deep technical flaws with sysvinit.
When you refer to dependency tracking, I assume you're referring to this block at the start of the file:
Debian's dependency tracking (which is based on LSB) is a serious hack, and one I am aware of. When I was saying sysvinit has no concept of dependencies, I was referring to the daemon itself.
update-rc.d de-facto works by dynamically ordering the scripts as part of update-rc.d. Boot dependencies are statically specified, and from the original thread where this was implemented, Tollef Fog Heen [debian.org] provides an example of why this is a bad thing. The system is incredibly brittle because it can't resolve dependency loops or the case of something failing to start. The reason it all appears to work fine is considerable effort is put in by DDs to keep sysvinit chugging along.
I actually had to go look this up and it turns out you're right. It's been in unstable for years, but it didn't actually ship until Debian wheezy (2013). Debian reference manual [debian.org]. I didn't switch back to Debian as my primary OS until jessie so I completely missed this.
See previous point on "dependency" based startup on sysvinit. The fact is sysvinit has no mechanism to detach a process on boot beyond standard shell control. This generally isn't a problem for servers which can wait for a slow process to get going (and generally want to), but it is problematic on desktops for getting short startup times. Basically, in a sysvinit world, you can either wait for a init script to finish entirely, or detach and clear that boot item. Once you detach, you have no way of if a service has fully come up, or is still starting in a generic way.
A big focus is from kernel start to GDM in about 10 seconds. The system can keep booting beyond that, but you need enough up to get X11 fired up and you need a way to signal that its safe to run Xsession (which generally requires other system daemons be loaded).
Upstart handed this via a socket and dbus. systemd has an entire event hook API (https://www.freedesktop.org/software/systemd/man/sd_notify.html)
I actually missed this point. I wanted a Debian-based OS because of bad experiences with Red Hat ones. Architecturally, Debian has very strict constrains on library dependencies and preventing skew which Ubuntu inherited. (man dpkg-shlibs) combined with a very well tested in-place upgrade system. I would have been perfectly fine if we had standardized on Debian stable at the time. I was against using CentOS due to the fact that its notorious for shipping dated software combined with a very high chance of breaking everything if you upgrade in place. Ultimately, we split the difference, that the rehash machines would run ubuntu, and we'd run CentOS elsewhere. In practice, after the one person really pushing for CentOS left, everything new became Ubuntu to standardize.
In contrast, with one minor hiccup, the upgrade from Ubuntu 12.04->14.04 was a non-event. Beryllium (the CentOS box) will get wiped and reinstalled to match everything else. What we may be running on is unknown at this time.
What ultimately made the decision was that when I setup the site, I installed Ubuntu because I done the core work on my laptop which also ran Ubuntu so I didn't have to worry about library skew or anything.
I'm not sure how much I can actually say on this due to most of the Debian side being discussed on d-private. I can say the primarily reason upstart didn't go anywhere was political vs technical. What I can say, on the Ubuntu front, Mark Shuttleworth himself made the call we were going to systemd because Debian decided to do it and we couldn't justify the engineering resources.
As for the rest you mentioned, well, I'll just say I left Canonical for many reasons.
Ubuntu has sysvinit, upstart fired it up at the end of boot processing to run the legacy scripts. You are correct that Ubuntu had a hard dependency on booting with upstart as I believe we had hooked it into our udev. I generally didn't have much issue with this as upstart did one thing, and did it well, vs. systemd's kitchen sink approach. Debian's support for other init systems to my knowledge is at best spotty in the real world though ...
sysvinit had valid technical reasons to be replaced or overhauled. I just think the new 'standard' is a pile of crap. I posted my long list of technical reasons why I hate systemd elsewhere in this thread. If systemd was content to stay on desktop Linux, I probably won't care much; sysvinit is fine for servers for the most part and I've never had a serious issue with it in real life in a server-role. If systemd did one thing and did one thing well (aka, if it had limited itself to being an upstart competitor), I'd also probably not care enough to work up a damn. It's the entire thing combined with the tendrils its run through the stack that makes me throw my hands up and say enough is enough and look at getting off this ship.
I'm well aware that the boot situation on FreeBSD is extremely primitive in comparison. You have rc.conf and rc.local and that's about it. Everything is sanely logged. For a server, that's more than acceptable.
(Score: 2) by Marand on Wednesday August 10 2016, @05:37AM
You do realize I'm a Debian Developer in addition to an Ubuntu Core Dev, right?. Anyway, my point is sysvinit has issues, and was due for either update and replacement. I thought upstart good replacement for technical reasons. The primary reason it didn't get adopted by Debian was political most over Canonical's copyright assignment policy; I remember it came up several times on d-private.
Had no idea about you being a DD, since I only saw mention of Canonical work in the other comment. You're as anonymous as anybody else here to me. :p
And yeah, the copyright assignment thing sucked, I recall there being some grumpiness over it in the past, so that's not surprising at all. What I meant though was that in the open discussions about the future of Debian's default init, upstart+systemd people (including some devs on both sides) seemed keen on shutting down any attempt to suggest anything else. People were determined to make it a two-party vote, and once that happened Upstart lost to politics like you said.
As for the sysvinit thing, I'm aware it's a script-based hack and has its issues, but it still does the job mentioned, so I took your statement as a literal "it can't do this thing it does". It's not hard to imagine someone would miss a change like that in a completely different distro, so I took the remark at face value. Just a communication error there: you were being more literal in referring to init itself while I was interpreting it more broadly.
Also, thanks for the answer about the server selection. I'm with you on the general aversion to Redhat-derived distros, and preferring Debian-based distros for stability and the upgrade process. It's why I've stuck with Debian since around 2000, and why I keep ending up coming back to it after experimenting with other options occasionally. I've mentioned it here before, but my primary OS is a Debian install that started on 2.2 (potato) and has been dist-upgraded through the years, with migrations to new hardware, new disks, and even updated in-place from 32bit to 64bit. I don't know any other OS or distro that I could have done this with. I probably should just reinstall it one day, but it amuses me to keep it going instead of starting fresh at this point. :)
Vaguely related distro talk: I keep intending to spend time with NixOS. I already get a lot of use out of Nix as a secondary package manager and really like its (admittely weird as hell) filesystem layout and philosophies, so it's on my radar but I haven't gotten around to it. It starts out with systemd though, bleck. Does seem to have other options (sysv, upstart, runit) but I have no idea how much trouble it would be to switch out.
At this point, my take on systemd is I'm mostly avoiding it for now. sysvinit and the patchwork of hacks on top of it may not be the way to go long-term (I agree about it being due for update or replacement, I just think it all happened a bit prematurely, at least for Debian), but I'd still rather use it than systemd right now. Like you said, if it were just a desktop thing, it wouldn't be such a big deal. I haven't had any problems with the "higher" parts of systemd; it's not really any better or worse than dealing with, say, HAL was in that regard. (I remember HAL being a nightmare at times early on...)
It's just that I don't want systemd as my init, so I stick to systemd-shim and running anything else underneath. The init just happens to be sysv because that's what Debian used before and it just worked, but it's not like I'm not open to alternatives. Just preferably alternatives that aren't systemd's init. Hopefully by the time I don't have a choice any more, Poettering and Sievers will have gotten bored and moved on so others can clean house. It mostly worked for Pulseaudio like that, at least...
On a related note, have you ever used runit for anything? It sounds interesting but I've never gotten around to spending time with it. I've only heard of one distro (Void Linux) using it by default [voidlinux.eu], but their page on how it works is intriguing. It also appears you can even use it alongside another init in case you only want to manage certain services with it, which sounds like it could be appealing.
(Score: 2) by Justin Case on Wednesday August 10 2016, @02:26AM
No dependency tracking, which means you're only method of controlling startup are Sxx numbers
Has never caused any problems for me. Which of course I'm not arguing it hasn't been a problem for others, but that's what I love about a non-proprietary OS: choice!
Can't paralizared startup of services
Don't want to.
one hung service can hang the boot
As it should. Fix it before proceeding.
reboot times minutes long. You can complain that "reboot times" don't matter, but end users disagree, and its nice not to take 30 minutes to reboot a server.
Again, not a problem for me. I'm not an "end user" and I want to barf every time someone forces me to use software written for the ignorant and lazy, because I am neither. Oh, and if something isn't right, I don't want it to take 30 minutes to boot a server, I want the server to not boot at all. That's an incredibly effective technique for making sure the problem gets noticed, prioritized, and fixed, instead of "oh sure let's proceed down the runway even though one jet engine didn't start..."
BTW please don't take any of this as a personal attack. Love the work you're doing, and praise for doing "the right thing" with https etc.
(Score: 4, Insightful) by shrewdsheep on Monday August 08 2016, @01:52PM
I am very impressed by the sysadmin prowess the soylentnews admin team has exhibited by establishing a scalable and reliable site. I would appreciate if the configuration management files used for the site would be exposed in some public repository. Of course this is at the discretion of the team. I for one am sure to be able to learn a lot from them.
Thank you very much for your good work.
(Score: 5, Informative) by NCommander on Monday August 08 2016, @02:40PM
Most of the site config stuff is really specific to our setup. While I don't think there's anything inherently private in them, I dunno how much help it would be. Off hand, here's all the stuff we use
* nginx (SSL termination)
* varnish
* Apache 2.2 and 2.4
* Hesiod
* BIND
* Kerberos keytabs
* rsync
* MySQL Cluster
Still always moving
(Score: 0) by Anonymous Coward on Monday August 08 2016, @02:58PM
i would appreciate a write up of how you deployed ipv6 for soylent. how is the transition being managed? are you serving over ipv4 and ipv6 simultaneously? how specifically are you doing that? overall strategy stuff along with whatever details you felt like sharing.
(Score: 3, Informative) by NCommander on Monday August 08 2016, @03:01PM
It's actually been written up before, but the short version is we only use IPv4 in two places:
* A record traffic coming into the site (IPv4)
* A few legacy services that can't handle pure v6 (MySQL Cluster).
Internally, the site runs on a li694-22 gTLD. i.e., the current loadbalancer would be sodium.li694-22 (we're named after elmenets, starting at hydrogen and going up). Every machine only has an AAAA record, and the reverse PTR record. For services that *must* be on IPv4, we use ipv4.helium.li694-22. Using IPv6 primarily means everything is end-to-end addressesable and we avoid a nightmare with NAT. our IPv4 only services on the other hand can't scale outside of the current datacentre. That point is mostly moot though as the only off DC site we have is oxygen for backups.
Still always moving
(Score: 1, Insightful) by Anonymous Coward on Monday August 08 2016, @03:12PM
HSTS is kind of pointless IMO. It doesn't fully fix the problem with downgrade attacks because the first connection to a server remains vulnerable. It could still have some value, but it turns out to not be compatible with "private browsing", because by design it requires the browser to keep a database of visited websites and the corresponding HSTS expiry date. That information of whether or not you visited before is also leaked to servers when you connect to them (via HTTP), allowing for a type of "supercookies".
Some browsers have "HSTS preloading", which means shipping a pre-populated database with the browser, which helps a bit. But instead of that they should just include the older HTTPS Everywhere [eff.org], which is way more powerful and useful. Soylentnews has been in the default ruleset of HTTPS Everywhere for some time now.
(Score: 3, Insightful) by NCommander on Monday August 08 2016, @03:43PM
HSTS isn't a magic bullet. Neither is HPKP. Trust on first use already has plenty of inherent problems with it but its a lot better than what it could be. If you're using the site as a "regular" use, it will protect you from the coffee shop MITN, esp if preload is active (which it will be. Just haven't gotten there). It also prevents accidental downgrade (aka, I accidently post a http vs https) link. The supercookie point you bring up is moot because if you simply connect to https://soylentnews.org [soylentnews.org] (which is the official URL), you'll never create a HTTP hop. The only reason we're running the port 80 service at all is browsers will always try that first if you don't specify the protocol.
Security is not a binary thing. Its something you improve and harden, and the goal is make your site hard enough to attack that nothing short of a targetted effort will do so. In the case of targetted attacks, well, my belief is nothing will stop a dedicated gunman between him and his target if he doesn't care about anything else.
Still always moving
(Score: 1, Interesting) by Anonymous Coward on Monday August 08 2016, @05:12PM
The supercookie thing is not a thing about soylentnews, but something that HSTS-enabled browsers make possible to (for example) malicious ad tracking networks. Soylentnews choosing to enable (or not enable) HSTS is irrelevant to the privacy issues. Website operators may as well enable HSTS because it's not their problem to deal with, but browser users should disable HSTS because of the privacy implications. Enabling "private browsing" basically does disable it, because this feature necessarily has to wipe the HSTS database clean for each session.
Here's a simplified implementation of the tracking issue:
Sending millions of image tags in every document is not practical so some kind of probablistic approach (e.g., bloom filters) is probably necessary. While not required for successful tracking, this is all made even easier and more reliable if you can get the client browser to execute javascript code that you supply.
(Score: 2) by bob_super on Monday August 08 2016, @05:55PM
> Security is not a binary thing.
Actually, it kinda is:
- if (machine always fully isolated from world) safe = 1;
- Else safe = 0;
Security tends to be a PEBKAC issue, with silly users actually wanting both machines that are ON, and the ability to connect to it. Then, you're squarely in the "else" case...
(Score: 0) by Anonymous Coward on Monday August 08 2016, @03:58PM
and just one thing remains; to make isp/mitm path analysis harder!
encrypted pages are all the same length as the source. please consider embedding random fluff comments and headers in random positions to vary the payload sizes to demonstrate some geek leadership here!
(Score: 2) by NCommander on Monday August 08 2016, @04:02PM
That's kinda pushing it. Playing with TLS message size means out bandwidth goes up for one, two, we'd have to pad the HTML which requires making a fair bit of changes to rehash. If you want to write a patch for the code to pad it out, I'd consider it, but even then, I'm not sure its worth the trade-off.
Still always moving
(Score: 0) by Anonymous Coward on Tuesday August 09 2016, @01:20PM
I appreciate the feedback nCommander. I really do. A plugin? I wrote an iis-filter that did this some 20 years ago and while i would like to, can't do that for you now.
but you got the point --- security without obscurity --- is no security at all -- when it comes to https.
sure, some stuff gets encrypted and hidden, but path analyst laugh at you since they can attribute everyone's posts to them by page-size. ... anyway, i salute you -- and in the meantime, how about a 'MAX-OBSCURE: ' entry in the header?
(Score: 2) by number6 on Monday August 08 2016, @04:35PM
**** Preamble: I'm on a Windows computer but the CLI tools I used below are available on *nix systems too ****
I spent hours and hours last night trying to find a way --with command-line tools-- to output soylentnews.org using the IP address 45.56.123.192 as input.
I gave up; I just couldn't do it.
I have created a batch script ("NetworkInfo-Query-IPAddress-DNS.bat") which I run on demand;
it runs two commands and displays the info in the console;
command 1 is: curl.exe ipinfo.io/%IPADDRESS%
command 2 is: dig.exe +short -x %IPADDRESS% // see note [1]
Here is what the console displays when I use "45.56.123.192" as input:
I am trying to find a third command to add to my script which will allow me to extract all hostnames/aliases of the IP address.
If I visit this webpage for reference: https://www.robtex.com/?dns=45.56.123.192 [robtex.com]
I see that "45.56.123.192" points to two host names: "grepnews.org" and "soylentnews.org" .
So what command-line tool can I use to extract those host names ?
[1] --------------------------------------------------------------------------
The "curl.exe ipinfo.io" command outputs its info formatted with indentation
and braces. The "dig.exe +short -x" command does not give the same output
format; so to make it have the same display formatting I used this command:
for /f "tokens=*" %%A in ('%CD%\dig.exe +short -x %IPADDRESS%') do (
echo {
echo "Reverse DNS": %%A
echo }
)
You may think that my use of the "dig" command is redundant given that the "curl" command
gave the same "hostname" info, but in practice the "curl" query many times gives me blank
results for the "hostname" field, whereas the "dig" command _always_ gets this info.
(Score: 2) by Scruffy Beard 2 on Monday August 08 2016, @05:00PM
You do know that you can have more than one domain pointing at the same IP, right?
(Straying outside my knowledge here): The only time you really need the reverse DNS to match is for mail (mainly to combat spam). I suspect there are other use-cases that I am not aware of.
(Score: 2) by NCommander on Monday August 08 2016, @05:13PM
Using powershell:
PS C:\Users\mcasa> resolve-dnsname soylentnews.org | ForEach-Object { Write-Host $_.IPAddress }
2600:3c00::f03c:91ff:fe98:b8fe
45.56.123.192
Still always moving
(Score: 2) by number6 on Monday August 08 2016, @05:31PM
Thanks for the reply NCommander +++++
unfortunately for me....
I'm on Windows XP and I hate PowerShell and do not have it on my system; I also hate Microsoft .NET and do not have it on my system.
I hate all post-XP/2003 Microsoft operating systems.....from a control/configuration/GUI preference perspective; nothing to do with kernel/memory/graphics/security subsystems improvements.
When my system eventually dies, I will be moving to FOSS and the less I have anything to do with Microsoft-centric or Apple-centric solutions the better.
Do you know of a standalone standard CLI tool which is compiled for all platforms which can extract this information?
Thanks in advance.
(Score: 2) by number6 on Tuesday August 09 2016, @01:02AM
***** Continuation of my first post *****
Okay..... I eventually found a way to extract the canonical hostnames ("grepnews.org" and "soylentnews.org")
which are pointing to "45.56.123.192" .
The command-line tools I needed were:
* httrack.exe - website copier, spider
* sed.exe - famous powerful text processing utility
* UnixToDos.exe - to convert line-endings from Unix (LF) to Windows (CR/LF) after processing with 'sed'.
Here is a preliminary draft of what I will be adding to the batch file (mentioned at first post):
Using the code and tools above I have captured the canonical hostnames for IP address "45.56.123.192".
I think the code needs more testing because splitting the string at numbered delimiters may not always work
for instance in cases where the hostname begins with a number or numbers.
I think I need some improvement to the regex pattern for splitting the string.
Do any of you guys have a better pattern?
(Score: 2) by martyb on Tuesday August 09 2016, @02:25AM
The wget utility is available at: eternallybored.org [eternallybored.org]
Please let me know how that works for you.
Wit is intellect, dancing.
(Score: 2) by number6 on Tuesday August 09 2016, @05:06PM
@martyb - WOW! your script gives a perfect result and completes the task like a bullet flying out of a gun!
Comparing the completion times of your script to mine:
* Your script --> start time: "xx.xx.27.17" seconds | end time: "xx:xx:28.85 " seconds --> total time: 1.68 seconds
* My script --> start time: "xx:50:35.68" seconds | end time: "xx:51:42.68 " seconds --> total time: 67.00 seconds
the program causing the most delay in my script was 'httrack' ; it takes forever to download the page ; 'wget' does the same thing in an instant!
Thanks a lot martyb, that was educational.
(Score: 2) by number6 on Tuesday August 09 2016, @07:05PM
Upon further testing of your code, I see wrong output...
Querying the IP address "216.34.181.45" (it is SlashDot)
your script outputs this to the console:
but if you visit the robtex webpage to look at the items you will see there are many more!
here is the webpage: https://www.robtex.com/?dns=216.34.181.45&rev=1 [robtex.com]
(Score: 2) by number6 on Wednesday August 10 2016, @05:00AM
Ok, I found the problem .....the Windows 'find' command has limitations!
Read this:
.http://ss64.com/nt/find.html
Limitations: "Although FIND can be used to scan large files, it will not detect any string that is positioned more than 1070 characters along a single line (with no carriage return) This makes it of limited use in searching binary or XML file types."
so, when I tried @martyb's code on IP address "45.56.12.192" (SoylentNews.org), the code worked and I got the correct result because the string to be extracted from the HTML file by the 'find' command was small (a 223 character string containing 2 hostnames).
but, when I tried @martyb's code on IP address "216.34.181.45" (Slashdot.org), the code worked but i got an incorrect result because the string to be extracted from the HTML file by the 'find' command was very large (a 1469 character string containing 19 hostnames).
PROOF .......let's run some commands and display the output (for IP address "216.34.181.45"):
This result is incorrect, only part of the string has been extracted from the HTML file:
This result is correct, all of the string has been extracted from the HTML file:
So, to get the correct output for use in my batch script, we modify @martyb's code, replacing "type .. | find .." with a "sed" string extraction command,
giving us this final result, which is correct (19 hostnames):
(Score: 2) by martyb on Wednesday August 10 2016, @02:44PM
I'm glad you found the code I offered to be helpful... and witness the value of comments expressing the *intent* of the code, too!
I failed to note that what I provided was NOT a general solution, but instead intended to solve only that specific case. Further, I was mindful of your being in a Windows environment and not having ready access to the full stack of Unixland commands — hence my using find.exe.
NOTE: Using regular expressions (regexps) to parse HTML is a known folly in the general case [stackoverflow.com]. It can be useful in specific circumstances, and I have done so in many one-off, controlled situations. But, you may well find yourself removing hair in vast quantities if you fail to understand this limitation!
I did not see the results you did, but on further exploration with other IP addresses, I found that my copy of find.exe (Windows 7 Pro x64) limited the returned string to 4095 bytes. (For the more extreme example, I tried "98.138.253.109" — yahoo.com, which had 62 domain names.). Using your sed pattern did work — I was not aware of that use for sed — many thanks!
That said, I would much prefer to find a native, command-line solution using userland commands (even GNU/Linux would be fine) instead of needing to query a web site. How DOES robtex do what they do?
Wit is intellect, dancing.
(Score: 0) by Anonymous Coward on Monday August 08 2016, @07:29PM
Thank you NC, you rock! This is super nice to see happen, and doing it 'in the open' like this makes me squirm with delight. The way things *should* be done!
Soylent is people, and soylent is good, so the people must be good!
(Score: 2) by korger on Monday August 08 2016, @07:29PM
Green site, take that!
(Score: 2) by Joe Desertrat on Monday August 08 2016, @07:56PM
Rats. I was hoping HSTS was Hunter S Thompson System or such.
Just trying for a cheap laugh. As usual, the staff is going above and beyond...
(Score: 4, Informative) by arslan on Tuesday August 09 2016, @12:28AM
How bout setting the http only flag while you're at it so the cookie is not accessible from javascript. Secure flag is also not set, but the single page I tested seems to be loading everything from https URL, so not a big deal.
(Score: 2) by NCommander on Sunday August 14 2016, @03:17AM
I vaguely remember there was a reason why we didn't do this, and I'll be damned if I can remember this. I think it related back to our initial SSL support which was optional, and issues with the way the site handled the login cookies. Most of the login code got a blowtorch when we went to IPv6 (which was also the first time we ever had to remove a feature from the site: login localization), so I don't know if the reason is still true or not but I rather not break it now that it all works.
Still always moving
(Score: 1) by maggotbrain on Tuesday August 09 2016, @02:06AM
Thanks for the details on site changes. I'm happy to see some larger sites taking more time to discuss this as well. Also, seeing greater transparency in outage postmortems is encouraging.
(Score: 0) by Anonymous Coward on Tuesday August 09 2016, @03:24AM
I love the site updates. The explanations of what went right or wrong and how to admin a high traffic website, at least relative to mine, are very informative and helpful.
(Score: 0) by Anonymous Coward on Tuesday August 09 2016, @02:32AM
Needs more security headers: https://securityheaders.io/?q=soylentnews.org&hide=on&followRedirects=on [securityheaders.io]
(Score: 3, Insightful) by kazzie on Tuesday August 09 2016, @05:50AM
Not to mention cowbell.