As I get through my remaining TODO items (such as restoring our second webfront, hydrogen to production), I find myself needing advice on a few final items: implementing HTTP Strict Transport Security and HTTP Public Key Pinning. Although simple in theory, both these options have complications that I want feedback from.
HSTS
For those not familiar with HSTS, it essentially says that this site ONLY uses HTTPS, and to never attempt a plain HTTP connection. This primarily helps in preventing HTTPS stripping attacks. The downside is once enabled, we won't be able to disable HTTPS for any reason; web browsers will refuse to open a HTTP-only connection, and the site will break for most captive proxies (this is incidentally the reason that if you're behind a captive proxy, and try to go to Facebook or Twitter, the connection fails with the usual SSL error page in Firefox and Chrome). Furthermore, we will not be able to implement the 'includeSubdomains' option at this time as several our of sites use self-signed certificates. We'll be replacing these with real SSL certificates or a wildcard certificate in the near future to allow this.
HPKP
HTTP Public Key Pinning on the other hand, works to solve the age-old problem that any CA can sign any site. By pinning the SSL public key, we can indicate to browsers that an SSL connection to us is only valid if the certificate is both signed by a CA, and that the public key is one we control (and paired with our private key). Other SSL certificates will be rejected since they will not have the same 2048-bit key we use. The CA chain-of-trust is reduced to the first connection to the site post-pinning. The browser will cache the HPKP header and remember the pins until they expire. The downside to this is it both complicates key management, and will break the site if someone tries to access it behind a SSL proxy. For those unfamiliar with SSL proxies, they are (usually) hardware devices preloaded with a valid intermediate SSL certificate, which dynamically generates a "valid" SSL certificate for any HTTPS connection, and allows the operator to decrypt any encrypted traffic passed through it in such a matter that is almost invisible to end-users. I'm not sure how common such devices are, but once HPKP is enabled, SSL proxies will cease to function unless a corporate administrator disables HPKP support in the browser.
What I want to know from the community is the following:
- Have you deployed HSTS/HPKP to any sites you administrator?
- What, if any gotchas did you run into?
- Have your users reported any unusual breakage or such?
Finally, for those wondering 'why bother', beside its obvious purposes, I want SoylentNews to be an example of a website done right; IPv6 support, strong SSL support, etc. We want to be an example other sites mimic, and as such, embrace technologies that protect the user from Man In The Middle, and such.
(Score: 3, Insightful) by bradley13 on Monday June 22 2015, @07:23PM
As someone who manages just one website for a small company (turned https-only through .htaccess), I'll be very interested in hearing how to do it right. I expect lots of other Soylents are in similar positions: managing one of a few websites, and without the time and expertise to dig all this out ourselves.
Everyone is somebody else's weirdo.
(Score: 2) by DarkMorph on Monday June 22 2015, @10:09PM
You can redirect requests for / to https effectively making no request over port 80 possible to be served a document without the redirection. Also looks handy for aliasing in general alluding to the name of that module, mod_alias. HSTS sounds helpful but what about clients intentionally attempting to request a document via http?
(Score: 2) by NCommander on Tuesday June 23 2015, @03:58AM
Our setup is a bit extreme for personal sites; we terminate SSL at our "load-balancer" which is really a glorified Linode 1024 with nginx. Here's the rule we use to bounce port 80 to 443:
The double listen directive is required to get nginx to listen on both IPv4 and IPv6. Our cipher/TLS setup gets us an A rank, and 90s in Key Exchange and Protocol Support. I'd like to tighten it up, but it would kill basically anything older than 1-2 years old, such as Android 2.3, and pre-IE10 ..
Still always moving
(Score: 0) by Anonymous Coward on Tuesday June 23 2015, @06:59AM
Of course with any redirection method, the original request (including the URL you requested) first goes through the wire unencrypted. So if it is a publicly accessible document, anyone seeing the original request can just request the same document himself to see what you looked at. IIUC HSTS tries to fix that problem as much as possible server-side: As soon as you've accessed it over https once, the browser will know that the site should be accessed via https only.
A client-side solution is the HTTPS everywhere [eff.org] extension. That's in one sense better because it prevents http even for the first contact for a site, but in another sense worse because it requires the user to be aware of the issue (and to use a browser for which the extension is available).
(Score: -1, Redundant) by Anonymous Coward on Monday June 22 2015, @07:24PM
No 16384-bit key?
(Score: 3, Interesting) by tempest on Monday June 22 2015, @07:28PM
HSTS headers are only valid if sent through https, and are not to be sent through http. Therefore the only way to end up with a browser pointed that way, is for them to have visited via https at some point. The nice thing about HSTS is that it solved an ATOM feed issue. Atom requires the entire URL be specified in the feed, but I only provide TLS optionally for those who want it (sign my own certs currently). This meant that anyone subscribing to atom feeds pulled them via plain http, and that the links were also plain http. HSTS secures everything in a roundabout way.
I previously used Key Pinning, but since I signed my own certs just pinned the cert itself (which you're not supposed to do). I set the period to one week, so provided I visit my site once a week this means the browser detects a third party certificate injection. I plan on trying lets-encrypt, so I've since let it lapse as I'll be pinning them as a CA.
Note that my site has virtually no traffic, and https is offered optionally. I'd assume it's unused aside from browsers using opportunistic encryption (which is unverified). I only allow 4 TLS algorithms, so it's not like I give a shit :p
(Score: 5, Informative) by darkengine on Monday June 22 2015, @07:46PM
I'm the server admin of a small/medium-sized community (~1,500 active posters, ~8,000 visitors) called lainchan [lainchan.org], and we use HSTS and plan to add HPKP.
HSTS is great. Very easy to set up and is definitely helpful in increasing security against would-be coffee shop MitMs. If at all possible, I'd strongly suggest adding the includeSubdomains and preload options, so you can add yourself Google's database of HSTS Preload sites. This way, first-time visitors will deny insecure connections as well.
HPKP is a bit trickier. Read this [mozilla.org] carefully before you start. The big takeaway here is to generate a backup key and store it somewhere safe, and preferably offline. The spec requires you to specify two keys for HPKP: the key being used, and a key for backup. This way, if your production key gets compromised or lost, you can use your backup key and HPKP-aware browsers won't freak out.
lainchan does not use HPKP yet, mostly because I haven't found the time to generate a backup key and test the header locally to make sure we can fall back to it properly. If we had to fall back on the backup and it turned out I had messed it up somehow, the result would of course be disastrous.
Good luck!
(Score: 3, Informative) by NCommander on Tuesday June 23 2015, @04:11AM
*slow clap*
Excluding the fact that we can't get A+ since we don't do HSTS yet, you score higher than SN on the SSLLabs test. Nicely done. Our protocol support is a bit broader though because as you have lainchan, it would prevent IE on Windows 7 from connecting at all:
It also breaks Android 2.3.
From our old piWik data, we have about 1/3rd of our users using IE, and a good chunk of that on Win7; we'd break the site for too many people if I tightened it up THAT much.
Still always moving
(Score: 2) by kaszz on Tuesday June 23 2015, @04:10PM
Make an separate site frontend that is fully tighted up?
Like:
realysecure.soylentnews.org
ordinary.soylentnews.org
(Score: 2) by NCommander on Tuesday June 23 2015, @06:54PM
There's technical and social reasons why that's difficult at best. when launchpad used to have pre-production code on edge.launchpad.net, you'd get links to edge everywhere fro the users that were using it. Given tighter TLS settings will break a good chunk of the userbase, it would mean random clickthroughs to SN just not working for some people. The logistical issues alone will be a nightmare. Perfect is the enemy of good in this case.
I'd love to disable TLS 1.0, and reduce the algos we support, but we would cause edge case failures and more and more pain. Not worth it. Right now, everything just "works"
Still always moving
(Score: 2) by gnuman on Monday June 22 2015, @08:23PM
TLS-DANE fixed the 2nd problem. You can pin your certificate via DNSSEC, but alas, no one uses it. So since no one uses it, these problems cannot really be solved. And since you have no idea where you certificate comes from, then what is HSTS good for?
From the wikipedia links about HSTS,
The initial request remains unprotected from active attacks if it uses an insecure protocol such as plain HTTP or if the URI for the initial request was obtained over an insecure channel.
and HPKP is not very useful - what happens when the CA is replaced with another because key was compromised? Where is 2nd authentication for the CA? What happens if initial request is already MiTM?
Anyway, this is all moot anyway. All that is required to hijack traffic is rogue DNS server. And ISPs use this method all the time to hijack traffic to their "notification" sites instead.
TL;DR? - deploy DNSSEC. Worry about rest later.
Name Server:NS1.LINODE.COM
Name Server:NS2.LINODE.COM
Name Server:NS3.LINODE.COM
Name Server:NS4.LINODE.COM
Name Server:NS5.LINODE.COM
DNSSEC:Unsigned
(Score: 1) by darkengine on Monday June 22 2015, @08:33PM
DNSSEC is, IMO, broken [sockpuppet.org], and a solution looking for a problem.
Use the backup key you were required by the standard to specify.
Add the preload header to your site and put yourself on the HSTS preload list.
(Score: 2) by NCommander on Tuesday June 23 2015, @03:43AM
I've read that post, and quite a bit of other DNSSEC rants that I honestly don't agree with. sockpuppet.org conflates the issue that DNSSEC is a replacement for the authentication properities in TLS. The argument on DNSSEC being government PKI proves that they don't understand what DNSSEC is. (arguably, KANE is, but I'll address that in a moment)
It's not. DNSSEC does one thing; it prevents DNS records from being spoofed via cache-poisoning attacks, or implementing a MITM attack via DNS (if you've got an ISP that redirects NXDOMAIN issues to a search page, you've been DNS MITMed). I fully admit the design of DNSSEC is rather clunky, but given the design of DNS, I'm not sure any replacement would look different. DNSCurve doesn't solve the issue because it requires the private key be installed on *every* root server; the IETF postiion is made very clear in this YouTube video [youtube.com]. DNSSEC allows the private key to be stored separately from the DNS server, which is a requirement for signing the roots.
DNSSEC can't stop nation-state attacks; not a lot can, nor was it designed to. Even taking in account that a nation-state could get the DNSSEC root signing keys (and I question if they realistically can without it being discovered), said nation-state would have difficulty since they'd have to rebuild the entire TLD to modify a record. There are easier ways to crack an egg. Furthermore, as I've said in other posts, nothing stops the NSA or other TLAs from getting a warrant and imaging our servers, or wiretapping the last-mile connections in the Linode data center which are unencyrpted (which due to practical reasons is difficult to fix).
When we sign soylentnews.org, to successfully impersonate us, you'd have to do all of the following:
* - only valid if a user has previously visited SN
I fully believe a nation-state could succeed at doing this, though it wouldn't be trivial. However, most attackers have much less resources; our biggest threat is a disgruntled user who wants to steal credentials of someone else, or an admin, and wreck havoc on the site.
Returning to the issue with KANE, its the only part of that rant that is semi-based in truth. KANE is essentially SSL pinning on a DNS level, against the certificate vs. the actual key. I consider that designed flawed, but its better than nothing. If KANE is used, then the cert chain is pinned in addition to HPKP and HSTS requirements. This stackexchange post has a LOT of good information on the subject, and where I found that YouTube video [stackexchange.com].
The problem most people don't get is security will never be perfect. A targeted attack against any site is incredibly hard to beat, and I'm fairly sure that if someone put their mind to it, they could break SoylentNews. However, which each layer of security added, attacks get harder and harder, to the point that most attackers will likely go for a softer target if its not specifically targetted at you. For example, for over a year, we used a very old version of Apache because that's what the site was dependent on. There are likely Day 0s out there against Apache 1.3 that are unpublished. I setup AppArmor to secure the Apache instance so even if you could achieve remote code execution, there wasn't a lot you could do with it.
Of course, then someone could break AppArmor, then you're running as the slash user. Then you need to elevate to root, and then figure out how to break further into our network. Each layer makes an attack harder and harder. There is no magic bullet, or magic unicorn when it comes to security. Its layering on enough armor that when that attack comes, *something* will stop it. And DNSSEC, HSTS, and HPKP is another layer of armor.
Still always moving
(Score: 2) by NCommander on Monday June 22 2015, @08:50PM
DNSSEC is on the to do list. We had a long lasting issue with the domain that prevented us from using our own BIND instance. As soon as Paul does the migration I'm going to sign the site records.
Still always moving
(Score: 2) by gnuman on Monday June 22 2015, @09:22PM
Excellent! DANE could be next addition then,
http://www.internetsociety.org/articles/dane-taking-tls-authentication-next-level-using-dnssec [internetsociety.org]
The only issue with DANE is inability to quickly change your certificate - you need to roll it over, like a DNSSEC key,
http://tools.ietf.org/html/rfc6698#page-31 [ietf.org]
Then this would completely solve the problem of rogue entities doing a MiTM. They would have to MiTM the entire root DNS, which is a little problematic considering root keys are hardcoded.
After adding DANE, adding HSTS would actually be worthwhile.
Sadly, I do understand problems supporting DNSSEC. Most DNS services out there still do not really support DNSSEC, and then there are issues with CDNs. Hopefully this is not a problem for Soylent.
(Score: 2) by NCommander on Monday June 22 2015, @09:35PM
We don't use a CDN (our average page load is a few kilobytes). So no problems there. The key rotation issue is why we havent deployed DNSSEC yet. Fortunately our registrar, gandi supports DNSEC fairly well so once we are hosting soylentnews.org on BIND, enabling it and DANE should be straightforward
Still always moving
(Score: 2) by NCommander on Monday June 22 2015, @08:57PM
On the hpkp note, we pin the key the site uses, not the CA. We can change CAs freely as long as we use the same key pair.
Still always moving
(Score: 1, Funny) by Anonymous Coward on Monday June 22 2015, @08:34PM
Having a laptop with a faulty battery here, I get problems with certs all the time.
WTF? you ask, what does the battery have to do with certs?
Well, the battery keeps the clock correct and sometimes I forget to set it before browsing...then things get real interesting.
Anyway, ya I could set up some daemon to auto sync it but I don't like anything running on auto.
Regardless, it got me thinking that it will be nice if a website comes back with a help page to offer suggestions on how to fix your connection instead of just a pure failure.
my unworthy 2 cents.
(Score: 4, Funny) by maxwell demon on Monday June 22 2015, @10:44PM
So I get you boot straight into a shell from where you start everything else by hand?
The Tao of math: The numbers you can count are not the real numbers.
(Score: 0) by Anonymous Coward on Tuesday June 23 2015, @02:01AM
His /sbin/init must be a link to /bin/sh.
(Score: -1, Troll) by Anonymous Coward on Tuesday June 23 2015, @05:00AM
cents are way too modern.
(Score: 3, Insightful) by JeffPaetkau on Monday June 22 2015, @08:57PM
I just want to say thank you for continuing to make the site better. Keep up the good work.
Since launch I have been visiting this site and "the other site" about 50/50 and have noticed that Soylent News keeps getting better (slowly but surely) and the other site worse. The current share button fiasco at the other site now has me coming here first and only visiting the site occasionally. The other site still has an advantage in terms of the size of the community but that gap seems to be consistently closing as well.
In terms of improvement, what can be done to enhance how the site displays on mobile devices?
(Score: 4, Informative) by paulej72 on Monday June 22 2015, @09:05PM
We have mobile device support on the todo list. There are some things that need cleaned up in the UI first for the normal site before I start tackling the mobile css.
Team Leader for SN Development
(Score: 1, Interesting) by Anonymous Coward on Tuesday June 23 2015, @07:04AM
How exactly do you plan to support mobile devices? I hope not with separate URLs.
(Score: 2) by paulej72 on Tuesday June 23 2015, @01:11PM
No not separate urls. I was thinking of a setting that gets saved as a cookie that will send out a different set of css files. This way a user can select the version of the site based on device. The switching between these would be in an easily accessible place on the front page, possibly in the footer so it is available everywhere.
I currently hate auto switching UIs as I as a user then cannot set the mode I want. My phone is high enough resolution to show a normal desktop site and I prefer that to some the over simplified mobile sites.
Team Leader for SN Development
(Score: 0) by Anonymous Coward on Tuesday June 23 2015, @04:16PM
That sounds great. However you could do a combination of auto-selection for users not having that cookie set, and cookie selection for users with that cookie. That way there's a good chance that users will get the right version for their system right out of the box, but they still have the option to set it the way they want if the auto selected version doesn't fit their way (no matter what you do, you have to decide on a version to show if there's not yet a cookie set). Also, it would keep the site cookie free for anonymous users who agree with the automatically chosen version. A cookie would only be set for those who explicitly request the other version.
(Score: 1, Touché) by Anonymous Coward on Monday June 22 2015, @09:12PM
Oblig Dilbert [dilbert.com] from yesterday.
Will someone think of the Elbonians? How is this going to help Fred monetize his free wi-fi?
(Score: 5, Informative) by _NSAKEY on Tuesday June 23 2015, @12:08AM
I set up HSTS on a couple of personal sites with nginx. This was the basis of my config: https://gist.github.com/plentz/6737338 [github.com]
The big gotcha is that my TLS set-up is so hardcore that people with older browsers can't view the two sites in question (Granted, IE was never tested because almost nobody uses it anymore, and the person I remember having trouble was using a G3 Mac with a period correct version of OS X). Another thing: If you're going to do this the "right" way using modern best practices, you are required to use forward secrecy. You may want to include a fallback that doesn't implement forward secrecy, but I wouldn't.
One more thing: Since we're talking about going hard on crypto, you might want to consider hardening ssh too, even though it has nothing to do with the experience of normal users. I'd suggest reading and understanding everything in stribika's rather awesome "Secure Secure Shell" guide, found here: https://stribika.github.io/2015/01/04/secure-secure-shell.html [github.io] If you want a friendly version without the required reading and comprehension, this github project of mine implements everything: https://github.com/NSAKEY/happy-dance [github.com] I promise it doesn't install any Chinese backdoors.
(Score: 2) by NCommander on Tuesday June 23 2015, @03:19AM
Interesting read. Not practical for us to setup as we'd have to backport OpenSSH 6.5 (I believe all machines are using 6.0 with security stuff backported). The paranoia factor is a bit high, but understandable given recent circumstances. We already had to backport OpenSSH once before we upgraded the backend to Ubuntu 14.04 and it was a nightmare all around; I have no desire to repeat the experience.
The author only concerns himself with using OpenSSH; I'm not sure PuTTY's support of algorithms is as strong; we have non-technical folks requiring access to our landing box as a minimum, and setting up openssh on Windows requires cygwin and a lot of pain (Windows 10 will help but I can't ask everyone to go upgrade to that).
Furthermore, the weakness IMHO isn't the algo; I'm fairly sure that even if the NSA can crack SSH traffic, it still requires a considerable amount of effort, it wouldn't let them gain access to the box (and given we're US based on a VPS, they could just warrant Linode and take a readonly dump of our machines; no need to do it the hard way). The weakness is that public key auth lives on the machine itself, and if you can copy the key from .ssh at any moment (i.e. that one time you forget to lock your laptop when using the restroom), you're hosed. If it was at all practical to do so, I'd have our sysadmins use kerberos over the internet and use that to autheticate to our systems; unfortunately, kerberos requires quite a bit of configuration to get going on any platform, close to impossible to make it fly on Windows (I've tried), *and*, can break in interesting ways if rDNS or timestamps get fucked up. It works fine for interserver use (we use SSH+kerberos vs public keys since we can revoke principals in a single place; great if we're ever compromised), but over the internet?
Iffy at best.
Still always moving
(Score: 2) by _NSAKEY on Tuesday June 23 2015, @04:06AM
Ubuntu 14.04? It should work out of the box, according to my testing. Of course, that does nothing for your putty users, or anyone using older clients. Those people will almost certainly get left in the cold by such a setup, but now we're straying further and further off-topic.
(Score: 3, Interesting) by jmorris on Tuesday June 23 2015, @12:34AM
Why does everyone think encryption is the answer to every problem, regardless how many time it fails or creates new ones?
Sign in should always be via https. Content generally only need be signed to stop the growing practice of pond scum wanting in inject advertising and worse. And some content probably doesn't even need that level of protection, ad networks for example would probably rate power savings over the possibility of evil manipulation.
We should be thinking of the downsides to a 100% encrypted world before stampeding into it out of fear. Am I the only without amnesia? Didn't exactly this sort of stampede give us the USAPATRIOT act in the first place? Caching is a distant memory just for starters. Power consumption is going to probably be a factor, especially in an increasingly battery powered world.
For example, take this site. Other than my password there is absolutely nothing worth encrypting in my interaction with this site. I post under my own name with my live 1990's era mailbox all but openly visible on every post. Why in the wide world of sports would I care who sees the pageviews in transit? Others might feel a need to post AC and care enough to connect to the https version and that too is equally valid and worthy of supporting.
(Score: 4, Interesting) by Anonymous Coward on Tuesday June 23 2015, @02:43AM
Because what you visit, when you visit it, how long you linger on it, where you came from before you visited it, where you go after you visit it are all interesting to bad apples/actors like StaSi/NSA and other pond scum.
What you do on your pipe is your business, not theirs, not mine, not anyone else's.
Sure, with this solution, I still get to see you visited soylentnews.org but I don't see what you visit inside of sn.org. Because that's NONE OF MY BUSINESS...
Sadly, may actors in the world try to make it their business to see what you do so they can target you for whatever they want: manipulation, social engineering, echo chamber, etc... It is time to fight back against this invasion of privacy. (Imagine your precious founding fathers retiring to the idea that "you know, these Brits and their taxes... We've gotten used to it... let's not rock the boat too much"...
But hey, call me a tin-foil hatter... I don't care! Unfortunately, in practice, I've been proven correct on many occasions and those calling me a tin-foil hatter have been shown incorrect...
(Score: 2) by NCommander on Tuesday June 23 2015, @03:49AM
^- AC is correct on all counts.
As current web technologies go, as far as I'm aware, only I2P and Tor can protect against actually seeing what you browse due to the design of TCP/IP, and its pretty trivial to see if a user is using tor if you're watching network traffic so even that doesn't help as much as one would think. We've already done what we could on this front by implementing a tor hidden service for the site. Tor users will likely always be a minority, so we need to take steps to help the majority who aren't aware of tor or are unwilling or unable to use it.
Still always moving
(Score: 0) by Anonymous Coward on Tuesday June 23 2015, @09:45AM
but I don't see what you visit inside of sn.org.
You'd have to do a lot better to hide that information. An attacker only needs to look at the size of the response to know what you're looking at. There's lots of ongoing research on uncovering information about the contents of encrypted communication without decrypting it.
Still, encrypt everything. It makes snooping more difficult.
(Score: 2) by _NSAKEY on Tuesday June 23 2015, @04:04AM
Because you have intelligence agencies going wild with their effectively unlimited budgets and injecting exploits into http streams. That's why YouTube went https-only a while back. That's really the only reason. Of course, by solving that problem, you create more complexity, and all of your points have some validity.
(Score: 2) by jmorris on Tuesday June 23 2015, @08:22AM
If the FedGov wants to get to you via YouTube they just get on the other side of the encrypted link. They were deep inside just about everybody's network once and you can bet yer ass if anything else goes Foom! on the scale of 9/11 they will again. And everybody will roll over and let them, again. But this time the magic crypto pixiedust will have lulled everyone into a false sense of security. I.e. created at least as many problems as it set out to solve.
Tech almost never solves people problems. Tech often can't solve political problems. I'm old enough to remember PGP and the lawsuits around that, the EFF convinced that if they could just win and make crypto available to the masses it would be some sort of win against 'the man.' I believed that too, back when I was young and stupid, bought a membership and everything; yes I even have the t-shirt. Ok, they won, and the lame exportable crypto of the DVD was soon replaced with crypto, leading finally to the XBox360 and PS3 and soon all PCs locked with unbreakable crypto. We forgot that 'the man' would also be freed to use the good crypto too if the export regs went away. We have traded Tor for Linux. Not a good trade if you ask me because if 'they' own all the PCs from the bootloader on Tor is klnda pointless.
(Score: 2) by kaszz on Tuesday June 23 2015, @04:38PM
Secure crypto and all that is good but please keep a dirty simple frontend around for the situations when all else fails. Like plain http over IPv4 but with a stern warning at the top of every page "Your connection is vurnable" in bold red with normal font size. Perhaps login should be prevented from that frontend. But one could add a PGP signing as a html comment.