Since people seem to rather enjoy when I run articles on backend upgrades, here's another set of changes I made over the last week as I get back into the full swing working on the site.
The short list:
Read past the fold for more information.
Beryllium is our "misc" services box. It basically hosts everything that isn't related to site infrastructure such as the wiki, our IRC server, and mail. Last week, I went through and fixed our SSL configuration on this machine to make sure that we were serving properly validated certificates, and that we had strong encryption on this box. While I succeeded on that front, for performance reasons, Apache 2.4 needed to be upgraded to support a somewhat obscure feature of TLS known as OCSP stapling.
What is OCSP stapling you ask? Well, to answer that, I need to take a moment to go into how SSL certificates work. Whenever a CA generates a certificate, they're essentially saying "this site is who it is and we're attesting to it". In a perfect world, a CA would never make a mistake, private keys would never leak, and we could always assume that a certificate is good. We don't live in that world, as such certificate authorities sometimes need to void a certificate. OCSP (which stands for the Online Certificate Status Protocol) is one of two ways to do this, and is the only method Let's Encrypt supports for certificate revocation.
OCSP is a replacement for older certificate revocation lists (CRLs) which in real-life rarely if ever worked as advertised. It's meant to allow the browser to update in real-time knowledge if a certificate is good or bad and react accordingly. OCSP however requires that the browser checks with a certificate authority's OCSP server, leaking the fact that user X is connecting to site Y. It also means that if access to the OCSP server is blocked, a user might not be aware that a certificate has been revoked. OCSP stapling solves both problems by having our servers grab the OCSP reply (which is timestamped), and sending it as part of the initial connection to our site, both increasing performance, and preventing a privacy leak.
Unfortunately, OCSP stapling requiring Apache 2.4 which required me to build it from source, and then migrate sites over from the older Apache 2.2 install. At the same time, I went through and upgraded PHP 7, and updated the other web applications we were using. For the most part, this was rather painless though I'm still tinkering with MediaWiki to make it happy on the new setup.
Beside the usual Apache pain, I went through and scanned our other major services and disabled SSLv3 support on postfix (SMTP) and dovecot. I need to go through and replace our self-signed certs with real ones here but that's a 'one step at a time thing'
During the last site status article, an AC pointed us at this handy site showing security headers. As such, TheMightyBuzzard and I will be going through and enabling these (with the exception of public key pinning) on production sometime this week. HPKP requires quite a bit of planning to deploy and we're not ready to take that step just yet.
I've talked about wanted to deploy DNSSEC before, but various other things kept cropping up. That, and combined with outdated and misleading documentation kept me from actually getting around to doing this for ages. Over the weekend, I finally dug down and figured out the current best practices for DNSSEC, and with the help of audioguy, configured BIND to do automatic signing of the domain and uploaded our keys to our register.
As such, sylnt.us now has a fully validated signature chain, and a green key when checked with the DNSSEC validator. We will be signing soylentnews.org sometime in the near future, however, we ran into some DNS zone transfer issues between our nameservers and Linode which caused the RRSIG records to not properly upload. While this has been resolved for now, we're currently talking with Linode to understand why the transfer went pear-shapped and to prevent a second occurrence.
That's it for now. As always, post questions, comments below. I'll be reading!
~ NCommander
(Score: 2) by VLM on Monday August 15 2016, @07:47PM
Since people seem to rather enjoy when I run articles on backend upgrades
Fun poll idea:
1) I do crazy admin stuff at home to someday get a sysadmin job in the future (remember when successfully setting up a linux box and installing apache got you an automatic hire job, like in '95 or '97 or so? That actually kinda happened to me...)
2) I sleeplessly herd cats on a regular basis as a paid sysadmin
3) I used to be a sysadmin until I pulled all my hair out
4) NCommander is my Sysadmin, or some nonsense option like that for those out of the admin fraternity
Back when you could still get a job doing CCNA/CCNP type stuff I did a lot of that kind of work too. I would imagine some kind of "hey baby talk BGP to me" would also get a response from this audience. Go ahead, multihome SN with a couple ISPs and BGP feeds, I haven't done that in over a decade but it used to be lots of fun. Do people still use ISIS? How about EIGRP? I'm so old that I remember when admins trusted other admins and didn't filter other admin's BGP routes or require LoA to advertise IP space... I'm so old I remember when some of the weirder OSPF stubby areas were invented. I remember when AS numbers were only 16 bits but there were only a couple thousand so it was all good. I'm so old I remember when someone invented the idea of a web accessible BGP looking glass.
(Score: 3, Touché) by The Mighty Buzzard on Monday August 15 2016, @07:50PM
But... NCommander actually is my sysadmin!
My rights don't end where your fear begins.
(Score: 2) by NCommander on Monday August 15 2016, @07:54PM
But I thought audioguy was your sysadmin who-
oh dear, I've gone cross-eyed
Still always moving
(Score: 3, Insightful) by NCommander on Monday August 15 2016, @07:51PM
1. I've had this happen recently actually.
2. That too
3. Helps when you shave it to 2 mm
4. I am my own sysadmin
I never have landed any work as a network sysop. With most things being in the cloud these days, network configuration is something most businesses don't need much beyond basic SoHo needs. Even on upwork, where I get a lot of my clients from, very rarely do I even see posts for an actual network job.
Still always moving
(Score: 0) by Anonymous Coward on Monday August 15 2016, @10:25PM
remember when successfully setting up a linux box and installing apache got you an automatic hire job
Yes. The requirements have changed today though. Now if you can successfully install apache on linux, you will automatically get the job, only if you're indian.