from the fiddling-for-the-greater-good dept.
Since people seem to rather enjoy when I run articles on backend upgrades, here's another set of changes I made over the last week as I get back into the full swing working on the site.
The short list:
- Migrated Beryllium (which hosts wiki+IRC+mail) to Apache 2.4
-
- Upgraded said machine to PHP7
- Needed to support OCSP stapling
- Validating final checks before deploying HSTS to all public domains
- Upgraded MediaWiki, SquirrelMail, and YOURLS to PHP7 compatible versions
- Worked with TheMightyBuzzard and user comments to determine additional XSS protection headers we should deploy
- Found (and removed) SSLv3 support on postfix and dovecot
- Deployed DNSSEC on sylnt.us in preparation for signing soylentnews.org (here's the test results)
Read past the fold for more information.
Beryllium Upgrades
Beryllium is our "misc" services box. It basically hosts everything that isn't related to site infrastructure such as the wiki, our IRC server, and mail. Last week, I went through and fixed our SSL configuration on this machine to make sure that we were serving properly validated certificates, and that we had strong encryption on this box. While I succeeded on that front, for performance reasons, Apache 2.4 needed to be upgraded to support a somewhat obscure feature of TLS known as OCSP stapling.
What is OCSP stapling you ask? Well, to answer that, I need to take a moment to go into how SSL certificates work. Whenever a CA generates a certificate, they're essentially saying "this site is who it is and we're attesting to it". In a perfect world, a CA would never make a mistake, private keys would never leak, and we could always assume that a certificate is good. We don't live in that world, as such certificate authorities sometimes need to void a certificate. OCSP (which stands for the Online Certificate Status Protocol) is one of two ways to do this, and is the only method Let's Encrypt supports for certificate revocation.
OCSP is a replacement for older certificate revocation lists (CRLs) which in real-life rarely if ever worked as advertised. It's meant to allow the browser to update in real-time knowledge if a certificate is good or bad and react accordingly. OCSP however requires that the browser checks with a certificate authority's OCSP server, leaking the fact that user X is connecting to site Y. It also means that if access to the OCSP server is blocked, a user might not be aware that a certificate has been revoked. OCSP stapling solves both problems by having our servers grab the OCSP reply (which is timestamped), and sending it as part of the initial connection to our site, both increasing performance, and preventing a privacy leak.
Unfortunately, OCSP stapling requiring Apache 2.4 which required me to build it from source, and then migrate sites over from the older Apache 2.2 install. At the same time, I went through and upgraded PHP 7, and updated the other web applications we were using. For the most part, this was rather painless though I'm still tinkering with MediaWiki to make it happy on the new setup.
Beside the usual Apache pain, I went through and scanned our other major services and disabled SSLv3 support on postfix (SMTP) and dovecot. I need to go through and replace our self-signed certs with real ones here but that's a 'one step at a time thing'
XSS Mitigation
During the last site status article, an AC pointed us at this handy site showing security headers. As such, TheMightyBuzzard and I will be going through and enabling these (with the exception of public key pinning) on production sometime this week. HPKP requires quite a bit of planning to deploy and we're not ready to take that step just yet.
DNSSEC + sylnt.us
I've talked about wanted to deploy DNSSEC before, but various other things kept cropping up. That, and combined with outdated and misleading documentation kept me from actually getting around to doing this for ages. Over the weekend, I finally dug down and figured out the current best practices for DNSSEC, and with the help of audioguy, configured BIND to do automatic signing of the domain and uploaded our keys to our register.
As such, sylnt.us now has a fully validated signature chain, and a green key when checked with the DNSSEC validator. We will be signing soylentnews.org sometime in the near future, however, we ran into some DNS zone transfer issues between our nameservers and Linode which caused the RRSIG records to not properly upload. While this has been resolved for now, we're currently talking with Linode to understand why the transfer went pear-shapped and to prevent a second occurrence.
That's it for now. As always, post questions, comments below. I'll be reading!
~ NCommander
Related Stories
So, during the last site update article, a discussion came up talking about how those who work and write for this site should get paid for said work. I've always wanted to get us to the point where we could cut a check to the contributors of SoylentNews, but as it stands, subscriptions more or less let us keep the lights on and that's about it.
As I was writing and responding to one specific thread, part of me started to wonder if there would be enough interest to try and crowdfund articles on specific topics. In general, meta articles in which we talk deploying HSTS or our use of Hesiod tend to generate a lot of interest. So, I wanted to try and see if there was an opportunity to both generate interesting content, and help get some funds back to those who donate their time to keep the lights on.
One idea that immediately comes to mind that I could write is deploying DNSSEC in the real world, and an active example of how it can help mitigate hijack attacks against misconfigured domains. Alternatively, on a retro-computing angle, I could cook something in 16-bit real mode assembly that can load an article from soylentnews.org. I could also do a series on doing (mostly) bare metal work; i.e., loading an article from PXE boot or UEFI.
However, before I get in too deep into building this idea, I want to see how the community feels about it. My initial thought is that the funds raised for a given article would dictate how long it would be, and the revenue would be split between the author, and the staff, with the staff section being divided at the end of the year as even as possible. The program would be open to any SN contributor. If the community is both interested and willing, I'll organize a staff meeting and we'll do a trial run to see if the idea is viable. If it flies, then we'll build out the system to be a semi-regular feature of the site
As always, leave your comments below, and we'll all be reading ...
~ NCommander
(Score: -1, Troll) by Anonymous Coward on Monday August 15 2016, @07:11PM
Your tireless dedication to the greatest of all social news sites is appreciated as always.
(Score: 3, Interesting) by Scruffy Beard 2 on Monday August 15 2016, @07:30PM
So are we supposed to mod the comments, titles or both?
Only the Parent's subject seems to be flamebait.
(Score: 0) by Anonymous Coward on Monday August 15 2016, @07:35PM
You forgot the "social" slur in the comment.
(Score: 0) by Anonymous Coward on Monday August 15 2016, @09:06PM
Well, at least there was no justice.
(Score: 4, Funny) by edIII on Monday August 15 2016, @07:48PM
I think it was just a very polite attempt at a compliment from one of our members suffering from Tourette's syndrome, and beaten as a child. Hopefully.
Awesome work on the upgrades guys!
Technically, lunchtime is at any moment. It's just a wave function.
(Score: 2) by VLM on Monday August 15 2016, @07:47PM
Since people seem to rather enjoy when I run articles on backend upgrades
Fun poll idea:
1) I do crazy admin stuff at home to someday get a sysadmin job in the future (remember when successfully setting up a linux box and installing apache got you an automatic hire job, like in '95 or '97 or so? That actually kinda happened to me...)
2) I sleeplessly herd cats on a regular basis as a paid sysadmin
3) I used to be a sysadmin until I pulled all my hair out
4) NCommander is my Sysadmin, or some nonsense option like that for those out of the admin fraternity
Back when you could still get a job doing CCNA/CCNP type stuff I did a lot of that kind of work too. I would imagine some kind of "hey baby talk BGP to me" would also get a response from this audience. Go ahead, multihome SN with a couple ISPs and BGP feeds, I haven't done that in over a decade but it used to be lots of fun. Do people still use ISIS? How about EIGRP? I'm so old that I remember when admins trusted other admins and didn't filter other admin's BGP routes or require LoA to advertise IP space... I'm so old I remember when some of the weirder OSPF stubby areas were invented. I remember when AS numbers were only 16 bits but there were only a couple thousand so it was all good. I'm so old I remember when someone invented the idea of a web accessible BGP looking glass.
(Score: 3, Touché) by The Mighty Buzzard on Monday August 15 2016, @07:50PM
But... NCommander actually is my sysadmin!
My rights don't end where your fear begins.
(Score: 2) by NCommander on Monday August 15 2016, @07:54PM
But I thought audioguy was your sysadmin who-
oh dear, I've gone cross-eyed
Still always moving
(Score: 3, Insightful) by NCommander on Monday August 15 2016, @07:51PM
1. I've had this happen recently actually.
2. That too
3. Helps when you shave it to 2 mm
4. I am my own sysadmin
I never have landed any work as a network sysop. With most things being in the cloud these days, network configuration is something most businesses don't need much beyond basic SoHo needs. Even on upwork, where I get a lot of my clients from, very rarely do I even see posts for an actual network job.
Still always moving
(Score: 0) by Anonymous Coward on Monday August 15 2016, @10:25PM
remember when successfully setting up a linux box and installing apache got you an automatic hire job
Yes. The requirements have changed today though. Now if you can successfully install apache on linux, you will automatically get the job, only if you're indian.
(Score: 0) by Anonymous Coward on Monday August 15 2016, @08:08PM
Did the update to MediaWiki break some templates? Take a look at https://wiki.soylentnews.org/wiki/WhosWho [soylentnews.org]
(Score: 2) by NCommander on Monday August 15 2016, @08:16PM
Yes, yes it did. I just haven't gotten humpy-dumpy fixed yet. the wiki doesn't get much use so its been fairly low priority but I'll put my ass on it tonight to fix it.
Still always moving
(Score: 1) by gmrath on Tuesday August 16 2016, @02:17AM
"I'll put my ass on it tonight. . ."
Somehow the image of the proverbial lard-ass hopping onto an over-stuffed suitcase to squash it closed pops into imagination, and that once imagined, cannot be un-imagined. . .
I wish you good luck on the fix. Yes. Yes I do.
(Score: 2) by NCommander on Tuesday August 16 2016, @03:21AM
I got the templates fixed right now. The checkuser module is broken upstream and abandoned so I had to rip out its hooked but the wiki is now about 90% of the way fixed.
Still always moving
(Score: 0, Troll) by shanen on Monday August 15 2016, @10:14PM
What is your time worth? Did you get appropriately compensated for the time you spent? If not, why did you spend the time and why should we expect you to spend more time?
Perhaps more importantly, since you've already done these things, why should they matter to us?
Perhaps most importantly, how do you know that any of these things (or future improvements) actually do matter to us?
My reaction to looking over the list of improvements is that I would not chip in to support any of them, but perhaps that's just because you were not trying to sell them. I don't see anything in there about why these improvements are such good things that they will make Soylent News better from my perspective--but remember that "my perspective" does NOT matter. It's "our perspective", referring to the readers in a collective sense, that should matter.
I continue to regard Soylent News as a nice idea that is going nowhere. Not even a brisk pace in the non-movement, but I can't blame you. Too bad there is no mechanism to do better, eh? (I think there is, but it seems I'm always in the wrong time zone. My clock is several years fast again.)
#1 Freedom = (Meaningful - Coerced) Choice{5} ≠ (Beer^4 | Speech) and your negative mods prove you are a narrow prick.
(Score: 2, Insightful) by kramulous on Monday August 15 2016, @11:34PM
Somebody doesn't keep up2date with security.
Sometimes you do things for the love of it. Oh, and it improves your general skills and knowledge that you then apply to your main profession. I guess it is professional development.
What is that worth?
(Score: 0, Troll) by shanen on Tuesday August 16 2016, @09:56AM
You are missing the point. I think he deserves to get paid for his hard work.
Now if you have donated to support him and his work, then I do congratulate you. If you didn't then you are only proving my point. Or do you want to argue that he should take a vow of poverty?
Funny story about security. Based on a few minor pieces I wrote for IEEE Computer before I joined the Big Blue food chain, some of my coworkers (especially in the research lab) thought I was a computer security expert. I am NOT and never was, though I still enjoy reading such relevant books as the recent Data and Goliath by Bruce Schneier.
From his lingo-filled description, I didn't even know that any of the improvements were related to security, but in that case they should NOT be waiting for his convenience. If there had been a support project listing urgent security upgrades, then I certainly hope enough members would have chipped in to get them implemented ASAP. Security updates should actually be prefunded, and I might chip in on that basis because I know that remedial steps after you've been breached are much more expensive. Or aren't you keeping up with security?
#1 Freedom = (Meaningful - Coerced) Choice{5} ≠ (Beer^4 | Speech) and your negative mods prove you are a narrow prick.
(Score: 2) by JNCF on Tuesday August 16 2016, @02:03PM
You are missing the point. I think he deserves to get paid for his hard work.
Here you go. [soylentnews.org] It might not go directly in his pockets*, but it's clearly where those contributing time (I'm not one of them) would like you to throw your money.
From his lingo-filled description, I didn't even know that any of the improvements were related to security,
Really? This lingo was too dense for you (emphasis original): "XSS Mitigation During the last site status article, an AC pointed us at this handy site showing security headers."
Methinks somebody didn't read the post.
but in that case they should NOT be waiting for his convenience. If there had been a support project listing urgent security upgrades, then I certainly hope enough members would have chipped in to get them implemented ASAP. Security updates should actually be prefunded, and I might chip in on that basis because I know that remedial steps after you've been breached are much more expensive.
Here you go, again. [soylentnews.org] The more funds there are, the more on top of issues like this you can logically expect them to be.
*Or it might go to his pockets, I don't know if he's currently taking a loss or not. I think he was originally, but he might have been paid back for it.
(Score: 1) by shanen on Tuesday August 16 2016, @10:06PM
For someone writing on a journalism-related website, my main conclusion is that you [JNCF] don't write very clearly. I'll try to deal with your "objections" as well as I can understand them.
I am not saying that essentially blind subscriptions are a terrible business model. Mostly I am saying that I want more transparency. Among other reasons, I am not so rich that I can throw money at every good cause that wants more money.
My primary suggestion is for an alternative funding mechanism that would allow small donors to see more about how their money is actually used. Perhaps the larger value of the approach is that the same mechanism would drive documentation of the features and services that the website is providing. The project proposals that get funded should become accurate descriptions of what exists (as long as the success criteria are satisfied).
If I were rich, then the shoe would be on the other foot. There are plenty of problems with big-donor financial models, but one thing you can be sure of is that big donors get to see how their money is being spent. I'm suggesting an approach to let small donors have a bit of that.
Sometimes I do donate to support journalism. For example, I am quite sure that I donated some money to the website that broke Romney's 47% comment. If I were going to subscribe, then that is probably one of the top candidates. SN in its current form would not be in the list of my top ten candidates.
By the way, your example from the original article was an example of poor writing. How do you interpret your selection to indicate a security problem on SN? Of course XSS is a hint, but even the replacement with "cross-site scripting mitigation" would be ambiguous. (And no, I did not have to look that up. I even edited some technical papers for security experts who were working on related problems.)
#1 Freedom = (Meaningful - Coerced) Choice{5} ≠ (Beer^4 | Speech) and your negative mods prove you are a narrow prick.
(Score: 2) by JNCF on Tuesday August 16 2016, @10:59PM
For someone writing on a journalism-related website, my main conclusion is that you [JNCF] don't write very clearly.
I have never been fond of most journalistic writing, but I see your point about subjects. To clarify, "he" meant NCommander. There are more reasons to choose words and phrasings than clarity alone. When my intention is strictly clarity, I am very clear. This is rarely the case on SoylentNews.
I am not so rich that I can throw money at every good cause that wants more money.
Me neither. I really like SoylentNews, even with all its nasty warts. It's certainly not for everyone, and donation is totally optional. You seem very concerned about funding, though.
My primary suggestion is for an alternative funding mechanism that would allow small donors to see more about how their money is actually used. Perhaps the larger value of the approach is that the same mechanism would drive documentation of the features and services that the website is providing. The project proposals that get funded should become accurate descriptions of what exists (as long as the success criteria are satisfied).
Maybe this is a good suggestion. Maybe not. It doesn't seem very fleshed out. Rereading your original posts in this discussion, the idea doesn't even seem to be present. It seems like a much more general complaint that touches on a number of other issues. For what it's worth, I modded you "underrated" -- not because I agreed with anything you said, but because I didn't think you were trolling.
By the way, your example from the original article was an example of poor writing. How do you interpret your selection to indicate a security problem on SN? Of course XSS is a hint, but even the replacement with "cross-site scripting mitigation" would be ambiguous. (And no, I did not have to look that up. I even edited some technical papers for security experts who were working on related problems.)
The emphasis was original. I felt that the heading added context to the sentence that came after it, and I wanted to clearly identify it as a heading without breaking it into a multi-line block quote as I was using for your text. If you go back and read the rest of the quote, it uses the word "security." That was the lingo you either couldn't interpret as being related to security, or didn't read.
(Score: 1) by shanen on Wednesday August 17 2016, @12:33AM
I have to leave soon so I'm just going to focus on a few main points. The funding suggestion was much more fleshed out in some of my earlier comments on SN, after I had observed the system for a few weeks. It has also been fleshed out in various other venues, often modified for more specific purposes. The original form was called RACS (Reverse Auction Charity Shares) because of some viral marketing aspects involving discounted shares, but these days I usually start from the more generic perspective of "charity share brokerage", focusing on the key entity that is holding the money. They show up in some search engines, but I don't know of any so-called early adopters. (There's also some stuff I wrote about couch potatoes...)
The reason I focus on the funding is not because I think money is important, but because it influences behavior. People tend to respond to clear incentives, and monetary incentives tend to quite clear. If an organization is getting its funding from certain stakeholders, then it will natural evolve to focus on the needs and preferences of those stakeholders over the others. The model I am suggesting would make "wannabe problem solvers" into key stakeholders and solving problems would be linked to the organization's success, including its success in funding itself.
Getting diverted again, but I think economics sucks, to put it politely. Currently reading Seven Bad Ideas along those lines. My proposed solution is called ekronomics, perhaps a form of time-based economics.
#1 Freedom = (Meaningful - Coerced) Choice{5} ≠ (Beer^4 | Speech) and your negative mods prove you are a narrow prick.
(Score: 2) by NCommander on Tuesday August 16 2016, @07:44PM
None of these updates are critical. I'm realistic that much of what I do is relatively low value, and to be more blunt, there's very little of value here for someone to break into. At best someone could supposedly grab an admin's credentials in flight and delete all the stories but that's why we have backups.
Back when Heartbleed happened (and we (and most of the internet)) were affected, I believe we had the certs changed out within 24 hours. Critical security stuff does happen timely, but this is mostly hardening and increasing the armor plating vs. replacing a hole.
Still always moving
(Score: 1) by shanen on Tuesday August 16 2016, @09:06PM
I'm glad to hear that, but I still think you deserve to be well paid for good work, and you didn't really address my original question. Putting too much reliance on the Subject: line?
Let me clarify that I am not interested in the real cost of those improvements from a managerial perspective. You don't want to get me started on my low opinion of some managers... However, I think that Soylent News is supposed to be or wants to be a new kind of journalism, and some of that involves project management, including security-related projects.
In this particular case, I think that security should be a high-priority ongoing-cost project. Perhaps this particular work would have justified a special implementation project, too? However, my focus is that someone ought to get paid for the skills.
Regarding the criticality of these updates, I prefer to err on the side of "Better safe than sorry." I actually think the real threat of a breach on Soylent News would be something like a malware installer, and from that perspective SN seems to be a low priority target because it seems to be a low traffic website. If one of the goals is to increase the traffic and influence, I think that would be good, but it would also increase the value of the website as a target for attack--and in that case the greatest threat might be an attacker who is planning ahead and installing backdoors now.
#1 Freedom = (Meaningful - Coerced) Choice{5} ≠ (Beer^4 | Speech) and your negative mods prove you are a narrow prick.
(Score: 2) by NCommander on Tuesday August 16 2016, @10:08PM
Speaking candidly, personal issues really prevented me from putting the effort in to building out the site as I planned w/ more original journalism, plus a somewhat lukewarm response from the community. We'll run original articles if someone submits one but thats once in a bluemoon sorta thing.
I've thought about wondering ways we can bring in more money for SN; maybe a hosted DNS stuff which is DNSSECed or something but to be honest, I dunno if we could really make much of anything doing that beyond what subscriptions bring in ...
Still always moving
(Score: 1) by shanen on Tuesday August 16 2016, @10:36PM
Well, I don't know if you ever saw my much ballyhooed suggestion, but I'll recap it in the context of your specific example for this thread.
You would have begun by preparing a summary of the work as a project proposal. Since it was a security-related project, I think the links to the project proposal should have been featured pretty prominently on SN. Members who saw the proposal would see what you wanted to do, your schedule, your compensation, the testing plans (which I believe to be important in every software proposal but even more so when it comes to security, even if it's just a code walk-through with another programmer), and the success criteria. When enough members have chipped in, then the money would be released and you would do the project, and after it was finished, the results would be evaluated and reported to the donors.
The funding mechanism I suggested could be described as a "charity share brokerage". As a supporter of SN, I could donate some money to my account, and periodically buy shares in projects that I like. If enough people agree with me, then my projects get funded, but if I pick a loser project, then it runs past its funding schedule without getting funded and I can pledge the money to some other project. (Of course the people who submit the unfunded project can try to improve it and submit it again.)
Not sure what part of this idea caused so much umbrage. Perhaps the idea of going beyond internal projects to actually help fix problems in the real world? Hey, sorry, but that's where I live.
#1 Freedom = (Meaningful - Coerced) Choice{5} ≠ (Beer^4 | Speech) and your negative mods prove you are a narrow prick.
(Score: 2) by NCommander on Tuesday August 16 2016, @11:04PM
Hrm ... I could def. see how it could work for some things. Right now, a lot of this is just basic site maintence; updating old software and such, but this gives an idea I think I need to pitch to the community and such.
Still always moving
(Score: 1) by shanen on Wednesday August 17 2016, @12:21AM
You're welcome to any part of it you can use, but I've been thinking about variations of alternative funding models long before I ever heard of crowdfunding. There are a number of variations that might be relevant or suitable for journalistic purposes.
Lord knows the existing models of journalism are totally broken. Primarily disaster porn and fake reality shows for eyeballs to sell to advertisers driving journalism to depths of perfidy never before seen. That's just the stage setting, but the real problems are the bad actors like terrorists and the Donald who exploit the broken systems.
#1 Freedom = (Meaningful - Coerced) Choice{5} ≠ (Beer^4 | Speech) and your negative mods prove you are a narrow prick.
(Score: 2) by NCommander on Wednesday August 17 2016, @12:38AM
There's an article in the queue set to go live at 8AM EST talking about seeing if we can get a funding model based on posts like this. If it flys with the community, it could effectively kill three birds with one stone: getting interesting original content on the site, getting those high-labor updates out the door, and getting money to the contributors.
Still always moving
(Score: 1) by shanen on Wednesday August 17 2016, @09:25PM
I thought I was watching for it, but either I missed it or it didn't go live? How about a link?
#1 Freedom = (Meaningful - Coerced) Choice{5} ≠ (Beer^4 | Speech) and your negative mods prove you are a narrow prick.
(Score: 5, Touché) by fleg on Tuesday August 16 2016, @05:56AM
>Soylent News as a nice idea that is going nowhere.
can i get your coat for you? call a taxi perhaps?