I just finished updating the certs for SoylentNews.
We get our certs through Let's Encrypt. Yes, we could automate the whole process, but it has been discussed and decided that given our... unique configuration, it is best to have a human in the loop than to let a script somehow run amok and then try to restore things when who-all-knows-what got deployed and things have gone sideways.
I have checked our web sites for production, dev, and staff as well as sending and retrieving e-mail; all seemed to be okay.
More than anything else, this is a check on us to see if we (well, me, actually) overlooked anything. If you do detect any issues, please post a comment to this story.
(Hat tip to The Mighty Buzzard for standing by in case I bollixed up something.)
[Update: Unless, of course, you cannot post a comment to this story! Then pop onto the #Soylent channel on our Internet Relay Chat (IRC) server and let us know over there. --martyb]
(Score: 3, Funny) by Gaaark on Thursday August 22 2019, @12:15PM (6 children)
SoylentNews's Site's Certs Updated Today SoylentNews's Site's Certs Updated Today SoylentNews's Site's Certs Updated Today SoylentNews's Site's Certs Updated Today SoylentNews's Site's Certs Updated Today
Martyb sells SN shells by the handful?
--- Please remind me if I haven't been civil to you: I'm channeling MDC. ---Gaaark 2.0 ---
(Score: 4, Funny) by martyb on Thursday August 22 2019, @01:03PM (4 children)
Thanks for the laugh!
Was my first time doing the cert upgrade solo, so I was a wee bit nervous. But now that I had a much better idea of what was going on, it went smoothly, all-in-all. That said, this is when things tend to start getting "exciting" when I *think* that I know what is going on! So, out of an abundance of caution, I put this story out so as to catch any shortcomings as soon as possible.
P.S. I have no SN shells to sell; see C shell seller Sally's cellar instead.
Wit is intellect, dancing.
(Score: 2) by Fnord666 on Thursday August 22 2019, @02:55PM (2 children)
And you documented everything in detail and posted it on the wiki, right?
(Score: 2) by martyb on Thursday August 22 2019, @03:52PM (1 child)
Already done; there is a staff "tech wiki" where TMB had posted the instructions and which I was dutifully following.
Wit is intellect, dancing.
(Score: 2) by Fnord666 on Thursday August 22 2019, @11:27PM
(Score: 2) by Gaaark on Thursday August 22 2019, @10:03PM
Thank you for the job well done!
See C shell Sally sells her what?
--- Please remind me if I haven't been civil to you: I'm channeling MDC. ---Gaaark 2.0 ---
(Score: 2) by DeathMonkey on Thursday August 22 2019, @05:28PM
Superb cert service, Sirs!
(Score: 2) by janrinok on Thursday August 22 2019, @01:21PM
I can't post a comment because I can see nothing wrong. Thanks martyb.
(Score: 2) by isostatic on Thursday August 22 2019, @01:54PM (1 child)
If a task is difficult, do it more often
(Score: 2) by martyb on Thursday August 22 2019, @02:57PM
Our certs from Let's Encrypt (LE) are free. That fits in well with our [limited] budget.
As you can see from these URLs (cannot seem to get them to appear as actual links, sorry!):
LE certs are good for up to 90 days. We still had a couple weeks' time before the old certs were due to expire.
It is my understanding that LE certs cannot be renewed at less than 60 days (or so) after they were issued.
In short, the certs were updated earlier than was needed. And, to clarify, it is not so much that upgrading certs is difficult, but that it is somewhat tedious and exacting.
Wit is intellect, dancing.
(Score: 0) by Anonymous Coward on Thursday August 22 2019, @02:35PM (2 children)
If you're not going to automate, why not spend $14 and only do it every 2 years instead of every 90 days?
(Score: 2) by Fnord666 on Thursday August 22 2019, @02:57PM
Is there a reputable CA that sells two year site certs with SANs for $14?
(Score: 4, Informative) by martyb on Thursday August 22 2019, @03:13PM
In addition to the preceding, sibling comment, I would add that our needing to regularly update the certs keeps it in the collective site-maintenance mind share. After, say, 18 months' time, who is going to be thinking about when the certs are going to expire? We've reported here several stories where even large multinational companies have inadvertently let domain registrations, certs, etc. expire and were soundly ridiculed for doing so.
Also, this is not within my usual area of expertise, but I am willing to learn and expressed an interest to TheMightyBuzzard. He kindly wrote up some instructions and watched over my shoulder the first couple of times I did the cert updates. This time, I let him know I was doing the update and he basically just wished me well.
Analogy time, think of the first couple of programs you wrote in a new programming language, but with the understanding that a coding error could cause thousands of people to be unable to access a web site. And IRC. And our e-mail. And... you get the idea.
And let me take a moment to mention how fortunate SoylentNews is to have the team of sysops we have. They quietly take care of the low-level plumbing that is needed for this site to work. Load balancers, web server, database servers, e-mail, IRC, the list goes on. It is a testament to their skill and expertise that this site — which originally would crash several times each day at the beginning — now regularly goes months without even a hiccup!
Wit is intellect, dancing.
(Score: 2) by SomeGuy on Thursday August 22 2019, @02:43PM (3 children)
Will there be a more detailed write up about that big crash the other day? Or has it just been chalked up to a Rube Goldberg-esq derp?
(Score: 2) by martyb on Thursday August 22 2019, @09:41PM (2 children)
As much as I would VERY much like to find out what happened, I doubt that will happen. As best as I can recall off the top of my (balding) head, SemperOSS and Chromas were first on the scene and had pursued a few attempts at problem isolation and remediation with janrinock standing by and cheerleading. I was next on the scene, but could do little more than to join the cheerleading team. We realized we were stymied and resigned ourselves to waiting for additional help. I put out a ping on the staff-only channel looking for assistance, but everyone was sleeping at the time, as they (and we!) should have been.
Like clockwork, TheMightyBuzzard (TMB) woke up and arrived at his usual time and was briefed on the situation. He proceeded with some ideas and put up a "we know the site is down" page. About a half hour into attempting problem isolation came the decision: "We pay for backups on this server, might as well get our money's worth." (The site had been down for something like 4 or 5 hours at this point.) The backup was unceremoniously dropped on the recalcitrant server, TMB hit the figurative BRS (Big Red Switch), and restarted the system. Had a couple minor hiccups at first in getting the right services going and then we were back up and running.
In retrospect, it would have been nice to have saved an image of the state-of-the-world before restoring the backup, but by the time I realized it, it was too late.
tl;dr: it was a "Rube Goldberg-esq derp".
Wit is intellect, dancing.
(Score: 2) by AthanasiusKircher on Thursday August 22 2019, @11:15PM (1 child)
Well, thanks again to martyb, and to the rest of the team.
I'll just put in another plug for my attempt [soylentnews.org] to drive some new subscriptions to this site (as well as people already subscribed to donate a bit more). There's a link in that journal post to a more detailed explanation, as my primary motivation was to build on goodwill after this very quick site recovery last week.
Unfortunately, the comments devolved into a crapfest of BS argumentation, which honestly makes me sad.
(Score: 2) by martyb on Friday August 23 2019, @02:53AM
I very much appreciate your most generous subscription and attempt to build interest in others doing the same. I wish you the best on that one!
What I am unable to contribute in money, I have tried to make up for in other ways. Obviously, my contributions as an editor here stands out, but there are other things as well. I don't have nearly enough time to actually read the site as I would like, but when I do, I am also mindful that performing comment moderation is a contribution, too. We hand out 10 mod points every day at 00:10 UTC &mdash for a reason: use them to help your fellow soylentils to find the wheat amongst the chaff.
Another helpful contribution is to post a comment. Fact-filled, reasoned comments are a real gift for me to read in this day and age of fake news.
Lastly, submit a story! Was reading something on-line that you thought was interesting, informative, or entertaining and that had some kind of technological angle to it? Please! Send. It. In. The Submit Story [soylentnews.org] link appears on the left-hand side of the main page (and many other pages, too). On the submission page is a link to suggestions on how to craft a well-written story and, thus, improve the chances of its getting accepted for posting to the site.
Wit is intellect, dancing.
(Score: 2) by maxwell demon on Thursday August 22 2019, @05:48PM
Probably it is not related to the update, but several times today I got the browser message that soylentnews.org could not be connected. It always works immediately fine on retry.
I can't completely exclude a local problem, but since only soylentnews is affected, I suspect a problem of the site.
The Tao of math: The numbers you can count are not the real numbers.
(Score: 1, Insightful) by Anonymous Coward on Thursday August 22 2019, @06:00PM
After over 10 years of browsers screaming blue murder over self-signed certs, and several high profile cases of de-factor censorship via cert-withdrawals (e.g. Sci-hub), I've turned off the idea of site certification being a good thing.
The usual mantra uttered here is "Security = Encryption + Authentication". But we've learned to our cost that "Authentication = Money*Money - Censorship^Politics".
Like a lot of modern "meta-site" infrastructure, Certs are becoming a mandatory but less than reliable requirement for running a website, increasing both cost and complexity and making simple, small scale websites ever less feasible. An Internet protection rent charged by third parties as a cost of getting your content online "on your own computer". All websites used to need was a PC in a garage and a domain name.
(Score: 0) by Anonymous Coward on Thursday August 22 2019, @06:55PM
Exactly the kind of thing that Expect (https://core.tcl-lang.org/expect/index) was designed to handle.
You just have to create a rollback system to be used in the case that anything does go sideways.