SoylentNews is people
SoylentNews
I just finished updating the certs for SoylentNews.
We get our certs through Let's Encrypt. Yes, we could automate the whole process, but it has been discussed and decided that given our... unique configuration, it is best to have a human in the loop than to let a script somehow run amok and then try to restore things when who-all-knows-what got deployed and things have gone sideways.
I have checked our web sites for production, dev, and staff as well as sending and retrieving e-mail; all seemed to be okay.
More than anything else, this is a check on us to see if we (well, me, actually) overlooked anything. If you do detect any issues, please post a comment to this story.
(Hat tip to The Mighty Buzzard for standing by in case I bollixed up something.)
[Update: Unless, of course, you cannot post a comment to this story! Then pop onto the #Soylent channel on our Internet Relay Chat (IRC) server and let us know over there. --martyb]
(Score: 2) by martyb on Thursday August 22 2019, @09:41PM (2 children)
As much as I would VERY much like to find out what happened, I doubt that will happen. As best as I can recall off the top of my (balding) head, SemperOSS and Chromas were first on the scene and had pursued a few attempts at problem isolation and remediation with janrinock standing by and cheerleading. I was next on the scene, but could do little more than to join the cheerleading team. We realized we were stymied and resigned ourselves to waiting for additional help. I put out a ping on the staff-only channel looking for assistance, but everyone was sleeping at the time, as they (and we!) should have been.
Like clockwork, TheMightyBuzzard (TMB) woke up and arrived at his usual time and was briefed on the situation. He proceeded with some ideas and put up a "we know the site is down" page. About a half hour into attempting problem isolation came the decision: "We pay for backups on this server, might as well get our money's worth." (The site had been down for something like 4 or 5 hours at this point.) The backup was unceremoniously dropped on the recalcitrant server, TMB hit the figurative BRS (Big Red Switch), and restarted the system. Had a couple minor hiccups at first in getting the right services going and then we were back up and running.
In retrospect, it would have been nice to have saved an image of the state-of-the-world before restoring the backup, but by the time I realized it, it was too late.
tl;dr: it was a "Rube Goldberg-esq derp".
Wit is intellect, dancing.
(Score: 2) by AthanasiusKircher on Thursday August 22 2019, @11:15PM (1 child)
Well, thanks again to martyb, and to the rest of the team.
I'll just put in another plug for my attempt [soylentnews.org] to drive some new subscriptions to this site (as well as people already subscribed to donate a bit more). There's a link in that journal post to a more detailed explanation, as my primary motivation was to build on goodwill after this very quick site recovery last week.
Unfortunately, the comments devolved into a crapfest of BS argumentation, which honestly makes me sad.
(Score: 2) by martyb on Friday August 23 2019, @02:53AM
I very much appreciate your most generous subscription and attempt to build interest in others doing the same. I wish you the best on that one!
What I am unable to contribute in money, I have tried to make up for in other ways. Obviously, my contributions as an editor here stands out, but there are other things as well. I don't have nearly enough time to actually read the site as I would like, but when I do, I am also mindful that performing comment moderation is a contribution, too. We hand out 10 mod points every day at 00:10 UTC &mdash for a reason: use them to help your fellow soylentils to find the wheat amongst the chaff.
Another helpful contribution is to post a comment. Fact-filled, reasoned comments are a real gift for me to read in this day and age of fake news.
Lastly, submit a story! Was reading something on-line that you thought was interesting, informative, or entertaining and that had some kind of technological angle to it? Please! Send. It. In. The Submit Story [soylentnews.org] link appears on the left-hand side of the main page (and many other pages, too). On the submission page is a link to suggestions on how to craft a well-written story and, thus, improve the chances of its getting accepted for posting to the site.
Wit is intellect, dancing.