SoylentNews is people
SoylentNews
We had an hour or so or downtime today. After debugging, the root cause came from the SSL certificates we use to establish a database connection from the webserver to the actual DB. As a prelude GoLive, we migrated from unencrypted connections to encrypted connections as we have to cross the Linode internal LAN. In an attempt to improve data security, we generated a set of SSL certificates and used those to encrypt the MySQL connections. In the flurry of golive, no one thought to check the expiry date on said certificates. Out of the box, OpenSSL generates certificates with a one month expiry unless manually changed.
As you might expect, one month later, the certificates expired, and the database stopped accepting remote connections. New certificates were generated with a ten year expiration, and we continue to work towards better documenting our internal processes on the wiki to prevent this sort of thing from happening again. Apache, and slashd are running again, and we appear to be back to status-quo in terms of site operation.
A full incident report will be written up and posted to the wiki in the next few days.
As you might expect, one month later, the certificates expired, and the database stopped accepting remote connections. New certificates were generated with a ten year expiration, and we continue to work towards better documenting our internal processes on the wiki to prevent this sort of thing from happening again. Apache, and slashd are running again, and we appear to be back to status-quo in terms of site operation.
A full incident report will be written up and posted to the wiki in the next few days.
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Can anyone remember when the times were not hard, and money not scarce?
(Score: 1) by Magic Oddball on Friday March 14 2014, @11:08AM
Yikes -- thank you for working on this through vacation, let alone while in a totally different part of the world from most (all?) of us.
The thing to keep in mind during the "oh SHIT"Â moments is that most (if not all) of the visitors here have the basic knowledge needed to have realistic expectations. :-)
Adding after a preview: any odd characters alongside spaces in my posts are because of some odd bug in Slashcode that evidently only my system sets off.
(Score: 4, Informative) by NCommander on Friday March 14 2014, @12:12PM
The distortion is due to Slash's blasted UTF-8 bitrot. We enabled it during testing, but it was buggy. There's no "off" switch for UTF-8, so my guess is whatever magic the other site never got committed to the public branch, as there's no filter that I can find in the public codebase. Fixing UTF-8 to work properly remains on the TODO, but at least it semi-works if you're careful.
And yeah, my travel schedule was epically ill timed. The management handover happened while I was at a conference in Macau, so I've been running around like a chicken without a head.
Still always moving
(Score: 2) by Pslytely Psycho on Friday March 14 2014, @06:16PM
"so I've been running around like a chicken without a head."
Should we start calling you Mike then?
so I've been running around like a chicken without a head.
Alex Jones lawyer inspires new TV series: CSI Moron Division.
(Score: 2) by Pslytely Psycho on Friday March 14 2014, @08:52PM
OOPS.....Preview, preview preview
"so I've been running around like a chicken without a head."
Should we start calling you Mike then?
http://en.wikipedia.org/wiki/Mike_the_Headless_Chi cken [wikipedia.org]
Alex Jones lawyer inspires new TV series: CSI Moron Division.
(Score: 2) by zigbigadoorlue on Friday March 14 2014, @07:59PM
Good gracious you all are doing a lot of good work for free (and on your vacation!). Do you have a full time job in addition to running this marvelous and confounded site? You all are doing an excellent job particularly as you are currently not getting payed for any of it. Thanks for all that you've given this community.
(Score: 2) by NCommander on Saturday March 15 2014, @02:04AM
I can't speak for anyone else, but I work full time in FOSS technologies. This vacation was setup before I was involved with SoylentNews, which has caused me a lot of grief in hindsight (but then again, hindsight is always 20/20). I've been trying to manage to site, my sanity, and a crazy travel schedule all at once, but I've cleared out my schedule until September to try and get the business side of things assembled.
Still always moving