SoylentNews is people
SoylentNews
We had an hour or so or downtime today. After debugging, the root cause came from the SSL certificates we use to establish a database connection from the webserver to the actual DB. As a prelude GoLive, we migrated from unencrypted connections to encrypted connections as we have to cross the Linode internal LAN. In an attempt to improve data security, we generated a set of SSL certificates and used those to encrypt the MySQL connections. In the flurry of golive, no one thought to check the expiry date on said certificates. Out of the box, OpenSSL generates certificates with a one month expiry unless manually changed.
As you might expect, one month later, the certificates expired, and the database stopped accepting remote connections. New certificates were generated with a ten year expiration, and we continue to work towards better documenting our internal processes on the wiki to prevent this sort of thing from happening again. Apache, and slashd are running again, and we appear to be back to status-quo in terms of site operation.
A full incident report will be written up and posted to the wiki in the next few days.
As you might expect, one month later, the certificates expired, and the database stopped accepting remote connections. New certificates were generated with a ten year expiration, and we continue to work towards better documenting our internal processes on the wiki to prevent this sort of thing from happening again. Apache, and slashd are running again, and we appear to be back to status-quo in terms of site operation.
A full incident report will be written up and posted to the wiki in the next few days.
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Life is to you a dashing and bold adventure.
(Score: 3, Funny) by davester666 on Friday March 14 2014, @07:43AM
or do something crazy like replace them after only 9 years. I originally was going to say 8, but I know how expensive these things are.
(Score: 3, Insightful) by juggs on Friday March 14 2014, @07:55AM
Well not really if all you are doing is generating your own OpenSSL certs / keys etc. for internal LAN work (which is all application to DB requires). It's only when you start to involve external CA's that things get expensive.
With all the revelations about NSA et al. capabilities for MITM / Man to the side etc. infiltration, I'm starting to think I trust self-signed certs more than CA verified ones regardless of whether that gives me a lovely warm green "enhanced verified" in the browser or throws up scary warnings. Have you bothered to check the root certs that your browser accepts as kosher? It's generally a very long list.
(Score: 5, Informative) by NCommander on Friday March 14 2014, @08:25AM
We're using self-signed certificates at the moment with strict checking to prevent MITM since we're traversing a non-secure network for database lookups. At the moment, since this is a VPS, nothing prevents the NSA from warrenting Linode and getting direct access to the nodes. For the moment, we're staying with Linode for the foreseeable future, *but* I'd like to get us self-owned hardware which gives us control and/or knowledge of such things.
One thing at a time though ...
Still always moving
(Score: 4, Insightful) by juggs on Friday March 14 2014, @09:36AM
Self owned hardware in a hired rack in a DC is no better than VPS in reality.
Little black box is little black box, they get inserted all over the place in DCs and co-mingling areas - expect no privacy posting to any kind of "online" internet resource.
Maybe we should approach this as "spoon boy" ala The Matrix - it is not privacy that breaks, there is no privacy. there never was, it is only our belief that there once was privacy that makes it existant to the extent we try to protect it. Realise privacy does not and never has existed. Only when we realise no such thing existed can we move forward to create it. We have the intellectual wherewithal to create it but first we must accept we failed to create it last time around.
(Score: 2) by FatPhil on Friday March 14 2014, @10:21AM
With self-signed certs, you only need to trust one party. With CA's you need to trust 2. Or sometimes more.
The only CA authority I trust is Honest Achmed. How can you not trust someone whose uncle makes such a great shish kebab!
Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
(Score: 2) by NCommander on Friday March 14 2014, @12:14PM
Curious on your thoughts of CACert then ...
Still always moving
(Score: 3, Insightful) by FatPhil on Friday March 14 2014, @01:17PM
Certs issued by them? I don't trust them, basically. No reason to.
Well, there are reasons to - namely the web of trust that they maintain. The fact that they use web-of-trust rather than tree-of-trust makes me instantly more in favour of them than the current mainstream CA fiasco. But I probably don't trust anyone in that web of trust.
Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
(Score: 0) by Anonymous Coward on Friday March 14 2014, @05:52PM
I trust my self signed certs more than external CAs