posted by
martyb
on Thursday March 14 2019, @02:30PM
from the certs-are-not-just-a-breath-mint dept.
from the certs-are-not-just-a-breath-mint dept.
With many thanks to The Mighty Buzzard riding shotgun and helping me through some misunderstandings, I updated the certificates (certs) for all of SoylentNews' domains. Our certs are now good through: Wednesday, June 12, 2019.
Everything seemed to go as expected. If you experience any issues, please mention them here, or pop onto our IRC channel using your favorite client or the web interface and speak up in the #dev or #Soylent channel.
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
(1)
(1)
(Score: 2, Interesting) by Anonymous Coward on Thursday March 14 2019, @02:46PM (3 children)
After several days they still couldn't get the SSL certs right for a couple of their servers.
Every couple of days I pasted the error messages into their feedback box. With a message about firing their IT guys.
(Score: 3, Funny) by SomeGuy on Thursday March 14 2019, @05:14PM
You make the assumption that they have not already fired their IT. After all, those pesky computer guys just suck up money and don't do anything the big boss can actually see. When something breaks, just contract out to a new outsourcing company in India. The management brochures say that is how it should be done, herp derp. It doesn't matter if it take months to get the paperwork ready and everyone up to speed just to press a button.
(Score: 2, Funny) by driverless on Friday March 15 2019, @03:43AM (1 child)
Just out of interest, for martyb and Buzzard, how long did it take to do this and how much effort was it? I'm kinda curious. I'm sure you've discovered my deep and abiding interest in pain. At present I'm writing the definitive work on the subject. So, let's just start with what we have. What did this do to you? Tell me. And remember, this is for posterity, so be honest — how do you feel?
(Score: 2) by The Mighty Buzzard on Saturday March 16 2019, @12:35AM
Including having to start over once, miscommunication, double checking, general nervousness, and other first time doing it caution? A little under two hours. I very much approve of taking extra time to make damned sure you're doing things right in anyone doing something new as root though. Doing it the third or fourth time takes between two and ten minutes, depending on how fast DNS propagates out to linode's nameservers.
My rights don't end where your fear begins.
(Score: 5, Insightful) by Snow on Thursday March 14 2019, @02:52PM (40 children)
Thank you for your hard work on keeping this site great.
(Score: 4, Interesting) by martyb on Thursday March 14 2019, @03:01PM (38 children)
Looking back on things, I could have borked up everything so badly that all of our servers became inaccessible.
TheMightyBuzzard had written up the steps to follow, but I have a "gift" for finding surprises. I made a couple mistakes along the way, but apparently that is par for the course. TMB had me back on course in short order. I've logged all my commands and output, so the next time should [hopefully] go more smoothly.
And thanks for the support. It sometimes feels like a thankless task to keep the stories coming, but a little bit of appreciation goes a long ways!
Wit is intellect, dancing.
(Score: 4, Interesting) by Snow on Thursday March 14 2019, @03:09PM (24 children)
Don't worry about making mistakes... as long as you fix them :)
At my work, we have to reissue and assign certificates every 2 years. EVERY. SINGLE. TIME. it gets fucked up in some way. Usually by forgetting to assign the certificate to the service or not properly setting the permissions on the certificate. We find and fix the issue, then forget about it until we relearn everything again in two years.
We have an IT team of 30+ people.
(Score: 2) by martyb on Thursday March 14 2019, @03:17PM (3 children)
That's the main reason why I posted this story: if anything got borked, best to find out as soon as possible so we can fix it.
Wit is intellect, dancing.
(Score: 3, Interesting) by Runaway1956 on Thursday March 14 2019, @03:21PM (2 children)
I logged in a few minutes ago from a LiveUSB. The Firefox version on the USB balked at the cert, but I just told it to accept permanently. Been fighting network problems, didn't need an additional problem. I probably should have written it down, with a time stamp though.
(Score: 3, Informative) by NewNic on Thursday March 14 2019, @05:35PM
Cert looks good under Firefox for me (Firefox 60.1.0esr under Linux).
lib·er·tar·i·an·ism ˌlibərˈterēənizəm/ noun: Magical thinking that useful idiots mistake for serious political theory
(Score: 2) by The Mighty Buzzard on Thursday March 14 2019, @05:52PM
How old is the image? FF has had LetsEncrypt's CA trusted for a while now.
My rights don't end where your fear begins.
(Score: 4, Interesting) by isostatic on Thursday March 14 2019, @04:05PM (12 children)
1) Your monitoring infrastructure should be testing all your certs anyway, so any that are due for renewal get flagged up as warnings, and that are invalidly deployed or expired get flagged up as criticals
2) You should be automating this -- at the very least have a process that generates the correct number and contents of CSRs in the right format to send to your certificate authority, even if it doesn't automatically renew them
3) you should be planning on reducing your certs from 2 years, not just for security reasons (and that's good enough anyway), but because CAB are likely going to be pushing it down to 1 year relatively soon anyway.
(Score: 5, Informative) by martyb on Thursday March 14 2019, @04:44PM (11 children)
Last things, first... our new certs (from Let's Encrypt) are set to expire on: Wednesday, June 12, 2019. So we are already at renewing every 3 months They kindly send out an email giving us a few weeks' advance notice of expiring certs.
Next, there is some automation in place, but with checks along the way for manual confirmation before advancing to the next step.
As for the other points, waaay back when, someone stood up an instance of Icinga [icinga.com]. They left, things changed, and it was not maintained. Eventually it was shut down. (Spoken only as an observer; I had nothing to do with the rollout or shutdown.)
My focus/skill lies more on the upper layers of the software stack. Not so much with the setting up and running of the underlying services on which SoylentNews depends (Bind, Apache, MySql, Nginx, Perl, email, IRC, etc.) Further, we have a mix of OS platforms. Last I checked, we have one Centos, one OpenVZ (our backup server -- IIRC, it's an entirely different provider), a couple on gentoo, and the rest on Ubuntu LTS.
Have you any experience with a mixed environment and can make a recommendation (preferably one that is light weight in resource needs)?
Even better, would you like to volunteer? =)
Wit is intellect, dancing.
(Score: 2) by NewNic on Thursday March 14 2019, @05:38PM
My recommendation is to get rid of the mixed environment.
Pick a distro that is supported long term and use only that. Multiple distros provide needless complications.
lib·er·tar·i·an·ism ˌlibərˈterēənizəm/ noun: Magical thinking that useful idiots mistake for serious political theory
(Score: 3, Informative) by NewNic on Thursday March 14 2019, @05:40PM (9 children)
And how well is that working out for you?
If you fully automate it, you will have to fully fix any issues in the process.
lib·er·tar·i·an·ism ˌlibərˈterēənizəm/ noun: Magical thinking that useful idiots mistake for serious political theory
(Score: 4, Informative) by The Mighty Buzzard on Thursday March 14 2019, @05:56PM (4 children)
I decided against automating changes to DNS for dns-01 challenges from LetsEncrypt. I prefer to make the whole domain utterly unreachable manually.
My rights don't end where your fear begins.
(Score: 2) by NewNic on Thursday March 14 2019, @08:27PM (3 children)
Why not use http challenges instead?
lib·er·tar·i·an·ism ˌlibərˈterēənizəm/ noun: Magical thinking that useful idiots mistake for serious political theory
(Score: 3, Informative) by The Mighty Buzzard on Thursday March 14 2019, @09:33PM (2 children)
You can't on wildcard certs.
My rights don't end where your fear begins.
(Score: 2) by NewNic on Thursday March 14 2019, @09:42PM (1 child)
So don't use wildcards. Let's encrypt makes it very easy to use certs with multiple names in them.
lib·er·tar·i·an·ism ˌlibərˈterēənizəm/ noun: Magical thinking that useful idiots mistake for serious political theory
(Score: 2) by The Mighty Buzzard on Friday March 15 2019, @12:28AM
Sure, if you're hosting everything on one box with one webserver to do the authing. Using them on our network is complicated as nine kinds of fuck though.
My rights don't end where your fear begins.
(Score: 3, Informative) by isostatic on Thursday March 14 2019, @08:32PM (3 children)
Looks like they moved to wildcard certs for
DNS Name: *.soylentnews.org
DNS Name: *.sylnt.us
DNS Name: soylentnews.org
DNS Name: sylnt.us
last July.
However before then there were 17 certs with lets encrypt
chat.soylentnews.org
chat.sylnt.us
dev.soylentnews.org
irc1.sylnt.us
irc2.sylnt.us
irc-logs.soylentnews.org
irc.soylentnews.org
irc-stats.soylentnews.org
irc.sylnt.us
lists.soylentnews.org
logs.sylnt.us
mail.soylentnews.org
postfixadmin.soylentnews.org
stats.sylnt.us
vm.soylentnews.org
webmail.soylentnews.org
wiki.soylentnews.org
There was also a cert for www.soylentnews.org with Gandi, but that expired last june. Go back to 2015 and there was also "chillax.soylentnews.org", which had a Startcom cert (I think they were free -- they were/are a Chinese CA that got into some wrongdoing a couple of years ago)
All of those appear to host pages on port 80, so I'm interested in the reason to not use /.well-known/acme-challange authentication, with a weekly renewal cronjob running. Avoid spreading a wildcard cert/key so far and wide, and have nothing manual to do.
(Score: 2) by NewNic on Thursday March 14 2019, @08:52PM
Exactly.
It's very easy to have multiple names in a certificate with Let's encrypt.
lib·er·tar·i·an·ism ˌlibərˈterēənizəm/ noun: Magical thinking that useful idiots mistake for serious political theory
(Score: 2) by The Mighty Buzzard on Thursday March 14 2019, @09:34PM (1 child)
You can't use http challenges for wildcard certs, must be dns-01.
My rights don't end where your fear begins.
(Score: 2) by isostatic on Friday March 15 2019, @04:35PM
Which goes back to the question of why use a wildcard cert
(Score: 1) by shrewdsheep on Thursday March 14 2019, @04:24PM (6 children)
Put it into your configuration management system right after getting it to work. It is tempting to forget about it because the next task is waiting. However, tt will safe you time, even if you waste some time on configurations that you might not need later (been there several times).
(Score: 2) by Snow on Thursday March 14 2019, @04:35PM (3 children)
HAHA, that's funny.
Configuration Management System... Good one! I'll enter that as a suggestion in the Problem Management system and bring it up at the next CAB meeting, LOL!
Seriously though, all our documentation is scattered all over the place in a disaster of an implementation of Sharepoint. It's definitely a factor in screwing up certs every time. Sharepoint is so shitty that no one wants to use it, so they just wing it every time.
(Score: 0) by Anonymous Coward on Thursday March 14 2019, @07:37PM (1 child)
Sharepoint? You poor bastards.
(Score: 0) by Anonymous Coward on Friday March 15 2019, @12:39PM
Shitpoint
(Score: 0) by Anonymous Coward on Friday March 15 2019, @02:55AM
DHS?
(Score: 2) by The Mighty Buzzard on Thursday March 14 2019, @06:05PM (1 child)
Our configuration management system is a multi-process system. Different bits run in each of the admins' brains. Except for the bits that were set up by admins who've wandered on to greener pastures. We just wait until something breaks to monkey with those.
My rights don't end where your fear begins.
(Score: 2) by driverless on Friday March 15 2019, @04:53AM
Our configuration management system is Paul. "Hey Paul, what's the latest on X?". Occasionally we have to reinstall beer in Paul, particularly on Friday afternoons, but apart from that it works OK.
(Score: 2) by NotSanguine on Thursday March 14 2019, @04:36PM (11 children)
Yes. Your hard work makes this place go. I know I appreciate it. Thanks to you and all the volunteers!
As an aside, I'm curious why it was complicated. I just updated several Let's Encrypt certs on one of my servers and installed a new Let's Encrypt cert on a new server a few hours ago. Using Certbot [gentoo.org] it took less than 30 seconds to renew three certs (via 'certbot renew') and almost ten minutes (as I had to use 'certbot certonly' and then modify my http server config and restart) for the new server.
I'm not poking you or Buzzard, I'm genuinely curious as to what's more complicated, aside from having to replicate the certs to multiple servers.
In fact, when renewing my certs I chastised myself for not just automating the process.
Perhaps a cron job that runs every 6-8 weeks like this:
#!/bin/sh
#
# Renew certs
/usr/bin/certbot renew
for i in host1 host2 ... hostn
/usr/bin/scp [path to renewed cert] ${i}:[path to cert]
Or something similar. I'm probably missing some complexity in the SN environment, but like I said I'm curious.
No, no, you're not thinking; you're just being logical. --Niels Bohr
(Score: 2) by martyb on Thursday March 14 2019, @05:04PM (2 children)
As this was my very first time ever updating certs, that just might have been a factor. I have a high-level concept of DNS and certs, but actually messing around with actual files and their syntax... let's just say I was very cautious.
A long time ago I came upon some words of wisdom that have served me well:
I knew I did not understand, so I took my time as I went. Here's another:
Until it is absolutely clear what ALL the success -- and failure -- paths are, my experience has been that it is best to keep a human in the loop.
For further details on possible additional automation I will have to defer to the others on staff who have way more experience with this than I do.
Wit is intellect, dancing.
(Score: 2) by NotSanguine on Thursday March 14 2019, @05:22PM
As the wonderful G.B. Shaw said:
I try to be unreasonable. :)
No, no, you're not thinking; you're just being logical. --Niels Bohr
(Score: 0) by Anonymous Coward on Friday March 15 2019, @03:02AM
This demonstrates the attitude of a good sys admin.
https://xkcd.com/705/ [xkcd.com]
(Score: 3, Informative) by The Mighty Buzzard on Thursday March 14 2019, @06:00PM (7 children)
dns-01 challenges are required for wildcard certs. I don't want to automate DNS changes, I prefer to screw those up on my own. Otherwise it would be a "run this script and go back to what you were doing" thing.
My rights don't end where your fear begins.
(Score: 2) by NotSanguine on Thursday March 14 2019, @06:35PM
That's sensible. I'm not a huge fan of wildcard certs, but I can see how they'd be quite useful in the SN environment.
No, no, you're not thinking; you're just being logical. --Niels Bohr
(Score: 2) by NewNic on Thursday March 14 2019, @08:30PM (5 children)
Don't use wildcards.
lib·er·tar·i·an·ism ˌlibərˈterēənizəm/ noun: Magical thinking that useful idiots mistake for serious political theory
(Score: 2) by The Mighty Buzzard on Thursday March 14 2019, @09:41PM (4 children)
We have many different things serving up http pages for all the hostnames we have on many different boxes. And we have hostnames that don't have web content associated with them at all. It was always a much bigger pain in the ass managing the multi-name certs than having to manually update four values in DNS every few months.
My rights don't end where your fear begins.
(Score: 2) by NewNic on Thursday March 14 2019, @09:55PM (3 children)
For a traditional certificate issuance, I can see that. With Let's Encrypt, it is trivial to manage multi-name certs. For those machines without a web server, you can use the Standalone plugin, which starts its own web server.
Oh well, if you want to persist with an error-prone and time wasting process, who am I to argue with you.
lib·er·tar·i·an·ism ˌlibərˈterēənizəm/ noun: Magical thinking that useful idiots mistake for serious political theory
(Score: 2) by The Mighty Buzzard on Friday March 15 2019, @12:38AM (2 children)
You're not understanding how much of a mess our setup is. If you put all the hostnames that just beryllium uses in one cert, you have to make more than one webserver with many different vhosts on them serve the right response for each and every vhost (certbot trying to do this automatically breaks half the vhosts), then you have to make the multiple irc hostnames serve up the proper response, then you have to make the mail server hostnames serve up the right response. And when you want to add or remove a hostname from use on the box, you have to redo the cert from scratch.
Seriously, it's much quicker and easier to use a wildcard cert. I've never had a multihost SN cert take less than an hour worth of work to renew.
My rights don't end where your fear begins.
(Score: 2) by NewNic on Friday March 15 2019, @06:53PM (1 child)
No, you exclude the "/.well-known" location from the Vhosts. This can be achieved with an alias command.
https://community.letsencrypt.org/t/apache-multidomain-webroot/10663 [letsencrypt.org]
lib·er·tar·i·an·ism ˌlibərˈterēənizəm/ noun: Magical thinking that useful idiots mistake for serious political theory
(Score: 2) by The Mighty Buzzard on Saturday March 16 2019, @12:43AM
Or I could do like I'm doing and never have to touch the configs of anything but the one we pull the cert from. And never have to remake the entire enormous cert, hoping I don't miss a hostname but knowing I will, if Deucalion thinks we need a new IRC hostname on one of the existing boxes.
My rights don't end where your fear begins.
(Score: 0) by Anonymous Coward on Thursday March 14 2019, @05:11PM
I hope you also updated the documentation with the caveats you bumped into. :)
All things are simple when you know how to do them.
(Score: 2) by driverless on Friday March 15 2019, @03:49AM
Helping to MthissiteGA?
(Score: 4, Interesting) by Whoever on Thursday March 14 2019, @03:00PM (5 children)
I hope you are now automatically updating the certificates. It's quite easy to do this with Let's Encrypt.
(Score: 5, Interesting) by martyb on Thursday March 14 2019, @03:08PM
Yes, it is possible. No, it's not likely. TheMightyBuzzard expressed misgivings about automagically updating DNS records. I must say I share, them, too. I see a couple places where some automation would come in handy, but I would still prefer to have a human in the loop... Just. In. Case.
Besides, you are talking to the QA guy for the site. I am positively gifted in making things go sidedays which is NOT something you want happening in a running system.
Do be aware that we have a total of 10 systems to keep in sync, as well.
So, I'm not saying never, but it will be a long while before we would go fully automated, and there are reasons for it.
Wit is intellect, dancing.
(Score: 2) by isostatic on Thursday March 14 2019, @03:53PM (3 children)
We should be pushing certificate lengths down to 3 month maximum at a minimum, and probably shorter than that.
(Score: 3, Informative) by NotSanguine on Thursday March 14 2019, @04:37PM (2 children)
Three months is the default for Let's Encrypt certificates.
No, no, you're not thinking; you're just being logical. --Niels Bohr
(Score: 2) by isostatic on Thursday March 14 2019, @08:17PM (1 child)
And they want to go shorter than that, but given they issue something like a million certificates every day for free, they can't currently justify it.
(Score: 2) by NotSanguine on Thursday March 14 2019, @08:58PM
Thanks. I had no idea. I just figured they were like TGV. /sarc
If you want, you can teach me to tie my shoes or shake properly after pissing.
No, no, you're not thinking; you're just being logical. --Niels Bohr