Our certificates expire in a few hours time and there is nobody available to update them. You may see warnings in your browser. Please accept our apology.
UPDATE: We think that we have succesfully renewed the certificates - but if anyone encounters anything unusual please let us know either here or on IRC.
This discussion was created by janrinok (52) for logged-in users only, but now has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
(1)
(1)
(Score: 4, Insightful) by Mojibake Tengu on Sunday November 05 2023, @04:20PM (1 child)
Why not replace web
allowancescertificates a month before expiration? They are free anyway...Rust programming language offends both my Intelligence and my Spirit.
(Score: 2) by drussell on Monday November 06 2023, @04:53PM
The answer to that is very simple...
None of the (few) people with the appropriate access to the system(s) involved is paying attention.
(Score: 3, Insightful) by PiMuNu on Sunday November 05 2023, @04:42PM (7 children)
Thanks
(Score: 5, Informative) by janrinok on Sunday November 05 2023, @04:44PM (6 children)
We think that we have succesfully updated them - but if anyone encounters anything unusual please let us know either here or on IRC.
[nostyle RIP 06 May 2025]
(Score: 2) by PiMuNu on Sunday November 05 2023, @04:59PM
Well done!
(Score: 2) by number11 on Sunday November 05 2023, @10:05PM
Looks good.
(Score: 2) by canopic jug on Monday November 06 2023, @08:05AM
Thanks for all you and the rest of the team do in keeping this site going.
Money is not free speech. Elections should not be auctions.
(Score: 2) by drussell on Monday November 06 2023, @04:55PM (2 children)
The mail/irc server still hasn't had the certificate copied over...
This has been pointed out several times in IRC, but nobody with the appropriate access is listening.
(Score: 3, Informative) by janrinok on Monday November 06 2023, @06:22PM (1 child)
The team are working on this as I write. There are more than half a dozen places where something different has to be done. We are sorting them out in turn and documenting what we are doing.
[nostyle RIP 06 May 2025]
(Score: 2) by drussell on Monday November 06 2023, @06:58PM
Yeah, thanks guys...
Postfix is now responding with the new certificate, just need to fix the IRC server, the logs.sylnt.us webserver, etc. on the same machine.
I'm sure fixes are all in progress... 🙂
(Score: 1) by pTamok on Sunday November 05 2023, @04:53PM (13 children)
Just a thought - I know it's a bit steam age, but could a cron job kick off a script to check expiry dates of certificates and send a warning with a little more advance notice.
It's one of these things that is important, but isn't urgent, so gets relegated to the 'drain the swamp' list rather than the 'fight alligators' list, until it suddenly becomes an alligator.
Well done on taking action before the deadline.
(Score: 2) by loonycyborg on Sunday November 05 2023, @05:48PM (1 child)
Just use let's encrypt if you want automation.
(Score: 2) by Whoever on Monday November 06 2023, @11:47PM
The site does. But it doesn't use the HTTP-01 challenge method to update some sites, making automatic renewal problematic.
(Score: 5, Informative) by janrinok on Sunday November 05 2023, @06:45PM (3 children)
We do - but the configuration of other services requires the keys to be copied around the system in some way - I don't have the details.
As I understand it, the complete reconfig of the system cannot take place until we have control of all of the servers and at the moment they are still under the control of NCommander. Hence the reason why we have to create a new company to buy the current SN and then we can reconfigure it the way that we think it ought to be configured. k0lie has already reconfigured his staging servers to update automatically.
It is little things such as this which eventually become bigger things which in turn cause the sys-admin guys to be pulling out their hair. Everyone knows how to solve the problem but for that we have to create a new business and buy everything from the current site. However, this is not my part of ship so I speak with no authority here at all!
And in a few days time we can 'celebrate' the first year anniversary of the live stream showing the 'software update' that created this whole mess.
[nostyle RIP 06 May 2025]
(Score: 3, Insightful) by pTamok on Sunday November 05 2023, @07:15PM (2 children)
This is why you need a script, or some other reliable monitoring mechanism, to check things have been copied round as they should.
If all the systems are accessible to the extent that curl can grab the necessary information [nickjanetakis.com], then it can be done without ownership.
You end up with a script that documents where all the necessary certificates are, and checks for proximity of the expiry date on a regular basis, and can write to a log and email a mailing list with status and warnings (if and when necessary).
If you have private systems behind firewalls, it gets a little more complicated, but only to the extent that you have one or several trusted hosts running the script. That could be the issue if the owner prevents this.
(Score: 2) by drussell on Monday November 06 2023, @04:58PM (1 child)
There IS a monitoring bot that reports the certificate status to #staff on IRC, but none of the (few) people with access to the system(s) has been paying attention.
THAT'S the problem, not the lack of monitoring scripts.
(Score: 2) by kolie on Tuesday November 07 2023, @10:47PM
AFAIK all former staff have access restored to prior when it was removed for winding down the site. No one's made an attempt to clarify their access is missing with me, and I did my best when restoring it to put back what I knew was changed. No one's come forward for any access requests to me that I am currently aware of and hasn't been granted it - and I'm not the only gatekeeper as I specifically restored enough people to make sure they could expand that access back as well.
(Score: 4, Insightful) by janrinok on Monday November 06 2023, @07:34AM (6 children)
Is I was lying in bed pondering the world and its problems I recalled that you had mentioned cron jobs.
We had various scripts until last November when, during the software 'update' the configuration of the servers, and much of the code itself, changed without anyone doing any extensive testing. Many of the scripts that we had no longer worked, the documentation did not reflect what subsequently existed. Despite our best efforts to recover things, the staff were eventually locked out of the system. We still do not have the control or access that we once had.
Many of our community have forgotten the reason that we are in this predicament. The staff are still trying to establish a site where the control is returned to those whose job it is to manage such things. Most of us still cannot access parts of the system that we used to use daily.
[nostyle RIP 06 May 2025]
(Score: 1) by pTamok on Tuesday November 07 2023, @07:35AM
Thank you for the update.
I hope the ongoing recovery work is successful. The process sounds somewhat frustrating to me.
(Score: 2) by kolie on Tuesday November 07 2023, @10:45PM (4 children)
I'm not sure who doesn't have access to what.
It was my understand all access was restored. No ones piped up about missing access.
(Score: 3, Interesting) by janrinok on Wednesday November 08 2023, @06:48AM (3 children)
Can we install Gentoo on all of the servers which we had started doing about 16 months ago?
Can we start changing the configuration of the linode servers to something that is closer to what we want?
Why is automating the cert updates causing us such a problem? Can we reconfigure the site better?
When some of us (myself and FNord666) directly query the database we are getting reports of cluster errors. Can somebody fix this problem please? I can log in to the database (is it the correct database?) but I cannot use it. We don't want access via another software interface, we want the ability to directly query the database.
[nostyle RIP 06 May 2025]
(Score: 2) by kolie on Wednesday November 08 2023, @06:53PM (1 child)
Can we install Gentoo on all of the servers which we had started doing about 16 months ago?
sure
Can we start changing the configuration of the linode servers to something that is closer to what we want?
sure
Why is automating the cert updates causing us such a problem? Can we reconfigure the site better?
sure
When some of us (myself and FNord666) directly query the database we are getting reports of cluster errors.
your probably using the wrong one because there is no db cluster.
(Score: 2) by janrinok on Wednesday November 08 2023, @07:14PM
[nostyle RIP 06 May 2025]
(Score: 2) by kolie on Wednesday November 08 2023, @06:54PM
re certs
It's not a problem. It's just not setup to do it.
The staging configuration does all this. It replicates the entire stack and functionality and has way better maintainability and documentation.
(Score: 5, Funny) by Rosco P. Coltrane on Sunday November 05 2023, @08:47PM
You'll never read this on any other website. I love it!
(Score: 0) by Anonymous Coward on Monday November 06 2023, @01:32AM (1 child)
(Score: 2) by kolie on Monday November 06 2023, @04:36AM
Idk and honestly for the main site it probably could work now with cert auto renewing. Just not setup currently.
(Score: 2) by ledow on Monday November 06 2023, @08:50AM (1 child)
Please automate your system. No site should fall over just because someone forgot to look at the calendar, and nowadays it renders the site almost inaccessible given the various browser warnings and automatic HTTPS connection upgrades.
If it's not automatic to renew your certs and insert them everywhere they need to be (certbot, and an scheduled rsync) then you are really just creating panic and stress and work for yourself.
(Score: 2) by janrinok on Monday November 06 2023, @10:20AM
[nostyle RIP 06 May 2025]