Stories
Slash Boxes
Comments

SoylentNews is people

Meta

Log In

Log In

Create Account  |  Retrieve Password


posted by janrinok on Sunday November 05 2023, @04:09PM   Printer-friendly

Our certificates expire in a few hours time and there is nobody available to update them. You may see warnings in your browser. Please accept our apology.

UPDATE: We think that we have succesfully renewed the certificates - but if anyone encounters anything unusual please let us know either here or on IRC.

This discussion was created by janrinok (52) for logged-in users only, but now has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
(1)
  • (Score: 4, Insightful) by Mojibake Tengu on Sunday November 05 2023, @04:20PM (1 child)

    by Mojibake Tengu (8598) on Sunday November 05 2023, @04:20PM (#1331548) Journal

    Why not replace web allowances certificates a month before expiration? They are free anyway...

    --
    Respect Authorities. Know your social status. Woke responsibly.
    • (Score: 2) by drussell on Monday November 06 2023, @04:53PM

      by drussell (2678) on Monday November 06 2023, @04:53PM (#1331678) Journal

      Why not replace web allowances certificates a month before expiration? They are free anyway...

      The answer to that is very simple...

      None of the (few) people with the appropriate access to the system(s) involved is paying attention.

  • (Score: 3, Insightful) by PiMuNu on Sunday November 05 2023, @04:42PM (7 children)

    by PiMuNu (3823) on Sunday November 05 2023, @04:42PM (#1331551)

    Thanks

    • (Score: 5, Informative) by janrinok on Sunday November 05 2023, @04:44PM (6 children)

      by janrinok (52) Subscriber Badge on Sunday November 05 2023, @04:44PM (#1331552) Journal

      We think that we have succesfully updated them - but if anyone encounters anything unusual please let us know either here or on IRC.

      --
      I am not interested in knowing who people are or where they live. My interest starts and stops at our servers.
      • (Score: 2) by PiMuNu on Sunday November 05 2023, @04:59PM

        by PiMuNu (3823) on Sunday November 05 2023, @04:59PM (#1331557)

        Well done!

      • (Score: 2) by number11 on Sunday November 05 2023, @10:05PM

        by number11 (1170) Subscriber Badge on Sunday November 05 2023, @10:05PM (#1331587)

        Looks good.

      • (Score: 2) by canopic jug on Monday November 06 2023, @08:05AM

        by canopic jug (3949) Subscriber Badge on Monday November 06 2023, @08:05AM (#1331621) Journal

        Thanks for all you and the rest of the team do in keeping this site going.

        --
        Money is not free speech. Elections should not be auctions.
      • (Score: 2) by drussell on Monday November 06 2023, @04:55PM (2 children)

        by drussell (2678) on Monday November 06 2023, @04:55PM (#1331679) Journal

        The mail/irc server still hasn't had the certificate copied over...

        This has been pointed out several times in IRC, but nobody with the appropriate access is listening.

        • (Score: 3, Informative) by janrinok on Monday November 06 2023, @06:22PM (1 child)

          by janrinok (52) Subscriber Badge on Monday November 06 2023, @06:22PM (#1331698) Journal

          The team are working on this as I write. There are more than half a dozen places where something different has to be done. We are sorting them out in turn and documenting what we are doing.

          --
          I am not interested in knowing who people are or where they live. My interest starts and stops at our servers.
          • (Score: 2) by drussell on Monday November 06 2023, @06:58PM

            by drussell (2678) on Monday November 06 2023, @06:58PM (#1331702) Journal

            Yeah, thanks guys...

            Postfix is now responding with the new certificate, just need to fix the IRC server, the logs.sylnt.us webserver, etc. on the same machine.

            I'm sure fixes are all in progress... 🙂

  • (Score: 1) by pTamok on Sunday November 05 2023, @04:53PM (13 children)

    by pTamok (3042) on Sunday November 05 2023, @04:53PM (#1331555)

    Just a thought - I know it's a bit steam age, but could a cron job kick off a script to check expiry dates of certificates and send a warning with a little more advance notice.

    It's one of these things that is important, but isn't urgent, so gets relegated to the 'drain the swamp' list rather than the 'fight alligators' list, until it suddenly becomes an alligator.

    Well done on taking action before the deadline.

    • (Score: 2) by loonycyborg on Sunday November 05 2023, @05:48PM (1 child)

      by loonycyborg (6905) on Sunday November 05 2023, @05:48PM (#1331562)

      Just use let's encrypt if you want automation.

      • (Score: 2) by Whoever on Monday November 06 2023, @11:47PM

        by Whoever (4524) on Monday November 06 2023, @11:47PM (#1331769) Journal

        Just use let's encrypt if you want automation.

        The site does. But it doesn't use the HTTP-01 challenge method to update some sites, making automatic renewal problematic.

    • (Score: 5, Informative) by janrinok on Sunday November 05 2023, @06:45PM (3 children)

      by janrinok (52) Subscriber Badge on Sunday November 05 2023, @06:45PM (#1331567) Journal

      We do - but the configuration of other services requires the keys to be copied around the system in some way - I don't have the details.

      As I understand it, the complete reconfig of the system cannot take place until we have control of all of the servers and at the moment they are still under the control of NCommander. Hence the reason why we have to create a new company to buy the current SN and then we can reconfigure it the way that we think it ought to be configured. k0lie has already reconfigured his staging servers to update automatically.

      It is little things such as this which eventually become bigger things which in turn cause the sys-admin guys to be pulling out their hair. Everyone knows how to solve the problem but for that we have to create a new business and buy everything from the current site. However, this is not my part of ship so I speak with no authority here at all!

      And in a few days time we can 'celebrate' the first year anniversary of the live stream showing the 'software update' that created this whole mess.

      --
      I am not interested in knowing who people are or where they live. My interest starts and stops at our servers.
      • (Score: 3, Insightful) by pTamok on Sunday November 05 2023, @07:15PM (2 children)

        by pTamok (3042) on Sunday November 05 2023, @07:15PM (#1331569)

        This is why you need a script, or some other reliable monitoring mechanism, to check things have been copied round as they should.

        If all the systems are accessible to the extent that curl can grab the necessary information [nickjanetakis.com], then it can be done without ownership.

        You end up with a script that documents where all the necessary certificates are, and checks for proximity of the expiry date on a regular basis, and can write to a log and email a mailing list with status and warnings (if and when necessary).

        If you have private systems behind firewalls, it gets a little more complicated, but only to the extent that you have one or several trusted hosts running the script. That could be the issue if the owner prevents this.

        • (Score: 2) by drussell on Monday November 06 2023, @04:58PM (1 child)

          by drussell (2678) on Monday November 06 2023, @04:58PM (#1331680) Journal

          This is why you need a script, or some other reliable monitoring mechanism, to check things have been copied round as they should.

          There IS a monitoring bot that reports the certificate status to #staff on IRC, but none of the (few) people with access to the system(s) has been paying attention.

          THAT'S the problem, not the lack of monitoring scripts.

          • (Score: 2) by kolie on Tuesday November 07 2023, @10:47PM

            by kolie (2622) on Tuesday November 07 2023, @10:47PM (#1331982) Journal

            AFAIK all former staff have access restored to prior when it was removed for winding down the site. No one's made an attempt to clarify their access is missing with me, and I did my best when restoring it to put back what I knew was changed. No one's come forward for any access requests to me that I am currently aware of and hasn't been granted it - and I'm not the only gatekeeper as I specifically restored enough people to make sure they could expand that access back as well.

    • (Score: 4, Insightful) by janrinok on Monday November 06 2023, @07:34AM (6 children)

      by janrinok (52) Subscriber Badge on Monday November 06 2023, @07:34AM (#1331618) Journal

      Is I was lying in bed pondering the world and its problems I recalled that you had mentioned cron jobs.

      We had various scripts until last November when, during the software 'update' the configuration of the servers, and much of the code itself, changed without anyone doing any extensive testing. Many of the scripts that we had no longer worked, the documentation did not reflect what subsequently existed. Despite our best efforts to recover things, the staff were eventually locked out of the system. We still do not have the control or access that we once had.

      Many of our community have forgotten the reason that we are in this predicament. The staff are still trying to establish a site where the control is returned to those whose job it is to manage such things. Most of us still cannot access parts of the system that we used to use daily.

      --
      I am not interested in knowing who people are or where they live. My interest starts and stops at our servers.
      • (Score: 1) by pTamok on Tuesday November 07 2023, @07:35AM

        by pTamok (3042) on Tuesday November 07 2023, @07:35AM (#1331814)

        Thank you for the update.

        I hope the ongoing recovery work is successful. The process sounds somewhat frustrating to me.

      • (Score: 2) by kolie on Tuesday November 07 2023, @10:45PM (4 children)

        by kolie (2622) on Tuesday November 07 2023, @10:45PM (#1331980) Journal

        I'm not sure who doesn't have access to what.

        It was my understand all access was restored. No ones piped up about missing access.

        • (Score: 3, Interesting) by janrinok on Wednesday November 08 2023, @06:48AM (3 children)

          by janrinok (52) Subscriber Badge on Wednesday November 08 2023, @06:48AM (#1332062) Journal

          Can we install Gentoo on all of the servers which we had started doing about 16 months ago?

          Can we start changing the configuration of the linode servers to something that is closer to what we want?

          Why is automating the cert updates causing us such a problem? Can we reconfigure the site better?

          When some of us (myself and FNord666) directly query the database we are getting reports of cluster errors. Can somebody fix this problem please? I can log in to the database (is it the correct database?) but I cannot use it. We don't want access via another software interface, we want the ability to directly query the database.

          --
          I am not interested in knowing who people are or where they live. My interest starts and stops at our servers.
          • (Score: 2) by kolie on Wednesday November 08 2023, @06:53PM (1 child)

            by kolie (2622) on Wednesday November 08 2023, @06:53PM (#1332146) Journal

            Can we install Gentoo on all of the servers which we had started doing about 16 months ago?
              sure
            Can we start changing the configuration of the linode servers to something that is closer to what we want?
              sure
            Why is automating the cert updates causing us such a problem? Can we reconfigure the site better?
              sure
            When some of us (myself and FNord666) directly query the database we are getting reports of cluster errors.
              your probably using the wrong one because there is no db cluster.

            • (Score: 2) by janrinok on Wednesday November 08 2023, @07:14PM

              by janrinok (52) Subscriber Badge on Wednesday November 08 2023, @07:14PM (#1332152) Journal
              Could you enlighten us to which one we should be using then please?
              --
              I am not interested in knowing who people are or where they live. My interest starts and stops at our servers.
          • (Score: 2) by kolie on Wednesday November 08 2023, @06:54PM

            by kolie (2622) on Wednesday November 08 2023, @06:54PM (#1332147) Journal

            re certs

            It's not a problem. It's just not setup to do it.

            The staging configuration does all this. It replicates the entire stack and functionality and has way better maintainability and documentation.

  • (Score: 5, Funny) by Rosco P. Coltrane on Sunday November 05 2023, @08:47PM

    by Rosco P. Coltrane (4757) on Sunday November 05 2023, @08:47PM (#1331577)

    Our certificates expire in a few hours time and there is nobody available to update them

    You'll never read this on any other website. I love it!

  • (Score: 0) by Anonymous Coward on Monday November 06 2023, @01:32AM (1 child)

    by Anonymous Coward on Monday November 06 2023, @01:32AM (#1331602)
    Isn't there already automation in place to renew the certs? The obligatory certbot comes to mind. Who would want to use 90-day Let's Encrypt certs without automatic renewal?!
    • (Score: 2) by kolie on Monday November 06 2023, @04:36AM

      by kolie (2622) on Monday November 06 2023, @04:36AM (#1331615) Journal

      Idk and honestly for the main site it probably could work now with cert auto renewing. Just not setup currently.

  • (Score: 2) by ledow on Monday November 06 2023, @08:50AM (1 child)

    by ledow (5567) on Monday November 06 2023, @08:50AM (#1331625) Homepage

    Please automate your system. No site should fall over just because someone forgot to look at the calendar, and nowadays it renders the site almost inaccessible given the various browser warnings and automatic HTTPS connection upgrades.

    If it's not automatic to renew your certs and insert them everywhere they need to be (certbot, and an scheduled rsync) then you are really just creating panic and stress and work for yourself.

(1)