Stories
Slash Boxes
Comments

SoylentNews is people

Meta

Site Back Up After Unexpected Downtime

posted by NCommander on Saturday November 12 2022, @08:43AM   Printer-friendly
from the its-in-flames dept.

So, quick update here. The site was down for most of the night because the database cluster shot itself in the head. I had restarted a machine to install updates, and this caused the backend cluster to entire to entirely loose its mind. Unfortunately, I didn't have a manual dump of the database made, just a VM snapshot, since, well, I wasn't tinkering with it directly. I've mostly been trying to patch things to the point that I can sleep, and leaving things down like IRC and email which need to be seriously overhauled before they can go back up.

As far as damages go, it looks like we lost 10 or so days of messages, which uh, sucks for multiple reasons. We're currently on ##soylentnews on Libera.Chat while I pull bits of the site out of the flames, but I'm at the point that if I don't sleep, I will make things worse. Corruption in the production database is very much not what I wanted, and we're very much in limp mode for the moment. I'm going to let staff handle IRC and comments while I sleep, and then I'll post another update when I'm awake.

See you in a few hours

~ NCommander

 
This discussion was created by NCommander (2) for logged-in users only, but now has been archived. No new comments can be posted.
Site Back Up After Unexpected Downtime | Log In/Create an Account | Top | 57 comments | Search Discussion
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

  • (Score: 1, Insightful) by Anonymous Coward on Saturday November 12 2022, @09:32AM (7 children)

    by Anonymous Coward on Saturday November 12 2022, @09:32AM (#1279347)

    I'm going with coincidence. I don't think there's any sort of conspiracy. Remember yesterday's meta? This site is held together with chewing gum and string.

    Starting Score:    0  points
    Moderation   +1  
       Insightful=1, Total=1
    Extra 'Insightful' Modifier   0  
    Total Score:   1  

  • (Score: 5, Insightful) by Anonymous Coward on Saturday November 12 2022, @10:07AM (5 children)

    by Anonymous Coward on Saturday November 12 2022, @10:07AM (#1279348)
    This site is a testimony on why not to use MySQL or derivatives.

    • (Score: 2) by RS3 on Saturday November 12 2022, @05:59PM (4 children)

      by RS3 (6367) on Saturday November 12 2022, @05:59PM (#1279385)

      Hmmm. Blanket? As in, never ever anywhere?

      I'm the only IT / admin for a smallish hosting / server operation. I took over for departed staff about 15 years ago. A bunch of the sites are WordPress, and I've never had any problems. I've updated, upgraded, migrated, no problems, ever, with mysql, or php or WordPress or plugins.

      My only worry about mysql is Oracle, so I may switch to mariadb soon, but for now everything has been great.

      What problems have you seen or had with mysql or derivatives?

      • (Score: 0) by Anonymous Coward on Sunday November 13 2022, @06:41AM (3 children)

        by Anonymous Coward on Sunday November 13 2022, @06:41AM (#1279454)

        What problems have you seen or had with mysql or derivatives?

        Is your brain running on MySQL too? Has it lost info on the recent/regular noticeable data loss on SoylentNews?

        If you store stuff in a real DB it won't get lost like that so easily.

        There's no good reason to use MySQL other than "legacy" aka some fools made the wrong choice years ago and you're stuck with it.

        • (Score: 3, Funny) by janrinok on Sunday November 13 2022, @09:25AM (2 children)

          by janrinok (52) Subscriber Badge on Sunday November 13 2022, @09:25AM (#1279469) Journal

          What a load of rubbish. Have you bothered to find out why we lost data?

          • (Score: 0) by Anonymous Coward on Sunday November 13 2022, @10:44AM

            by Anonymous Coward on Sunday November 13 2022, @10:44AM (#1279475)
            So why did you lose data?

          • (Score: 0) by Anonymous Coward on Sunday November 13 2022, @10:48AM

            by Anonymous Coward on Sunday November 13 2022, @10:48AM (#1279476)

            The site was down for most of the night because the database cluster shot itself in the head. I had restarted a machine to install updates, and this caused the backend cluster to entire to entirely loose its mind. Unfortunately, I didn't have a manual dump of the database made, just a VM snapshot, since, well, I wasn't tinkering with it directly.

            So what DB software is that DB cluster running?

  • (Score: 4, Funny) by janrinok on Saturday November 12 2022, @12:09PM

    by janrinok (52) Subscriber Badge on Saturday November 12 2022, @12:09PM (#1279364) Journal
    Yes, and the chewing gum has no flavour and the string is badly frayed. The duct tape is still holding though...