Stories
Slash Boxes
Comments

SoylentNews is people

Meta
posted by martyb on Friday May 21 2021, @12:25AM   Printer-friendly

As many of you noticed, we had a site crash today. From around 1300 until 2200 UTC (2021-05-20).

A HUGE thank you goes to mechanicjay who spent the whole time trying to get our ndb (cluster) working again. It's an uncommon configuration, which made recovery especially challenging... there's just not a lot of documentation about it on the web.

I reached out and got hold of The Mighty Buzzard on the phone. Then put him in touch with mechanicjay who got us back up and running using backups.

Unfortunately, we had to go way back until April 14 to get a working backup. (I don't know all the details, but it appears something went sideways on neon).

We're all wiped out right now. When we have rested and had a chance to discuss things, we'll post an update.

In the meantime, please join me in thanking mechanicjay and TMB for all they did to get us up and running again!

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 5, Informative) by bryan on Friday May 21 2021, @05:09PM (4 children)

    by bryan (29) <bryan@pipedot.org> on Friday May 21 2021, @05:09PM (#1137589) Homepage Journal

    Nice summary :) I was wondering what happened yesterday when I tried to click on an article and just got errors. Loosing a month of data sucks, but I see that not as a fault in Rehash but as the infrastructure being overly complex for the current need. Reverting MySQL to a simple and more common configuration will help greatly.

    As for my own site: still sleeping! Occasionally, some spam messages do get posted (common in all internet forums) - but the moderation system hides those once marked (unless you choose to see them in your preferences.)

    Starting Score:    1  point
    Moderation   +3  
       Interesting=1, Informative=2, Total=3
    Extra 'Informative' Modifier   0  
    Karma-Bonus Modifier   +1  

    Total Score:   5  
  • (Score: 2) by janrinok on Friday May 21 2021, @05:33PM

    by janrinok (52) Subscriber Badge on Friday May 21 2021, @05:33PM (#1137594) Journal

    I was wondering if you were still around, Bryan! Good to 'see' you again.

    Yeah, it is a system config problem and mechanicjay is looking at the best way to remove part of our setup completely. It was intended to give us redundancy and resilience but has proven to be the cause of most of our system downtime.

    Rehash is working fine but it is getting increasingly difficult to find people to maintain it. We are looking at various options but the priority is the get the site stable again so we can approach any long-term changes in a calm and methodical manner.

  • (Score: 2) by Reziac on Saturday May 22 2021, @02:33AM (2 children)

    by Reziac (2489) on Saturday May 22 2021, @02:33AM (#1137667) Homepage

    Hey there! sleeping is okay; I'd be very disappointed if it were dead and gone. It seems to work well enough, and the day may come when it's needed awake again.

    --
    And there is no Alkibiades to come back and save us from ourselves.
    • (Score: 3, Funny) by kazzie on Sunday May 23 2021, @08:02AM (1 child)

      by kazzie (5309) Subscriber Badge on Sunday May 23 2021, @08:02AM (#1137914)

      Now I'm getting crossover between Pipedot and the legend of King Arthur...

      • (Score: 3, Funny) by Reziac on Sunday May 23 2021, @02:32PM

        by Reziac (2489) on Sunday May 23 2021, @02:32PM (#1137957) Homepage

        Zombie fora are everywhere. :D

        --
        And there is no Alkibiades to come back and save us from ourselves.