Stories
Slash Boxes
Comments

SoylentNews is people

SoylentNews is powered by your submissions, so send in your scoop. Only 17 submissions in the queue.
Meta

Site Back Up After Unexpected Downtime

posted by NCommander on Saturday November 12 2022, @08:43AM   Printer-friendly
from the its-in-flames dept.

So, quick update here. The site was down for most of the night because the database cluster shot itself in the head. I had restarted a machine to install updates, and this caused the backend cluster to entire to entirely loose its mind. Unfortunately, I didn't have a manual dump of the database made, just a VM snapshot, since, well, I wasn't tinkering with it directly. I've mostly been trying to patch things to the point that I can sleep, and leaving things down like IRC and email which need to be seriously overhauled before they can go back up.

As far as damages go, it looks like we lost 10 or so days of messages, which uh, sucks for multiple reasons. We're currently on ##soylentnews on Libera.Chat while I pull bits of the site out of the flames, but I'm at the point that if I don't sleep, I will make things worse. Corruption in the production database is very much not what I wanted, and we're very much in limp mode for the moment. I'm going to let staff handle IRC and comments while I sleep, and then I'll post another update when I'm awake.

See you in a few hours

~ NCommander

 
This discussion was created by NCommander (2) for logged-in users only, but now has been archived. No new comments can be posted.
Site Back Up After Unexpected Downtime | Log In/Create an Account | Top | 57 comments | Search Discussion
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

  • (Score: 5, Informative) by janrinok on Saturday November 12 2022, @10:17AM (9 children)

    by janrinok (52) Subscriber Badge on Saturday November 12 2022, @10:17AM (#1279349) Journal

    We have also lost 493 'user' accounts, over 50 current submissions, and all of the work that had been prepared for the front page to cover the weekend, etc

    Yep, we might have also lost your journal.

    Of the accounts that were lost, only a handful of genuine accounts were among them (no more than 5). There were also 5 sock puppet accounts but fortunately somebody's spam bot was filling up the vast majority of the remainder (475+). If you think that your account was among those lost please email your nickname to [my_user_name][at]protonmail.com and I'll see what I can do to recover it. Sock puppets need not bother applying.

    Starting Score:    1  point
    Moderation   +4  
       Informative=4, Total=4
    Extra 'Informative' Modifier   0  
    Karma-Bonus Modifier   +1  
    Total Score:   5  

  • (Score: 1) by Runaway1956 on Saturday November 12 2022, @10:59AM (7 children)

    by Runaway1956 (2926) Subscriber Badge on Saturday November 12 2022, @10:59AM (#1279353) Journal

    Journal entries are missing. McGrew posted a journal entry just a couple days ago. DannyB and I both commented in that journal, and I just looked to see what DannyB may have replied. It's gone.

    I don't know how may others may have been affected.

  • (Score: 1) by Runaway1956 on Saturday November 12 2022, @11:04AM

    by Runaway1956 (2926) Subscriber Badge on Saturday November 12 2022, @11:04AM (#1279354) Journal

    Also, unable to connect to IRC here. "Disconnected (Remote host closed socket)."