Stories
Slash Boxes
Comments

SoylentNews is people

Meta
posted by NCommander on Saturday November 12 2022, @08:43AM   Printer-friendly
from the its-in-flames dept.

So, quick update here. The site was down for most of the night because the database cluster shot itself in the head. I had restarted a machine to install updates, and this caused the backend cluster to entire to entirely loose its mind. Unfortunately, I didn't have a manual dump of the database made, just a VM snapshot, since, well, I wasn't tinkering with it directly. I've mostly been trying to patch things to the point that I can sleep, and leaving things down like IRC and email which need to be seriously overhauled before they can go back up.

As far as damages go, it looks like we lost 10 or so days of messages, which uh, sucks for multiple reasons. We're currently on ##soylentnews on Libera.Chat while I pull bits of the site out of the flames, but I'm at the point that if I don't sleep, I will make things worse. Corruption in the production database is very much not what I wanted, and we're very much in limp mode for the moment. I'm going to let staff handle IRC and comments while I sleep, and then I'll post another update when I'm awake.

See you in a few hours

~ NCommander

 
This discussion was created by NCommander (2) for logged-in users only, but now has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 2) by RS3 on Saturday November 12 2022, @07:54PM (1 child)

    by RS3 (6367) on Saturday November 12 2022, @07:54PM (#1279397)

    Ah, it's something special, not generic web / cloud hosting. Pretty cool! I do kind of remember you writing about it sometime long ago...

    I'm deploying a new server, after keeping some old ones going perfectly well for years. Older CentOS (6), updates stopped, but I've had no problems. One has uptime of 390 days right now. Ain't broke, not fixing it... Well, new server is big CPU and RAM, so hypervisor is needed. I've run VMWare, Xen, messed with a few others, need to stay away from big $ software licensing / subscription costs. Most likely going with kvm. Trying to steer away from systemd- partly because I hear too much bad, and that dovetails with this IT gig not being full-time, nor any regular hours. The _last_ thing I need is emergencies 15+ miles from where I live or work. IE, I'm spoiled by systems that just run, and there have been time periods when I haven't visited the physical site in more than 1.5 years.

    I've tried and like Devuan, and MX is pretty cool, but neither are really server-oriented. Love Alpine, wish for a better package manager. About to evaluate Void. Longtime Slackware user- I just worry that if I depart the situation, I don't want to leave something complicated for whoever takes over next...

    Starting Score:    1  point
    Karma-Bonus Modifier   +1  

    Total Score:   2  
  • (Score: 2) by JoeMerchant on Saturday November 12 2022, @08:52PM

    by JoeMerchant (3937) on Saturday November 12 2022, @08:52PM (#1279403)

    Yeah, our system is single user but we have deployed thousands of systems around the world, different animal from a single server with thousands of users, but it's remarkable how many tools apply to both worlds.

    We got a minor burn from our systemd service file behavior the other day, it had Rabbit / Docker as a dependancy for another service we run, never expected killing that service would also kill docker... Now that we know it's not a problem, but it was about 20 hours of developer investigation to deal with that bit of nonsense.

    --
    🌻🌻 [google.com]