Stories
Slash Boxes
Comments

SoylentNews is people

Meta
posted by martyb on Friday May 21 2021, @12:25AM   Printer-friendly

As many of you noticed, we had a site crash today. From around 1300 until 2200 UTC (2021-05-20).

A HUGE thank you goes to mechanicjay who spent the whole time trying to get our ndb (cluster) working again. It's an uncommon configuration, which made recovery especially challenging... there's just not a lot of documentation about it on the web.

I reached out and got hold of The Mighty Buzzard on the phone. Then put him in touch with mechanicjay who got us back up and running using backups.

Unfortunately, we had to go way back until April 14 to get a working backup. (I don't know all the details, but it appears something went sideways on neon).

We're all wiped out right now. When we have rested and had a chance to discuss things, we'll post an update.

In the meantime, please join me in thanking mechanicjay and TMB for all they did to get us up and running again!

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 5, Insightful) by Anonymous Coward on Friday May 21 2021, @12:46AM (21 children)

    by Anonymous Coward on Friday May 21 2021, @12:46AM (#1137422)

    Hard disk space has been an issue in the past. That's something that could break both a live database and backups.

    Starting Score:    0  points
    Moderation   +5  
       Insightful=4, Informative=1, Total=5
    Extra 'Insightful' Modifier   0  

    Total Score:   5  
  • (Score: 5, Informative) by mechanicjay on Friday May 21 2021, @01:05AM (18 children)

    ^this guy gets it.
    --
    My VMS box beat up your Windows box.
    • (Score: 1, Interesting) by Anonymous Coward on Friday May 21 2021, @01:28AM (17 children)

      by Anonymous Coward on Friday May 21 2021, @01:28AM (#1137444)

      How the hell can hard disk space be an issue on a low volume web site on even a basic hosting plan? Seriously, are you keeping logs from 5 years ago? And everything else. If you “must” keep that crap, for sentimental reasons, keep it off-site. It will make backup and restore much easier.

      Time to face the music and get something that’s more widely used and mostly just works out of the box. Remember groklaw? That ran geeklog, a proper CMS.

      So you won’t have user moderation. Big deal - it’s been games for decades. We criticized Amazon for trying to family their warehouses to get more out of workers, ditto Microsoft when they experimented with it. We criticize social media having degenerated in a game of who can get the most likes and followers. Slashcode is just as bad in this respect. People have been abusing the mod system since it was first introduced. Time to grow up, ditch the crap.

      Because it’s not just in Putin’s Russia that Slashcode shits on YOU!

      • (Score: 0) by Anonymous Coward on Friday May 21 2021, @01:37AM

        by Anonymous Coward on Friday May 21 2021, @01:37AM (#1137450)

        How the hell can hard disk space be an issue on a low volume web site...

        I'm thinking probably a few trillion ASCII Goatse posts.

      • (Score: 5, Insightful) by RedIsNotGreen on Friday May 21 2021, @06:50AM (13 children)

        by RedIsNotGreen (2191) on Friday May 21 2021, @06:50AM (#1137476) Homepage Journal

        User moderation is what made Slashdot what it was, and is essential to keeping this site what it is, IMO. I think you are wrong to write it off so casually.

        • (Score: 4, Informative) by istartedi on Friday May 21 2021, @07:10AM (12 children)

          by istartedi (123) on Friday May 21 2021, @07:10AM (#1137478) Journal

          There seems to be a move to do that here lately, and I don't get it. It's like the last vestige of the Internet before it became TV. I'm not generally a conspiratorial type; but it seems like TPTB want to kill Slash and anything like it so that we've got nothing to do but play the anger games you see on other sites.

          Really though, what does that have to do with backups? Aren't most backup solutions going to be agnostic of how the site operates?

          --
          Appended to the end of comments you post. Max: 120 chars.
          • (Score: 0, Interesting) by Anonymous Coward on Friday May 21 2021, @10:45AM (1 child)

            by Anonymous Coward on Friday May 21 2021, @10:45AM (#1137507)

            Slash is dead. The repeated crashes show you can’t even claim it’s in maintenance mode.

            User moderation is a joke. Too many people trying to game the system. It’s one of the reasons people maintain sock puppets. User moderation is no better than Facebook likes. It’s just noise at this point. An experiment that failed.

            And now all the posts regarding NCommanders observations in his resignation post, and all the questions it raised about how it was deleted from the story queue after the submission was accepted, are conveniently gone.

            “Because the backups over the last month are corrupted.”

            Unfortunately believable given the lack of any real technical professionalism on a “tech” site.

            • (Score: 2) by Mykl on Friday May 21 2021, @11:17PM

              by Mykl (1112) on Friday May 21 2021, @11:17PM (#1137651)

              But did they fake the Moon landing or not?

          • (Score: 0) by Anonymous Coward on Friday May 21 2021, @12:10PM (9 children)

            by Anonymous Coward on Friday May 21 2021, @12:10PM (#1137523)

            I think there's a big split in the community - the ones who always log in, disparage ACs, and love moderation, and those who don't (the AC posters and those who log in just to set the site to -1 for easy reading).

            • (Score: 1, Interesting) by Anonymous Coward on Friday May 21 2021, @02:02PM (8 children)

              by Anonymous Coward on Friday May 21 2021, @02:02PM (#1137535)

              Wanna bet that the AC posters include people who used to log in but are fed up? I know that was my motivation - I gave up on this site’s libtard “post any hate shit you want” policy. I wanted this site to succeed, but now it’s become a cautionary tale more than anything else.

              The emphasis on moderation over everything else (including the articles and journals) is a destructive distraction. You want to let someone know what you think, get your lazy fingers moving and post a reply. Moderation is about as ineffective as signing an online petition - its literally the least you can do except doing nothing at all. Slactivism. A motivation for running sock puppets.

              It also discourages logged in users from posting unpopular truths because they’ll be mod-bombed into silence for challenging the groupthink. There was a time when anyone who posted about how RMS hasn’t done anything useful in decades would be modded to oblivion. Now? Not so much.

              Same thing with the long-running mantra that open source HAD to be more secure. Now we know that all software is insecure and buggy by nature, irrespective of whether it’s open or closed, because writing great code is HARD.

              But useless user moderation is still a sacred cow for too many. “It’s what made slashdot great”. That is a LIE. What made it great was the tech boom, being in the right place at the right time. It still has user moderation , so why is it not still great? Because user moderation didn’t save it.

              Ask yourself why this site is in decline during a pandemic, when you should have tons of submissions and so many talented volunteers that you’d have to beat them off with a stick.

              Nobody with serious skills wants to work with someone like TMB. Or enable hate speech like EF. The first tentative steps towards addressing this are too little too late. I don’t want colleagues, friends, or family to associate me with such a clusterfuck. It’s not as bad as 4chan, but that’s just damning with the faintest of praise.

              • (Score: 0) by Anonymous Coward on Friday May 21 2021, @03:05PM

                by Anonymous Coward on Friday May 21 2021, @03:05PM (#1137551)

                when it was posted politely, u disagreed
                when it was posted in frustration, u called it flame
                when it is now posted withou any makup, u now call it hatespeech...

                -zug

              • (Score: 3, Insightful) by istartedi on Friday May 21 2021, @04:12PM (2 children)

                by istartedi (123) on Friday May 21 2021, @04:12PM (#1137579) Journal

                How are you defining "great"? Are you defining it as having a larger user base? If so, that comes with a lot of problems that would, IMHO, not make a Slash site great.

                These sites have limited appeal *and that's good*.

                Are you defining great in terms of the quality of the posts?

                I think we're on a spectrum there. There are other tech sites from which I learn more, but there's also a lot of very popular social media that's garbage.

                We're somewhere in the middle. There's nothing wrong with that. The Internet has plenty of room for Lambda The Ultimate (which is coincidentally also down at the moment), Soylent, and reddit. They serve different communities.

                As for hate speech, you can't call Soylent a failure there. Hate speech always comes along for the ride, and as long as it's not the generally endorsed view of the community it doesn't harm us. If you think that's the situation we're in, you haven't really looked at too many other sites.

                Is there some group-think here? Sure, but it's nothing like reddit or a lot of other places. If it were, I wouldn't be here.

                It's almost like you're looking at a completely different site. Cautionary tale??? I just don't see that.

                We're small, people speak their mind here, and not everything is agreeable to you. That's not failure.

                --
                Appended to the end of comments you post. Max: 120 chars.
                • (Score: 1, Insightful) by Anonymous Coward on Friday May 21 2021, @04:20PM (1 child)

                  by Anonymous Coward on Friday May 21 2021, @04:20PM (#1137580)

                  "Hate speech" is a bullshit term with no meaning, or rather, it means whatever the person who uses it wants to mean, which is simply: "Shut up!"

                  Our language is constantly being degraded by invented words with no meaning and also removing meaning from old words.

                  • (Score: 3, Insightful) by PiMuNu on Friday May 21 2021, @04:24PM

                    by PiMuNu (3823) on Friday May 21 2021, @04:24PM (#1137581)

                    Get thee to a nunnery

              • (Score: 0) by Anonymous Coward on Friday May 21 2021, @05:22PM

                by Anonymous Coward on Friday May 21 2021, @05:22PM (#1137591)

                I would not feel safe in posting opinion under a constant identifier on today's internet. It won't help you, but it can come to hurt you.

              • (Score: 1, Interesting) by Anonymous Coward on Friday May 21 2021, @05:30PM (1 child)

                by Anonymous Coward on Friday May 21 2021, @05:30PM (#1137593)

                Total Pilpul comment, or bot comment.

                "I gave up on this site’s libtard “post any hate shit you want” policy."

                This isn't the '90's anymore. Free speech is now a conservative policy, not a "libtard" one. If you don't believe me, look at Reddit and Facebook.

                "It also discourages logged in users from posting unpopular truths because they’ll be mod-bombed into silence for challenging the groupthink. There was a time when anyone who posted about how RMS hasn’t done anything useful in decades would be modded to oblivion. Now? Not so much."

                I remember that discussion, and compared to the others, moderation was totally balanced there, it being a charged topic. If you're gonna complain about mod imbalances and being modded into oblivion, then try criticizing cancel culture, woke bullshit, unnecessary lockdowns/vaccinations, and BLM riots. Then, you will truly see the downmod sockpuppetry at work.

                "Same thing with the long-running mantra that open source HAD to be more secure. Now we know that all software is insecure and buggy by nature, irrespective of whether it’s open or closed, because writing great code is HARD. "

                Linux: All the corporate ownership of Windows, with none of the functionality. "Stupid Windows users." "Yeah I know, Microsoft is a corporation and won't let you see the code. But with Linux there are many eyes on the code so it's secure." It isn't the year 2000 anymore, Linux is completely corporate and spook-owned. Look at its "sponsors" if you don't believe me.

                " But useless user moderation is still a sacred cow for too many. “It’s what made slashdot great”. That is a LIE. What made it great was the tech boom, being in the right place at the right time. It still has user moderation , so why is it not still great? Because user moderation didn’t save it. Ask yourself why this site is in decline during a pandemic, when you should have tons of submissions and so many talented volunteers that you’d have to beat them off with a stick.

                Hey stupid, just because you went woke doesn't mean that everybody else in the tech industry did. Anti-wokers are the new "niggers" of the internet. When the woke fad starts to wind down in a year or three because it's alienated everybody except you, I doub't you'll take away any lesson from it.

                "Nobody with serious skills wants to work with someone like TMB. Or enable hate speech like EF. The first tentative steps towards addressing this are too little too late. I don’t want colleagues, friends, or family to associate me with such a clusterfuck. It’s not as bad as 4chan, but that’s just damning with the faintest of praise."

                Huh, I seem to remember TMB saving your asses, so apparently some people DO want to work with TMB -- I'd hire him over anybody with pronouns and a BLM stamp in their resume, because I know he's gonna get shit done rather than find faults with everybody else. A lot of EF's comments are crap but he's made plenty of good ones, many of which brought the crybabies out of the woodwork to whine because he bruised their fragile egos. It's like being the jock in a room full of fat White kids. I think a lot of people just don't want to admit that they were wrong about woke bullshit as their narratives crumble all around them.

                • (Score: 0) by Anonymous Coward on Saturday May 22 2021, @09:32PM

                  by Anonymous Coward on Saturday May 22 2021, @09:32PM (#1137810)

                  Hahahaha

                  That explains why all the conservative moderated platforms ban people for ideological violations. As usual you nutbars have it backwards but believe the lies you're told.

              • (Score: 2) by Joe Desertrat on Thursday June 03 2021, @08:28PM

                by Joe Desertrat (2454) on Thursday June 03 2021, @08:28PM (#1141556)

                It also discourages logged in users from posting unpopular truths because they’ll be mod-bombed into silence for challenging the groupthink.

                Go with "unpopular opinions" instead of "unpopular truths" and maybe you might have a complaint. Most of the "mod-bombed" comments, if they are posted in serious discussions, are down modded because they have no real basis in fact, they are usually just rehashes of various already discredited ideas. If something gets down modded that probably shouldn't have been, it usually gets modded back to at least its starting point by those that disagree.

      • (Score: 5, Interesting) by bzipitidoo on Friday May 21 2021, @04:52PM

        by bzipitidoo (4388) on Friday May 21 2021, @04:52PM (#1137588) Journal

        You ever done any sysadmin work? I have. Data safety takes more space than you seem to realize. There is never enough space for all the things you ought to do to keep the data safe. You have to make compromises. RAIDs protect you from hard drive failures, but little else. They aren't backups, no matter how much some people feel they are. You need to keep daily backups, real ones, actual complete copies of the data, on separate machines, preferably in separate facilities, because there are disasters that can wipe out or disable an entire server farm.

        No, you DON'T throw away yesterday's backup the moment you have finished creating today's backup. You keep it for a while. What you want to do is keep 7 days worth of backups, and as each backup passes the age of 7 days, delete them, keeping only one to be your weekly backup. Keep 4 weekly backups, use one to be the monthly backup, and delete the other 3 after a month. That way, when someone finds a mistake, finds out that they accidentally deleted the wrong file, or edited out some crucial verbiage, perhaps 3 weeks later, you can go back and get a copy from before the mistake was made. At any moment, you're going to have at least a dozen or two dozen backups on hand. That takes a lot of storage space. Lot of companies, particularly small companies, can't do it, don't have the expertise or the equipment or money.

        What do you do when you run out of space? I was at a company when that happened, and the person who ran out of space took matters into his own hands. He deleted one of the backups, and TURNED OFF THE BACKUP PROCESSES. He had the authority to do that. I wasn't notified, nor did I receive any warnings frmo the monitoring software that alerts people when something is wrong with the backups, because he also turned that off. Didn't tell anyone, and he should have. That was the state of affairs for a week. Then it happened. Disaster. Another developer screwed up and accidentally erased our entire production database. How'd that happen, you might wonder? The developers had demanded, for their convenience and against the database admin's strenuous protests, no password protection on the database. The admin wanted to at least have that, but he was overruled.

        So this developer thought he was erasing and rebuilding the test database, and didn't realize until too late that he had his little script pointed at production. DROP TABLE on all the tables. Asking for a password would have stopped it. I was on a call with the database admin when it happened. We turned to check some little something on the website, only to discover it was gone! Was working fine just a moment before. Then there was a mad scramble to find out what the H was going on. Our first thought was that we'd somehow been hacked. But it soon became clear that if it was a hack job, it was very weird in that it left all the servers intact and running, all the login credentials unchanged. I found no evidence of intrusion anywhere. Soon the guilty developer confessed. Then it was discovered that the backups had been turned off. Our newest backup was a week old. The only thing that allowed the database admin to recover was that all the database activity had been logged on yet another machine. Took him 3 weeks to run the logs to catch up the week old backup, but he did it.

      • (Score: 0) by Anonymous Coward on Sunday May 23 2021, @01:35AM

        by Anonymous Coward on Sunday May 23 2021, @01:35AM (#1137868)

        How the hell can hard disk space be an issue on a low volume web site on even a basic hosting plan? Seriously, are you keeping logs from 5 years ago? And everything else. If you “must” keep that crap, for sentimental reasons, keep it off-site. It will make backup and restore much easier.

        Time to face the music and get something that’s more widely used and mostly just works out of the box. Remember groklaw? That ran geeklog, a proper CMS.

        So you won’t have user moderation. Big deal - it’s been games for decades. We criticized Amazon for trying to family their warehouses to get more out of workers, ditto Microsoft when they experimented with it. We criticize social media having degenerated in a game of who can get the most likes and followers. Slashcode is just as bad in this respect. People have been abusing the mod system since it was first introduced. Time to grow up, ditch the crap.

        Because it’s not just in Putin’s Russia that Slashcode shits on YOU!

        Probably all the space used to warehouse Ari's BS.

  • (Score: 1, Touché) by Anonymous Coward on Friday May 21 2021, @12:08PM

    by Anonymous Coward on Friday May 21 2021, @12:08PM (#1137522)

    Only if you keep the database and the backups on the same storage. In which case, they're not "backups", merely "data snapshots".

  • (Score: 2) by mcgrew on Saturday May 22 2021, @04:08PM

    by mcgrew (701) <publish@mcgrewbooks.com> on Saturday May 22 2021, @04:08PM (#1137758) Homepage Journal

    Hard disk space for a text-only site? I have a 4 terabyte drive I paid $200 for a couple of years ago, saw a 2 tb thumb drive advertised for fifty bucks recently. ALL the Star Trek series and movies are on my $200 drive, all the Star Wars movies (and hundreds of other full length movies), a dozen multi-year TV series, all the rest of my files (music, photos, books) and it's still nearly empty. Five of them would hold the entire Library of Congress.

    Thanks to those who worked to fix it. I just looked, the last of my journals listed was from January. Hope the missing ones come back.

    --
    mcgrewbooks.com mcgrew.info nooze.org