Stories
Slash Boxes
Comments

SoylentNews is people

Meta
posted by martyb on Friday May 21, @12:25AM   Printer-friendly [Skip to comment(s)]

As many of you noticed, we had a site crash today. From around 1300 until 2200 UTC (2021-05-20).

A HUGE thank you goes to mechanicjay who spent the whole time trying to get our ndb (cluster) working again. It's an uncommon configuration, which made recovery especially challenging... there's just not a lot of documentation about it on the web.

I reached out and got hold of The Mighty Buzzard on the phone. Then put him in touch with mechanicjay who got us back up and running using backups.

Unfortunately, we had to go way back until April 14 to get a working backup. (I don't know all the details, but it appears something went sideways on neon).

We're all wiped out right now. When we have rested and had a chance to discuss things, we'll post an update.

In the meantime, please join me in thanking mechanicjay and TMB for all they did to get us up and running again!

Related Stories

Site Crash Recovery Progress Update: Databases and Subscriptions 39 comments

This is a follow-up to our site crash.

If you are just tuning in, SoylentNews experienced a database crash on 2021-05-20. We tried to restore from recent backups, but found they were corrupted and unusable. Thanks to heroic work by mechanicjay the site is back up and running!

Many thanks to mechanicjay for his 16-hour(!) day on Thursday to get us back up and then move us to a single back-end configuration. He didn't stop there, but has continued on gathering information and guiding work to get us to a more stable foundation. We'll keep you posted as to our progress.

Read on if interested... otherwise, another story will be along presently.

This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
(1) 2
  • (Score: 3, Interesting) by NateMich on Friday May 21, @12:28AM (27 children)

    by NateMich (6662) on Friday May 21, @12:28AM (#1137416)

    If I had to guess, I'd say that whatever broke your databases was also messing with your backups recently.
    I mean, unless you really only back up monthly or something.

    • (Score: 5, Insightful) by Anonymous Coward on Friday May 21, @12:46AM (21 children)

      by Anonymous Coward on Friday May 21, @12:46AM (#1137422)

      Hard disk space has been an issue in the past. That's something that could break both a live database and backups.

      • (Score: 5, Informative) by mechanicjay on Friday May 21, @01:05AM (18 children)

        ^this guy gets it.
        --
        My VMS box beat up your Windows box.
        • (Score: 1, Interesting) by Anonymous Coward on Friday May 21, @01:28AM (17 children)

          by Anonymous Coward on Friday May 21, @01:28AM (#1137444)

          How the hell can hard disk space be an issue on a low volume web site on even a basic hosting plan? Seriously, are you keeping logs from 5 years ago? And everything else. If you “must” keep that crap, for sentimental reasons, keep it off-site. It will make backup and restore much easier.

          Time to face the music and get something that’s more widely used and mostly just works out of the box. Remember groklaw? That ran geeklog, a proper CMS.

          So you won’t have user moderation. Big deal - it’s been games for decades. We criticized Amazon for trying to family their warehouses to get more out of workers, ditto Microsoft when they experimented with it. We criticize social media having degenerated in a game of who can get the most likes and followers. Slashcode is just as bad in this respect. People have been abusing the mod system since it was first introduced. Time to grow up, ditch the crap.

          Because it’s not just in Putin’s Russia that Slashcode shits on YOU!

          • (Score: 0) by Anonymous Coward on Friday May 21, @01:37AM

            by Anonymous Coward on Friday May 21, @01:37AM (#1137450)

            How the hell can hard disk space be an issue on a low volume web site...

            I'm thinking probably a few trillion ASCII Goatse posts.

          • (Score: 5, Insightful) by RedIsNotGreen on Friday May 21, @06:50AM (13 children)

            by RedIsNotGreen (2191) on Friday May 21, @06:50AM (#1137476) Homepage Journal

            User moderation is what made Slashdot what it was, and is essential to keeping this site what it is, IMO. I think you are wrong to write it off so casually.

            • (Score: 4, Informative) by istartedi on Friday May 21, @07:10AM (12 children)

              by istartedi (123) on Friday May 21, @07:10AM (#1137478) Journal

              There seems to be a move to do that here lately, and I don't get it. It's like the last vestige of the Internet before it became TV. I'm not generally a conspiratorial type; but it seems like TPTB want to kill Slash and anything like it so that we've got nothing to do but play the anger games you see on other sites.

              Really though, what does that have to do with backups? Aren't most backup solutions going to be agnostic of how the site operates?

              • (Score: 0, Interesting) by Anonymous Coward on Friday May 21, @10:45AM (1 child)

                by Anonymous Coward on Friday May 21, @10:45AM (#1137507)

                Slash is dead. The repeated crashes show you can’t even claim it’s in maintenance mode.

                User moderation is a joke. Too many people trying to game the system. It’s one of the reasons people maintain sock puppets. User moderation is no better than Facebook likes. It’s just noise at this point. An experiment that failed.

                And now all the posts regarding NCommanders observations in his resignation post, and all the questions it raised about how it was deleted from the story queue after the submission was accepted, are conveniently gone.

                “Because the backups over the last month are corrupted.”

                Unfortunately believable given the lack of any real technical professionalism on a “tech” site.

                • (Score: 2) by Mykl on Friday May 21, @11:17PM

                  by Mykl (1112) on Friday May 21, @11:17PM (#1137651)

                  But did they fake the Moon landing or not?

              • (Score: 0) by Anonymous Coward on Friday May 21, @12:10PM (9 children)

                by Anonymous Coward on Friday May 21, @12:10PM (#1137523)

                I think there's a big split in the community - the ones who always log in, disparage ACs, and love moderation, and those who don't (the AC posters and those who log in just to set the site to -1 for easy reading).

                • (Score: 1, Interesting) by Anonymous Coward on Friday May 21, @02:02PM (8 children)

                  by Anonymous Coward on Friday May 21, @02:02PM (#1137535)

                  Wanna bet that the AC posters include people who used to log in but are fed up? I know that was my motivation - I gave up on this site’s libtard “post any hate shit you want” policy. I wanted this site to succeed, but now it’s become a cautionary tale more than anything else.

                  The emphasis on moderation over everything else (including the articles and journals) is a destructive distraction. You want to let someone know what you think, get your lazy fingers moving and post a reply. Moderation is about as ineffective as signing an online petition - its literally the least you can do except doing nothing at all. Slactivism. A motivation for running sock puppets.

                  It also discourages logged in users from posting unpopular truths because they’ll be mod-bombed into silence for challenging the groupthink. There was a time when anyone who posted about how RMS hasn’t done anything useful in decades would be modded to oblivion. Now? Not so much.

                  Same thing with the long-running mantra that open source HAD to be more secure. Now we know that all software is insecure and buggy by nature, irrespective of whether it’s open or closed, because writing great code is HARD.

                  But useless user moderation is still a sacred cow for too many. “It’s what made slashdot great”. That is a LIE. What made it great was the tech boom, being in the right place at the right time. It still has user moderation , so why is it not still great? Because user moderation didn’t save it.

                  Ask yourself why this site is in decline during a pandemic, when you should have tons of submissions and so many talented volunteers that you’d have to beat them off with a stick.

                  Nobody with serious skills wants to work with someone like TMB. Or enable hate speech like EF. The first tentative steps towards addressing this are too little too late. I don’t want colleagues, friends, or family to associate me with such a clusterfuck. It’s not as bad as 4chan, but that’s just damning with the faintest of praise.

                  • (Score: 0) by Anonymous Coward on Friday May 21, @03:05PM

                    by Anonymous Coward on Friday May 21, @03:05PM (#1137551)

                    when it was posted politely, u disagreed
                    when it was posted in frustration, u called it flame
                    when it is now posted withou any makup, u now call it hatespeech...

                    -zug

                  • (Score: 3, Insightful) by istartedi on Friday May 21, @04:12PM (2 children)

                    by istartedi (123) on Friday May 21, @04:12PM (#1137579) Journal

                    How are you defining "great"? Are you defining it as having a larger user base? If so, that comes with a lot of problems that would, IMHO, not make a Slash site great.

                    These sites have limited appeal *and that's good*.

                    Are you defining great in terms of the quality of the posts?

                    I think we're on a spectrum there. There are other tech sites from which I learn more, but there's also a lot of very popular social media that's garbage.

                    We're somewhere in the middle. There's nothing wrong with that. The Internet has plenty of room for Lambda The Ultimate (which is coincidentally also down at the moment), Soylent, and reddit. They serve different communities.

                    As for hate speech, you can't call Soylent a failure there. Hate speech always comes along for the ride, and as long as it's not the generally endorsed view of the community it doesn't harm us. If you think that's the situation we're in, you haven't really looked at too many other sites.

                    Is there some group-think here? Sure, but it's nothing like reddit or a lot of other places. If it were, I wouldn't be here.

                    It's almost like you're looking at a completely different site. Cautionary tale??? I just don't see that.

                    We're small, people speak their mind here, and not everything is agreeable to you. That's not failure.

                    • (Score: 1, Insightful) by Anonymous Coward on Friday May 21, @04:20PM (1 child)

                      by Anonymous Coward on Friday May 21, @04:20PM (#1137580)

                      "Hate speech" is a bullshit term with no meaning, or rather, it means whatever the person who uses it wants to mean, which is simply: "Shut up!"

                      Our language is constantly being degraded by invented words with no meaning and also removing meaning from old words.

                  • (Score: 0) by Anonymous Coward on Friday May 21, @05:22PM

                    by Anonymous Coward on Friday May 21, @05:22PM (#1137591)

                    I would not feel safe in posting opinion under a constant identifier on today's internet. It won't help you, but it can come to hurt you.

                  • (Score: 1, Interesting) by Anonymous Coward on Friday May 21, @05:30PM (1 child)

                    by Anonymous Coward on Friday May 21, @05:30PM (#1137593)

                    Total Pilpul comment, or bot comment.

                    "I gave up on this site’s libtard “post any hate shit you want” policy."

                    This isn't the '90's anymore. Free speech is now a conservative policy, not a "libtard" one. If you don't believe me, look at Reddit and Facebook.

                    "It also discourages logged in users from posting unpopular truths because they’ll be mod-bombed into silence for challenging the groupthink. There was a time when anyone who posted about how RMS hasn’t done anything useful in decades would be modded to oblivion. Now? Not so much."

                    I remember that discussion, and compared to the others, moderation was totally balanced there, it being a charged topic. If you're gonna complain about mod imbalances and being modded into oblivion, then try criticizing cancel culture, woke bullshit, unnecessary lockdowns/vaccinations, and BLM riots. Then, you will truly see the downmod sockpuppetry at work.

                    "Same thing with the long-running mantra that open source HAD to be more secure. Now we know that all software is insecure and buggy by nature, irrespective of whether it’s open or closed, because writing great code is HARD. "

                    Linux: All the corporate ownership of Windows, with none of the functionality. "Stupid Windows users." "Yeah I know, Microsoft is a corporation and won't let you see the code. But with Linux there are many eyes on the code so it's secure." It isn't the year 2000 anymore, Linux is completely corporate and spook-owned. Look at its "sponsors" if you don't believe me.

                    " But useless user moderation is still a sacred cow for too many. “It’s what made slashdot great”. That is a LIE. What made it great was the tech boom, being in the right place at the right time. It still has user moderation , so why is it not still great? Because user moderation didn’t save it. Ask yourself why this site is in decline during a pandemic, when you should have tons of submissions and so many talented volunteers that you’d have to beat them off with a stick.

                    Hey stupid, just because you went woke doesn't mean that everybody else in the tech industry did. Anti-wokers are the new "niggers" of the internet. When the woke fad starts to wind down in a year or three because it's alienated everybody except you, I doub't you'll take away any lesson from it.

                    "Nobody with serious skills wants to work with someone like TMB. Or enable hate speech like EF. The first tentative steps towards addressing this are too little too late. I don’t want colleagues, friends, or family to associate me with such a clusterfuck. It’s not as bad as 4chan, but that’s just damning with the faintest of praise."

                    Huh, I seem to remember TMB saving your asses, so apparently some people DO want to work with TMB -- I'd hire him over anybody with pronouns and a BLM stamp in their resume, because I know he's gonna get shit done rather than find faults with everybody else. A lot of EF's comments are crap but he's made plenty of good ones, many of which brought the crybabies out of the woodwork to whine because he bruised their fragile egos. It's like being the jock in a room full of fat White kids. I think a lot of people just don't want to admit that they were wrong about woke bullshit as their narratives crumble all around them.

                    • (Score: 0) by Anonymous Coward on Saturday May 22, @09:32PM

                      by Anonymous Coward on Saturday May 22, @09:32PM (#1137810)

                      Hahahaha

                      That explains why all the conservative moderated platforms ban people for ideological violations. As usual you nutbars have it backwards but believe the lies you're told.

                  • (Score: 2) by Joe Desertrat on Thursday June 03, @08:28PM

                    by Joe Desertrat (2454) on Thursday June 03, @08:28PM (#1141556)

                    It also discourages logged in users from posting unpopular truths because they’ll be mod-bombed into silence for challenging the groupthink.

                    Go with "unpopular opinions" instead of "unpopular truths" and maybe you might have a complaint. Most of the "mod-bombed" comments, if they are posted in serious discussions, are down modded because they have no real basis in fact, they are usually just rehashes of various already discredited ideas. If something gets down modded that probably shouldn't have been, it usually gets modded back to at least its starting point by those that disagree.

          • (Score: 5, Interesting) by bzipitidoo on Friday May 21, @04:52PM

            by bzipitidoo (4388) Subscriber Badge on Friday May 21, @04:52PM (#1137588) Journal

            You ever done any sysadmin work? I have. Data safety takes more space than you seem to realize. There is never enough space for all the things you ought to do to keep the data safe. You have to make compromises. RAIDs protect you from hard drive failures, but little else. They aren't backups, no matter how much some people feel they are. You need to keep daily backups, real ones, actual complete copies of the data, on separate machines, preferably in separate facilities, because there are disasters that can wipe out or disable an entire server farm.

            No, you DON'T throw away yesterday's backup the moment you have finished creating today's backup. You keep it for a while. What you want to do is keep 7 days worth of backups, and as each backup passes the age of 7 days, delete them, keeping only one to be your weekly backup. Keep 4 weekly backups, use one to be the monthly backup, and delete the other 3 after a month. That way, when someone finds a mistake, finds out that they accidentally deleted the wrong file, or edited out some crucial verbiage, perhaps 3 weeks later, you can go back and get a copy from before the mistake was made. At any moment, you're going to have at least a dozen or two dozen backups on hand. That takes a lot of storage space. Lot of companies, particularly small companies, can't do it, don't have the expertise or the equipment or money.

            What do you do when you run out of space? I was at a company when that happened, and the person who ran out of space took matters into his own hands. He deleted one of the backups, and TURNED OFF THE BACKUP PROCESSES. He had the authority to do that. I wasn't notified, nor did I receive any warnings frmo the monitoring software that alerts people when something is wrong with the backups, because he also turned that off. Didn't tell anyone, and he should have. That was the state of affairs for a week. Then it happened. Disaster. Another developer screwed up and accidentally erased our entire production database. How'd that happen, you might wonder? The developers had demanded, for their convenience and against the database admin's strenuous protests, no password protection on the database. The admin wanted to at least have that, but he was overruled.

            So this developer thought he was erasing and rebuilding the test database, and didn't realize until too late that he had his little script pointed at production. DROP TABLE on all the tables. Asking for a password would have stopped it. I was on a call with the database admin when it happened. We turned to check some little something on the website, only to discover it was gone! Was working fine just a moment before. Then there was a mad scramble to find out what the H was going on. Our first thought was that we'd somehow been hacked. But it soon became clear that if it was a hack job, it was very weird in that it left all the servers intact and running, all the login credentials unchanged. I found no evidence of intrusion anywhere. Soon the guilty developer confessed. Then it was discovered that the backups had been turned off. Our newest backup was a week old. The only thing that allowed the database admin to recover was that all the database activity had been logged on yet another machine. Took him 3 weeks to run the logs to catch up the week old backup, but he did it.

          • (Score: 0) by Anonymous Coward on Sunday May 23, @01:35AM

            by Anonymous Coward on Sunday May 23, @01:35AM (#1137868)

            How the hell can hard disk space be an issue on a low volume web site on even a basic hosting plan? Seriously, are you keeping logs from 5 years ago? And everything else. If you “must” keep that crap, for sentimental reasons, keep it off-site. It will make backup and restore much easier.

            Time to face the music and get something that’s more widely used and mostly just works out of the box. Remember groklaw? That ran geeklog, a proper CMS.

            So you won’t have user moderation. Big deal - it’s been games for decades. We criticized Amazon for trying to family their warehouses to get more out of workers, ditto Microsoft when they experimented with it. We criticize social media having degenerated in a game of who can get the most likes and followers. Slashcode is just as bad in this respect. People have been abusing the mod system since it was first introduced. Time to grow up, ditch the crap.

            Because it’s not just in Putin’s Russia that Slashcode shits on YOU!

            Probably all the space used to warehouse Ari's BS.

      • (Score: 1, Touché) by Anonymous Coward on Friday May 21, @12:08PM

        by Anonymous Coward on Friday May 21, @12:08PM (#1137522)

        Only if you keep the database and the backups on the same storage. In which case, they're not "backups", merely "data snapshots".

      • (Score: 2) by mcgrew on Saturday May 22, @04:08PM

        by mcgrew (701) <publish@mcgrewbooks.com> on Saturday May 22, @04:08PM (#1137758) Homepage Journal

        Hard disk space for a text-only site? I have a 4 terabyte drive I paid $200 for a couple of years ago, saw a 2 tb thumb drive advertised for fifty bucks recently. ALL the Star Trek series and movies are on my $200 drive, all the Star Wars movies (and hundreds of other full length movies), a dozen multi-year TV series, all the rest of my files (music, photos, books) and it's still nearly empty. Five of them would hold the entire Library of Congress.

        Thanks to those who worked to fix it. I just looked, the last of my journals listed was from January. Hope the missing ones come back.

        --
        Free Martian whores! [mcgrewbooks.com]
    • (Score: 5, Funny) by Rosco P. Coltrane on Friday May 21, @05:49AM (4 children)

      by Rosco P. Coltrane (4757) on Friday May 21, @05:49AM (#1137458)

      I'll tell you what sucks: had I known this would happen, I could have trolled the living daylights out of this here forum for over a month and gotten my slate wiped clean today :)

      • (Score: 0) by Anonymous Coward on Friday May 21, @02:09PM (3 children)

        by Anonymous Coward on Friday May 21, @02:09PM (#1137537)
        Go for it - it’s only a matter of time before it crashes again. No proper backups, no proper docs, does anyone want to bet TMB doesn’t know how to use utilities that have been around for ages like SVN or Mantis or Doxygen? I’m not talking new cutting edge stuff here, but stuff that even “old skool” coders have used.

        Nobody’s going to want to work with someone who is going to bitch about having to use any sort of structure in their workplace processes like version control. Or a bug tracker.

        • (Score: 4, Insightful) by Rosco P. Coltrane on Friday May 21, @03:20PM

          by Rosco P. Coltrane (4757) on Friday May 21, @03:20PM (#1137559)

          How much do you pay for the quality of service you seem to expect, that you sarcastically point to the lack of?

          Yeah, that's what I thought.

        • (Score: 2) by bzipitidoo on Sunday May 23, @02:25AM (1 child)

          by bzipitidoo (4388) Subscriber Badge on Sunday May 23, @02:25AM (#1137873) Journal

          Subversion?? WTF? Do you know what you're talking about? That, and those other 2 tools you mention are software development tools, not system administration tools. A software repository is NOT a backup! Subversion isn't even the best, git is better. And git isn't even that new any more, been around for 15 years. git is distributed, cloning the repository in such a way that individual copies can act as backups. Not Subversion. Subversion is much more centralized. Lose that master repository, and you're going to have a rough time reconstructing it unless there is a backup.

          Backups are done with tools such as rsync. For monitoring, we used Nagios. Scheduling can be done with cron. Many other things were done with little scripts.

          • (Score: 0) by Anonymous Coward on Sunday May 23, @06:15AM

            by Anonymous Coward on Sunday May 23, @06:15AM (#1137906)

            That just a troll trolling, better off not to feed it.

            A completely unrelated point SVN == cathedral && git == bazaar

  • (Score: 3, Funny) by Anonymous Coward on Friday May 21, @12:31AM (35 children)

    by Anonymous Coward on Friday May 21, @12:31AM (#1137417)

    Especially TMB - throwing moves as most reluctant disco dancer ever.

    • (Score: 5, Insightful) by Tork on Friday May 21, @01:09AM (33 children)

      by Tork (3914) on Friday May 21, @01:09AM (#1137439)
      Just wanted to second the show of gratitude to TMB. Thanks, man.
      --
      Slashdolt Logic: "24 year old jokes about sharks and lasers are +5, Funny." 💩
      • (Score: -1, Flamebait) by Anonymous Coward on Friday May 21, @06:03AM (31 children)

        by Anonymous Coward on Friday May 21, @06:03AM (#1137461)

        It was his crap setup that probably caused this in the first place. AND I predicted it would probably happen in advance the next time Linode did network maintenance. But instead of listening and learning to the guy who has run high availability clusters professionally, he got all high and mighty about how I was misinformed. I just hope that whomever is in charge this time actually reads the manual and doesn't make basic mistakes like he did.

        • (Score: 0) by Anonymous Coward on Friday May 21, @06:10AM (20 children)

          by Anonymous Coward on Friday May 21, @06:10AM (#1137464)

          *yawn*

          Have you ever volunteered? No? OK then, we can safely ignore your prescience that can't be proved. Like every other medium who predicts catastrophies, you're probably just another charlatan.

          • (Score: 0) by Anonymous Coward on Friday May 21, @06:27AM (19 children)

            by Anonymous Coward on Friday May 21, @06:27AM (#1137470)

            Except it was documented on this site and I'm not the first nor only one to notice the problem pattern with their setup. And why would I have volunteered to butt heads with someone who apparently hasn't read the docs and doesn't understand the apparent basics of HA clusters like this.

            • (Score: 3, Touché) by janrinok on Friday May 21, @08:36AM (18 children)

              by janrinok (52) Subscriber Badge on Friday May 21, @08:36AM (#1137488) Journal

              I'm not about to go searching through every AC comment to find where you claim it is "documented".

              And it appears that you have already reached the conclusion of what caused the problem and who is responsible. If it was a full hard drive then, I believe, somebody on the current team should have been responsible for checking for that possible occurrence and taking the appropriate action before we get to a critical stage. And in such a case that person would be responsible - not TMB who is no longer managing our hardware and therefore cannot be responsible for our day-to-day running. And when the site was first set up there were quite a few people responsible for structuring the hardware to meet the requirements of the code that we inherited. TMB was only one of those involved.

              True, we can restructure the entire hardware and software configuration but that takes qualified people and there are precious few of them volunteering to give up any of their time to help this site. Fortunately we had 2 people - mechanicjay and TMB - who have given willingly and freely of their personal time over the last day or so. If you are volunteering to help, then you must understand that we cannot simply accept the word of an AC - you will have to be a little bit more approachable than that. You will be welcome in that event but, if not, then you can continue to rant on here as an AC and we will continue to take everything you say with a healthy pinch of salt.

              The site will crash again - of course, I don't know when or why. Making such a claim does not qualify me to operate as a sysadmin on this site, however by making this forecast I have now got exactly the same provable credentials as yourself.

              --
              It's always my fault...
              • (Score: -1, Flamebait) by Anonymous Coward on Friday May 21, @09:44AM (17 children)

                by Anonymous Coward on Friday May 21, @09:44AM (#1137497)

                Keep making excuses. You don’t need to search for any “documentation “ - the site crashes on a regular basis, and everyone knows it.

                And as a parent poster pointed out, nobody is going to volunteer to fix it if it means continually arguing with Mr “Proud I Don’t Need An Education” Buzztard.

                But stick with the current “plan”. Where you’ve built up so much technical debt that recovery is impossible. Because there’s no fool like an old fool.

                • (Score: 4, Informative) by janrinok on Friday May 21, @10:21AM (12 children)

                  by janrinok (52) Subscriber Badge on Friday May 21, @10:21AM (#1137502) Journal

                  You've read something that I didn't say.

                  We ARE looking at how to improve both the system configuration and the software that we use. We are not, however, simply throwing everything away to start from scratch again. The system can be simplified which should result in a more robust site. As mechanicjay has stated elsewhere, some of the software that is installed to provide resilience is actually causing more problems than it is intended to solve. We can get rid of that straight away. A content management system should be able to work with any chosen database, and that includes MySQL, so there is nothing that I am aware of to suggest that MySQL cannot fulfil the role we are asking of it. People may have their own personal preferences but changing the database will require changes to the perl code which will all need writing and testing.

                  There is a problem with documentation but that is also linked to the lack of staff that we currently have, and that goes back at least 3 years. What few staff we have are currently kept busy keeping the site going and, although we are aware of areas where work needs to be done it can only be done by those who understand the system configuration. You can only write more documentation when you have people who understand what it is they are documenting. We need more sysadmins because system failures do not occur when the only active sysadmin is sat at his computer with nothing better to do.

                  We need more programmers who are prepared to volunteer to help support the site. It doesn't matter which language the site is written in, we will still need programmers to do that work. We always need more editors - although even with just the handful that we currently have we a looking relatively well manned compared with every other part of the support team. QA is a one-man team - MartyB again, who actually fills several more roles in the team at the same time.

                  You definitely have got a bug about TMB though - in case you missed it he is no longer part of the support team although he remains a member of our community and he kindly gave advice to mechanicjay during the last 24 hours. If you have had your nose put out of joint during your earlier discussions with him then that is a personal matter between you two.

                  But stick with the current “plan”. Where you’ve built up so much technical debt that recovery is impossible. Because there’s no fool like an old fool.

                  So you can see that, far from your claim, we are not sticking with the current plan. With the limited resources that we have we will make progress at as fast a rate as is possible. When the site first went active there were 20-30 active participants who were all contributing to keeping the site going. I reckon that we have less than 10 available today. Rather than sitting back and criticising as an AC, wouldn't you prefer to join the team and help fix some of the problems?

                  --
                  It's always my fault...
                  • (Score: -1, Redundant) by Anonymous Coward on Friday May 21, @11:05AM (11 children)

                    by Anonymous Coward on Friday May 21, @11:05AM (#1137513)
                    Geeklog works just fine with MySQL and Marian, among others. Slash, in it’s current state, obviously is way too far gone to bother fixing. There comes a time in many software projects where you have to throw everything out and do it right, using the lessons learned.

                    But keep telling yourself that slash can be fixed. You need proper devs, something you haven’t had in years, who wouldn’t put up with wishful thinking but will speak the hard truths borne of the confidence of experience.

                    There are other CMS that will also do a decent job. But NONE OF THEM ARE WRITTEN IN PERL. So the language issue is entirely relevant. Because software has to be maintained. And nobody wants to use Perl for large projects any more. Not when there are better alternatives.

                    If your code is so great why isn’t anyone else running it? Because it’s an in maintained pile of patches over patches.

                    The month of corrupted backups is just a symptom, another red flag. But keep making excuses - reality will continue to bite, and bite increasingly harder. There are hard decisions to be made, and you either make them now or events will make them for you. The future waits for nobody.

                    • (Score: 2) by janrinok on Friday May 21, @11:37AM (6 children)

                      by janrinok (52) Subscriber Badge on Friday May 21, @11:37AM (#1137520) Journal

                      Rehash was not the cause of the latest crash (as far as we can ascertain) - it is not where the focus is at present. That is not to say it will never be replaced but, for the time being, it is still working as expected. Currently, we have not got the resources to replace Rehash with a different language or package. If it ain't broke, don't fix it.

                      You are focussing on an area that is not causing us a problem at the moment. The system configuration is where we continue to encounter problems and that is where mechanicjay is currently concentrating his efforts.

                      --
                      It's always my fault...
                      • (Score: 0) by Anonymous Coward on Friday May 21, @01:58PM (5 children)

                        by Anonymous Coward on Friday May 21, @01:58PM (#1137534)

                        If it ain't broke, don't fix it.

                        More often than not, this actually means "the fix is too hard, so it cannot possibly be broken".

                        • (Score: 2) by janrinok on Friday May 21, @02:08PM (4 children)

                          by janrinok (52) Subscriber Badge on Friday May 21, @02:08PM (#1137536) Journal

                          Rehash is working today as advertised, the system configuration isn't - with very limited resources which one would you work on first?

                          --
                          It's always my fault...
                          • (Score: 0) by Anonymous Coward on Friday May 21, @02:34PM (3 children)

                            by Anonymous Coward on Friday May 21, @02:34PM (#1137544)
                            It broke … again. And nobody else is using the code anyway. Why not? Because it’s fragile, and too much Perl causes brain damage. Someone mentioned pipedot.org as an example - a site that has been inactive since April 2017.

                            Another post mentioned using a separate process to update story counts. Someone needs to learn to code better, and brushing up on sql as well. Just because the original devs didn’t know how to do it right is no excuse to preserve shit like that. This is 2021, not 1995.

                            TMB fücked up by not using a LIMIT clause in SQL that would have avoided time-outs under load. Experienced devs will ALWAYS seek ways that guarantee the most efficient use of resources because they don’t want intermittent bugs. Rehash is a total hash. Either learn to code or get something that other people are maintaining because it’s widely used, in a language that is widely used for web development. But you won’t. You will continue to ignore the red flags.

                            Why the resistance to a clean-sheet rethink of the site? Articles, user comments , and user journals are the only essentials. The polls suck, but most CMS packages contain poll functionality, so keep pills if you must. But do you really want to waste part of your life dealing with stupid complaints about unfair moderation? What a time sink! Dump it. It’s far from essential, and keeping it didn’t preserve slashdot’s ability to generate the slashdot effect.

                            If you think that user moderation is the killer feature that keeps people on the site, well, it ain’t working here, same as it didn’t on the green site. Is it SO hard to grab a copy of geeklog and skin it so it looks the way you want while still allowing the essentials - stories, comments, and journals? It’s a one-day job (with breaks).

                            What do you have to lose at this point?

                            • (Score: 3, Insightful) by janrinok on Friday May 21, @04:39PM (1 child)

                              by janrinok (52) Subscriber Badge on Friday May 21, @04:39PM (#1137584) Journal

                              We are, at this very moment, discussing options on a private channel. And currently ALL of our resources are currently working on recovering from yesterday, or keeping the site going today.

                              The only thing that is causing a problem (repeatedly) is one element of the system configuration that is not providing us with any benefit whatsoever - so that is what we are currently working on removing. The rest of the site is working just as we want it to. Let me explain it in an auto analogy - which is the traditional way of doing things around here. What you are suggesting is that we currently have a flat tire but you are recommending that we also paint the car, change the upholstery and fit a new engine too.

                              If it can be done in a day I will await your contribution by, shall we say, Sunday evening? Show me something working to convince me - not just make ridiculous suggestions that we haven't got the resources to complete anyway.

                              --
                              It's always my fault...
                              • (Score: 0) by Anonymous Coward on Friday May 21, @10:46PM

                                by Anonymous Coward on Friday May 21, @10:46PM (#1137643)

                                The only thing that is causing a problem (repeatedly) is one element of the system configuration that is not providing us with any benefit whatsoever - so that is what we are currently working on removing.

                                Perhaps that is for the best since you all apparently don't know how to use it properly. Quite a number of people use it under higher loads with better uptimes, after all.

                            • (Score: 0) by Anonymous Coward on Saturday May 22, @05:03PM

                              by Anonymous Coward on Saturday May 22, @05:03PM (#1137762)

                              ...and too much Perl causes brain damage.

                              Ah yes, but only when it comes to inferior brains

                    • (Score: 5, Insightful) by martyb on Friday May 21, @02:30PM (3 children)

                      by martyb (76) Subscriber Badge on Friday May 21, @02:30PM (#1137543) Journal

                      One thing to keep in mind is the "heritage" of our code.

                      I was with /. before it even had userids! I've witnessed all kinds of attacks on the site. Page-widening trolls. Actual SPAM comments. Mod bombs. Whatever creative nerds could come up with, they threw it at /. and changes were made to mitigate them. It stood up under heavy fire.

                      Slashcode begat rehash which is the open-source, freely available code that powers this site. Our foundation is solid.

                      Also, there is MUCH MUCH more going on behind the scenes. I dare say the admin interface has AT LEAST as much going on as what is presented to the community. Quite possibly twice (or thrice) as much. Every once in a while I find yet-another setting or configuration that could be tweaked!

                      The foundation is solid.

                      Admittedly, the site would benefit from some tuning. When SoylentNews started, it was difficult to foresee what areas would grow fastest and what needed to be allocated. I mean, here is my first comment on the site: comment 255 [soylentnews.org], and here I am replying to comment number 1,137,513!

                      Remember, too, this site is run by volunteers in their spare time.

                      Sure, it would be wonderful to have paid, full-time staff monitoring the site 24/7/365 like on Reddit or the like. How much would that cost per year? 3 x 8-hour shifts per day x 365 days-per-year is 8760 hours. At $15.00 per hour (dirt cheap for these kinds of skills!) that works out to $131,400 per year! And that does not even include server hosting costs! More realistically, at just $30.00 per hour, that works out to $1,839,600 per year! And that does not even include hosing costs.

                      SoylentNews gets by on just $7,000 for an entire year!. And that includes the annual costs of being incorporated, filing taxes,hosting expense, everything!

                      .

                      --
                      Wit is intellect, dancing.
                      • (Score: 4, Informative) by martyb on Friday May 21, @02:38PM

                        by martyb (76) Subscriber Badge on Friday May 21, @02:38PM (#1137546) Journal

                        Oops! His submit instead of preview.

                        s/hosing/hosting/

                        s/131,400/919,800/

                        There's prolly some more; it was a LONG day yesterday!

                        --
                        Wit is intellect, dancing.
                      • (Score: 0) by Anonymous Coward on Friday May 21, @03:07PM (1 child)

                        by Anonymous Coward on Friday May 21, @03:07PM (#1137552)
                        Marty, I like you, but seriously, there are SO many flaws in your post.

                        Yes, it’s expensive keeping a full-time dev on the payroll. That’s why hobby sites like soylent don’t do that - they use widely used open source CMS packages that have proper documentation, a developer community, and use a broadly used language combo - the most popular being written in php and any MySQL or PostgreSQL variant.

                        Not slash. Not rehash. Perl lost the race a long time ago.

                        What do users want? Articles, the ability to post comments, and journals. Plenty of CMS packages using php and a database server can do that without the legacy of Perl.

                        Grab a copy of geeklog and play around with it. You should be able to have a functional site with stories, comments , and journals. And of c, the administrative backend contains all the functionality you want to hide from users.

                        About Geeklog

                        Geeklog is an open source application for managing dynamic web content. It is written in PHP and supports MySQL or PostgreSQL as the database backend.

                        "Out of the box", Geeklog is a CMS, or a blog engine with support for comments, trackbacks, multiple syndication formats, spam protection, and all the other vital features of such a system.

                        The core Geeklog distribution can easily be extended by the many community developed plugins and other add-ons to radically alter its functionality. Available plugins include forums, image galleries, and many more.

                        This is what you use when you can’t afford to keep a team of developers on staff. Php has a wide user base, so you might actually attract developers, because nobody wants to screw around with Perl. The whole “TMTOWTDI” is a bug, not a feature.

                        You might even want to give the site a new, fresher look.

                        Seriously, give it a try. Take a shitbox computer, install Linux or FreeBSD on it, and give geeklog a try. It worked for groklaw under traffic you can only dream of. Don’t be fooled by groklaw’s blah appearance. If you know HTML and CSS, and have any graphics talent , you can make it look clean and modern and spiffy as all. Icons for stories? Screw that - real images or graphics that the text wraps around. (You can still keep the topic icons if you must, but they’re really dated).

                        As for the whole editorial process, you’d best run a private copy for the editors to edit submissions before someone posts them to the main site. I get that the subs queue is there so people can check before submitting a story, but multiple submissions are a good thing if they contain more information. You’ll probably end up dropping ICQ if editors can see what their proposed stories and edits and included graphics look like, and other editors can cut n paste and change and tweak it, and see the changes right there in the thread in their comments.

                        • (Score: 1, Insightful) by Anonymous Coward on Saturday May 22, @11:04AM

                          by Anonymous Coward on Saturday May 22, @11:04AM (#1137719)

                          Those php sites get pwned every now and then too.

                          Geeklog's security track record is crap and the types of vulnerabilities are not confidence inspiring: https://www.google.com/search?q=%22Geeklog%22+exploit [google.com]

                          Go get a clue. If you're not going to spend much time and money on a site you don't pick shit that needs to be patched every month.

                • (Score: 1) by khallow on Saturday May 22, @11:35PM (3 children)

                  by khallow (3766) Subscriber Badge on Saturday May 22, @11:35PM (#1137836) Journal

                  And as a parent poster pointed out, nobody is going to volunteer to fix it if it means continually arguing with Mr “Proud I Don’t Need An Education” Buzztard.

                  Even when TMB was in house, nobody was continually arguing with Mr. "Proud". I wonder how much else of your narrative is just as imaginary?

                  • (Score: 0) by Anonymous Coward on Sunday May 23, @06:09AM (2 children)

                    by Anonymous Coward on Sunday May 23, @06:09AM (#1137905)

                    And then you wonder why a system that has a five nines guarantee in a 2/2/2 setup doesn't even have two.

                    • (Score: 1) by khallow on Sunday May 23, @01:14PM (1 child)

                      by khallow (3766) Subscriber Badge on Sunday May 23, @01:14PM (#1137942) Journal

                      And then you wonder why a system that has a five nines guarantee in a 2/2/2 setup doesn't even have two.

                      I already know of real world systems - the Space Shuttle, that failed that hard. There's no wondering over here.

                      • (Score: 0) by Anonymous Coward on Monday May 24, @12:37AM

                        by Anonymous Coward on Monday May 24, @12:37AM (#1138082)

                        Because operating on the edge of science and technology at the extremes of risk with single points of failure meeting Swiss Cheese model of reality is directly analogous to running an incorrectly deployed bog-standard cluster deployment that is failing to meet its uptime guarantees despite hundreds of thousands of deployments operating successfully in worse conditions when they do deploy it correctly.

                        Right.

        • (Score: 5, Insightful) by Tork on Friday May 21, @06:15AM (9 children)

          by Tork (3914) on Friday May 21, @06:15AM (#1137465)

          Whether he is blameless or not the fact is he quit. I've butted heads with him before, so it's not like I'm being charitable when I say: I can't think of a reason to fault him for lending a hand.

          --
          Slashdolt Logic: "24 year old jokes about sharks and lasers are +5, Funny." 💩
          • (Score: -1, Troll) by Anonymous Coward on Friday May 21, @06:20AM (8 children)

            by Anonymous Coward on Friday May 21, @06:20AM (#1137468)

            I'm not faulting him for lending a hand, it's everything else that got this site to this point.

            • (Score: 0, Disagree) by Anonymous Coward on Friday May 21, @11:28AM (7 children)

              by Anonymous Coward on Friday May 21, @11:28AM (#1137519)
              It’s Stockholm Syndrome. Or Battered Spouse Syndrome. TMB shit on everything for so long that people were grateful when he stopped/quit. He quit, and donations that had been stalled since the beginning of the month started again, as people voted their approval of his leaving with their money.

              Money talks.

              • (Score: 0) by Anonymous Coward on Friday May 21, @02:17PM

                by Anonymous Coward on Friday May 21, @02:17PM (#1137540)

                What utter nonsense. This site is run by volunteers and the staff (including TMB) have been excellent.

              • (Score: 2) by janrinok on Friday May 21, @02:21PM (4 children)

                by janrinok (52) Subscriber Badge on Friday May 21, @02:21PM (#1137542) Journal

                MartyB is responsible for recording donations, and he has been rather busy for the last year or two - not TMB. Anything you else you want to accuse TMB of while you are at it?

                --
                It's always my fault...
                • (Score: 1, Interesting) by Anonymous Coward on Friday May 21, @03:19PM (3 children)

                  by Anonymous Coward on Friday May 21, @03:19PM (#1137558)
                  Whoosh! TMB quit, and donations-which had been stalled, started again. Talk about missing the point.

                  Plenty of people saw buzz as a large part of the problem. After a week of “playing nice” he packed it in.

                  If you had even ONE serious developer on tap you could have a different CMS up and running in a day, stories, comments, user journals, stupid polls, etc. ONE DAY. Take a few weeks to get user feedback, tweak things a bit, etc.

                  But NOBODY with a clue wants to maintain a mess of obsolete Perl that has no documentation. It’s one of those cases where you have to close your eyes and ignore the sink costs of the time and emotions tied up in rehash.

                  Software development is brutal. You can’t be sentimental over old code. You need to “knife the baby” on a regular basis to progress. Otherwise we’d still be stuck with GWBASIC.

                  • (Score: 3, Informative) by janrinok on Friday May 21, @04:49PM

                    by janrinok (52) Subscriber Badge on Friday May 21, @04:49PM (#1137587) Journal

                    The Perl code is working as expected. This is a system configuration problem. Stop trying to fix the wrong problem.

                    --
                    It's always my fault...
                  • (Score: 5, Informative) by DECbot on Saturday May 22, @03:06AM

                    by DECbot (832) on Saturday May 22, @03:06AM (#1137680) Journal

                    To add to what Janrinok has said, slashcode regularly handled a traffic load that would take offline mainstream static sites and ddos ISPs with less resources then what people throw at a regular Wordpress or Drupal sites today. The perl code is proven, mysql is proven, but there is a configuration issue causing stability issues with the SN implementation of the slashcode. That is what needs to be addressed. It's not time to replace the horse cause it threw an ill fitting shoe.

                    --
                    cats~$ sudo chown -R us /home/base
                  • (Score: 2) by bzipitidoo on Sunday May 23, @03:08AM

                    by bzipitidoo (4388) Subscriber Badge on Sunday May 23, @03:08AM (#1137879) Journal

                    One day, for one person to do a migration to another CMS? You're dreaming. I am not familiar with CMSes, but if it's anything like a database migration, you're talking weeks at the least. Yeah, one expert could migrate a little toy of a database in one day, but a real one with millions of rows and complexities such as stored procedures and non-standard SQL, no way.

              • (Score: 2) by Tork on Friday May 21, @04:03PM

                by Tork (3914) on Friday May 21, @04:03PM (#1137574)

                TMB shit on everything for so long that people were grateful when he stopped/quit.

                He 'shit' on everything by babbling just like I do. If I couldn't take his silliness I certainly wouldn't be able to deal with the AC-holes around here. (I'll leave it ambiguous for now whether or not I'm including you in that.)

                I'm not 'in love with my abuser', my panties are simply wrinkle-resistant.

                --
                Slashdolt Logic: "24 year old jokes about sharks and lasers are +5, Funny." 💩
      • (Score: 2) by Reziac on Saturday May 22, @02:29AM

        by Reziac (2489) on Saturday May 22, @02:29AM (#1137665) Homepage

        Third. And to whoever else stayed up all night getting the site back up. Made my day.

    • (Score: 1, Insightful) by Anonymous Coward on Saturday May 22, @08:49AM

      by Anonymous Coward on Saturday May 22, @08:49AM (#1137706)

      Dude came out of retirement to help put out the fire. That speaks volumes as to his character.

  • (Score: 5, Insightful) by WizardFusion on Friday May 21, @12:31AM (13 children)

    by WizardFusion (498) Subscriber Badge on Friday May 21, @12:31AM (#1137418) Journal

    Once you are well rested, it might be worth documenting how it's currently configured. :)

    • (Score: 5, Informative) by janrinok on Friday May 21, @08:49AM (12 children)

      by janrinok (52) Subscriber Badge on Friday May 21, @08:49AM (#1137490) Journal

      It is documented - but obviously not well enough. The configuration is extremely unusual but that it in part due to the software that we inherited and the fact that many of the original team have fallen by the wayside as real world priorities interfered with their lives. Once the team gets depleted it takes all the effort of those remaining to keep the site running without taking on the tasks that, at the time, seem less important.

      I think that we have all learned one lesson already - our team of volunteers is now much too small for the work involved. We need to expand in all areas in my opinion, thus freeing up those who are already overstretched to take things at a more reasonable pace whilst ensuring that there is sufficient expertise to handle critical events such as the downtime that we have just experienced.

      So for anybody in our community who thinks that they have something to offer and is willing to give it a try, please get in contact via email or IRC and come and join us. Even if it is only for a limited duration, you might still be able to make a valuable contribution to the site.

      --
      It's always my fault...
      • (Score: 0) by Anonymous Coward on Friday May 21, @09:46AM (11 children)

        by Anonymous Coward on Friday May 21, @09:46AM (#1137499)

        pipedot always looked neat and modern.

        • (Score: 2) by janrinok on Friday May 21, @10:26AM (10 children)

          by janrinok (52) Subscriber Badge on Friday May 21, @10:26AM (#1137503) Journal

          It is based on our software so I don't see much of a benefit there.

          --
          It's always my fault...
          • (Score: 0) by Anonymous Coward on Friday May 21, @11:22AM (1 child)

            by Anonymous Coward on Friday May 21, @11:22AM (#1137516)
            Are you serious. Pipedot is DEAD! Hint: the date of the top story is 2017. Unless you get lucky and get the site with the top story being from 2038. And stuff about lady gaga. Looks like they’re having “issues “, people using it as a spam pot, so it’s reset to 2017-03-14 every now and then.
            • (Score: 1, Touché) by Anonymous Coward on Friday May 21, @05:44PM

              by Anonymous Coward on Friday May 21, @05:44PM (#1137597)

              Pipedot didn't have people like Janrinok, TMB, Martyb, and Mechanicjay.

          • (Score: 0) by Anonymous Coward on Friday May 21, @11:23AM (7 children)

            by Anonymous Coward on Friday May 21, @11:23AM (#1137517)

            always got the impression it was newer base code.

            http://pipedot.org/story/2014-02-17/pipedot-status-week-1 [pipedot.org]

            but the consensus here appeared to be SN landed on better PR and reader/dev group.

            i bow to your stronger knowledge and also add my thanks for the SN teams efforts to keep the site going.

            • (Score: 0) by Anonymous Coward on Friday May 21, @11:42AM

              by Anonymous Coward on Friday May 21, @11:42AM (#1137521)
              It’s dead, Jim. Currently the top story is from April 2017. Earlier, the top story was from 2038. With a bunch of junk from yesterday. The site is in demo mode, with a script resetting it to the last valid post from when it was still active. Just before it reset, there was page after page of junk like Lady Gaga.

              The wayback machine has a last snapshot from January 2nd 2018, with the April 2017 post being the last one. So it’s dead.

            • (Score: 5, Informative) by janrinok on Friday May 21, @01:57PM (5 children)

              by janrinok (52) Subscriber Badge on Friday May 21, @01:57PM (#1137533) Journal

              As best I can recollect from over 6 years ago: Soylent and Pipedot (Bryan) cooperated together in a friendly manner to produce something to replace slashdot.beta, but following 2 different paths. The idea was that if one project should fail the other would still be around to continue. Pipedot uses a lot of new code written in PHP. Our devs upgraded Slash to the then current libs and released it as Rehash on git from where anybody is free to use it. We both went operational and for a time it seemed that we were likely to compete against each other for a community. I think we had a slight head start and Pipedot decided that it made no sense for us to be fighting for the same readership. Pipedot was put into a sleeping mode, hence the reason why the site is still live but dormant. It could be restarted if Soylent should fail.

              I am a bit hazy on the details but that is roughly how I remember it.

              See https://soylentnews.org/article.pl?sid=19/08/19/0814218 [soylentnews.org] and https://github.com/pipedot/pipecode [github.com].

              --
              It's always my fault...
              • (Score: 5, Informative) by bryan on Friday May 21, @05:09PM (4 children)

                by bryan (29) <bryan@pipedot.org> on Friday May 21, @05:09PM (#1137589) Homepage Journal

                Nice summary :) I was wondering what happened yesterday when I tried to click on an article and just got errors. Loosing a month of data sucks, but I see that not as a fault in Rehash but as the infrastructure being overly complex for the current need. Reverting MySQL to a simple and more common configuration will help greatly.

                As for my own site: still sleeping! Occasionally, some spam messages do get posted (common in all internet forums) - but the moderation system hides those once marked (unless you choose to see them in your preferences.)

                • (Score: 2) by janrinok on Friday May 21, @05:33PM

                  by janrinok (52) Subscriber Badge on Friday May 21, @05:33PM (#1137594) Journal

                  I was wondering if you were still around, Bryan! Good to 'see' you again.

                  Yeah, it is a system config problem and mechanicjay is looking at the best way to remove part of our setup completely. It was intended to give us redundancy and resilience but has proven to be the cause of most of our system downtime.

                  Rehash is working fine but it is getting increasingly difficult to find people to maintain it. We are looking at various options but the priority is the get the site stable again so we can approach any long-term changes in a calm and methodical manner.

                  --
                  It's always my fault...
                • (Score: 2) by Reziac on Saturday May 22, @02:33AM (2 children)

                  by Reziac (2489) on Saturday May 22, @02:33AM (#1137667) Homepage

                  Hey there! sleeping is okay; I'd be very disappointed if it were dead and gone. It seems to work well enough, and the day may come when it's needed awake again.

                  • (Score: 3, Funny) by kazzie on Sunday May 23, @08:02AM (1 child)

                    by kazzie (5309) Subscriber Badge on Sunday May 23, @08:02AM (#1137914)

                    Now I'm getting crossover between Pipedot and the legend of King Arthur...

  • (Score: 5, Funny) by Anonymous Coward on Friday May 21, @12:34AM (4 children)

    by Anonymous Coward on Friday May 21, @12:34AM (#1137419)

    Or just systemd?

    • (Score: 5, Funny) by Acabatag on Friday May 21, @01:25AM (3 children)

      by Acabatag (2885) on Friday May 21, @01:25AM (#1137443)

      Explain the difference.

      • (Score: 5, Funny) by maxwell demon on Friday May 21, @05:53AM (2 children)

        by maxwell demon (1608) on Friday May 21, @05:53AM (#1137459) Journal

        Ransomware has a built-in way to undo the damage after you've paid the ransom.

        --
        The Tao of math: The numbers you can count are not the real numbers.
        • (Score: 3, Touché) by kazzie on Friday May 21, @06:39AM (1 child)

          by kazzie (5309) Subscriber Badge on Friday May 21, @06:39AM (#1137473)

          Correction: Ransomware *tells you* that is has a built-in way to undo the damage.

  • (Score: 1, Funny) by Anonymous Coward on Friday May 21, @12:36AM (3 children)

    by Anonymous Coward on Friday May 21, @12:36AM (#1137420)

    I demand all my karmas back.

    • (Score: 0) by Anonymous Coward on Friday May 21, @12:52AM (2 children)

      by Anonymous Coward on Friday May 21, @12:52AM (#1137425)

      I have 150 karma for some reason. You can have some of those.

      • (Score: 1) by Sulla on Friday May 21, @12:54AM (1 child)

        by Sulla (5173) Subscriber Badge on Friday May 21, @12:54AM (#1137427) Journal

        Since when has AC had positive karma?

        --
        Ceterum censeo Sinae esse delendam
        • (Score: 0) by Anonymous Coward on Friday May 21, @01:02AM

          by Anonymous Coward on Friday May 21, @01:02AM (#1137433)

          What? You think all us ACs have negative karma?

  • (Score: 5, Insightful) by srobert on Friday May 21, @12:49AM (4 children)

    by srobert (4803) on Friday May 21, @12:49AM (#1137423)

    Thank You mechanicjay and TMB. And also to you martyb, and everyone else who keeps this site running.

    • (Score: 3, Funny) by fustakrakich on Friday May 21, @01:01AM (2 children)

      by fustakrakich (6150) on Friday May 21, @01:01AM (#1137432) Journal

      Site goes down, who do ya call? Ghost Buzzard! [soylentnews.org]

      --
      Ok, we paid the ransom. Do I get my dog back? REDЯUM
      • (Score: 1) by Sulla on Friday May 21, @05:49PM (1 child)

        by Sulla (5173) Subscriber Badge on Friday May 21, @05:49PM (#1137598) Journal

        You seem to forget.

        That which has no record of death may eternal lurk, and in strange eons, even buzzards may fly.

        --
        Ceterum censeo Sinae esse delendam
    • (Score: 0) by Anonymous Coward on Friday May 21, @01:15PM

      by Anonymous Coward on Friday May 21, @01:15PM (#1137528)

      A big thank you from a regular anonymous reader/lurker! :)
      I read this site just about every day, and quite enjoy it. I appreciate the work of all the folks that keep this site up and running.
      So .... Thank You!!

  • (Score: 1, Funny) by Anonymous Coward on Friday May 21, @12:50AM (7 children)

    by Anonymous Coward on Friday May 21, @12:50AM (#1137424)

    is because they couldn't afford the malware ransom. The thieves kept lowering the ransom until they got to $100, but the SN admins said SN is not a goldmine.

    • (Score: 5, Funny) by Anonymous Coward on Friday May 21, @12:53AM (1 child)

      by Anonymous Coward on Friday May 21, @12:53AM (#1137426)

      I told them to install Russian and Ukrainian locales.

      • (Score: 2) by The Vocal Minority on Saturday May 22, @03:48PM

        by The Vocal Minority (2765) on Saturday May 22, @03:48PM (#1137756) Journal

        Ah ha I think we have found the culprits! Russian Hackers obviously hacked SN to remove that story, cunning bastards!

        Seriously though thanks Mechanicjay and TMB.

    • (Score: 5, Funny) by janrinok on Friday May 21, @08:53AM (3 children)

      by janrinok (52) Subscriber Badge on Friday May 21, @08:53AM (#1137491) Journal

      THIS IS NOT TRUE!

      They were offering us hard cash to close the site down ourselves - but we are too thick-headed to accept the offer! So, like a phoenix, it has risen from the software ashes until the next time it falls off its perch.

      --
      It's always my fault...
      • (Score: 2, Funny) by Anonymous Coward on Friday May 21, @12:13PM (2 children)

        by Anonymous Coward on Friday May 21, @12:13PM (#1137524)

        Nails, my friend. Nails will keep it on its perch.

        • (Score: 0) by Anonymous Coward on Friday May 21, @04:41PM

          by Anonymous Coward on Friday May 21, @04:41PM (#1137585)

          > Nails

          ??
          Are you sure you don't mean talons? (grin)

        • (Score: 1) by Sulla on Friday May 21, @05:51PM

          by Sulla (5173) Subscriber Badge on Friday May 21, @05:51PM (#1137599) Journal

          Well you have to do that, otherwise the site would be pushing up the daisies.

          --
          Ceterum censeo Sinae esse delendam
    • (Score: 0) by Anonymous Coward on Friday May 21, @10:52AM

      by Anonymous Coward on Friday May 21, @10:52AM (#1137509)

      Soylentnews paid the $100 but the restoration attempt didn't work. Soylentnews should have first demanded that the hackers decrypt and restore some random posts of Soylentnew's choice to prove that they can actually restore the rest of it before forking over the money.

(1) 2