As many of you noticed, we had a site crash today. From around 1300 until 2200 UTC (2021-05-20).
A HUGE thank you goes to mechanicjay who spent the whole time trying to get our ndb (cluster) working again. It's an uncommon configuration, which made recovery especially challenging... there's just not a lot of documentation about it on the web.
I reached out and got hold of The Mighty Buzzard on the phone. Then put him in touch with mechanicjay who got us back up and running using backups.
Unfortunately, we had to go way back until April 14 to get a working backup. (I don't know all the details, but it appears something went sideways on neon).
We're all wiped out right now. When we have rested and had a chance to discuss things, we'll post an update.
In the meantime, please join me in thanking mechanicjay and TMB for all they did to get us up and running again!
Related Stories
This is a follow-up to our site crash.
If you are just tuning in, SoylentNews experienced a database crash on 2021-05-20. We tried to restore from recent backups, but found they were corrupted and unusable. Thanks to heroic work by mechanicjay the site is back up and running!
Many thanks to mechanicjay for his 16-hour(!) day on Thursday to get us back up and then move us to a single back-end configuration. He didn't stop there, but has continued on gathering information and guiding work to get us to a more stable foundation. We'll keep you posted as to our progress.
Read on if interested... otherwise, another story will be along presently.
(Score: 3, Interesting) by NateMich on Friday May 21 2021, @12:28AM (27 children)
If I had to guess, I'd say that whatever broke your databases was also messing with your backups recently.
I mean, unless you really only back up monthly or something.
(Score: 5, Insightful) by Anonymous Coward on Friday May 21 2021, @12:46AM (21 children)
Hard disk space has been an issue in the past. That's something that could break both a live database and backups.
(Score: 5, Informative) by mechanicjay on Friday May 21 2021, @01:05AM (18 children)
My VMS box beat up your Windows box.
(Score: 1, Interesting) by Anonymous Coward on Friday May 21 2021, @01:28AM (17 children)
How the hell can hard disk space be an issue on a low volume web site on even a basic hosting plan? Seriously, are you keeping logs from 5 years ago? And everything else. If you “must” keep that crap, for sentimental reasons, keep it off-site. It will make backup and restore much easier.
Time to face the music and get something that’s more widely used and mostly just works out of the box. Remember groklaw? That ran geeklog, a proper CMS.
So you won’t have user moderation. Big deal - it’s been games for decades. We criticized Amazon for trying to family their warehouses to get more out of workers, ditto Microsoft when they experimented with it. We criticize social media having degenerated in a game of who can get the most likes and followers. Slashcode is just as bad in this respect. People have been abusing the mod system since it was first introduced. Time to grow up, ditch the crap.
Because it’s not just in Putin’s Russia that Slashcode shits on YOU!
(Score: 0) by Anonymous Coward on Friday May 21 2021, @01:37AM
I'm thinking probably a few trillion ASCII Goatse posts.
(Score: 5, Insightful) by RedIsNotGreen on Friday May 21 2021, @06:50AM (13 children)
User moderation is what made Slashdot what it was, and is essential to keeping this site what it is, IMO. I think you are wrong to write it off so casually.
(Score: 4, Informative) by istartedi on Friday May 21 2021, @07:10AM (12 children)
There seems to be a move to do that here lately, and I don't get it. It's like the last vestige of the Internet before it became TV. I'm not generally a conspiratorial type; but it seems like TPTB want to kill Slash and anything like it so that we've got nothing to do but play the anger games you see on other sites.
Really though, what does that have to do with backups? Aren't most backup solutions going to be agnostic of how the site operates?
Appended to the end of comments you post. Max: 120 chars.
(Score: 0, Interesting) by Anonymous Coward on Friday May 21 2021, @10:45AM (1 child)
Slash is dead. The repeated crashes show you can’t even claim it’s in maintenance mode.
User moderation is a joke. Too many people trying to game the system. It’s one of the reasons people maintain sock puppets. User moderation is no better than Facebook likes. It’s just noise at this point. An experiment that failed.
And now all the posts regarding NCommanders observations in his resignation post, and all the questions it raised about how it was deleted from the story queue after the submission was accepted, are conveniently gone.
“Because the backups over the last month are corrupted.”
Unfortunately believable given the lack of any real technical professionalism on a “tech” site.
(Score: 2) by Mykl on Friday May 21 2021, @11:17PM
But did they fake the Moon landing or not?
(Score: 0) by Anonymous Coward on Friday May 21 2021, @12:10PM (9 children)
I think there's a big split in the community - the ones who always log in, disparage ACs, and love moderation, and those who don't (the AC posters and those who log in just to set the site to -1 for easy reading).
(Score: 1, Interesting) by Anonymous Coward on Friday May 21 2021, @02:02PM (8 children)
Wanna bet that the AC posters include people who used to log in but are fed up? I know that was my motivation - I gave up on this site’s libtard “post any hate shit you want” policy. I wanted this site to succeed, but now it’s become a cautionary tale more than anything else.
The emphasis on moderation over everything else (including the articles and journals) is a destructive distraction. You want to let someone know what you think, get your lazy fingers moving and post a reply. Moderation is about as ineffective as signing an online petition - its literally the least you can do except doing nothing at all. Slactivism. A motivation for running sock puppets.
It also discourages logged in users from posting unpopular truths because they’ll be mod-bombed into silence for challenging the groupthink. There was a time when anyone who posted about how RMS hasn’t done anything useful in decades would be modded to oblivion. Now? Not so much.
Same thing with the long-running mantra that open source HAD to be more secure. Now we know that all software is insecure and buggy by nature, irrespective of whether it’s open or closed, because writing great code is HARD.
But useless user moderation is still a sacred cow for too many. “It’s what made slashdot great”. That is a LIE. What made it great was the tech boom, being in the right place at the right time. It still has user moderation , so why is it not still great? Because user moderation didn’t save it.
Ask yourself why this site is in decline during a pandemic, when you should have tons of submissions and so many talented volunteers that you’d have to beat them off with a stick.
Nobody with serious skills wants to work with someone like TMB. Or enable hate speech like EF. The first tentative steps towards addressing this are too little too late. I don’t want colleagues, friends, or family to associate me with such a clusterfuck. It’s not as bad as 4chan, but that’s just damning with the faintest of praise.
(Score: 0) by Anonymous Coward on Friday May 21 2021, @03:05PM
when it was posted politely, u disagreed
when it was posted in frustration, u called it flame
when it is now posted withou any makup, u now call it hatespeech...
-zug
(Score: 3, Insightful) by istartedi on Friday May 21 2021, @04:12PM (2 children)
How are you defining "great"? Are you defining it as having a larger user base? If so, that comes with a lot of problems that would, IMHO, not make a Slash site great.
These sites have limited appeal *and that's good*.
Are you defining great in terms of the quality of the posts?
I think we're on a spectrum there. There are other tech sites from which I learn more, but there's also a lot of very popular social media that's garbage.
We're somewhere in the middle. There's nothing wrong with that. The Internet has plenty of room for Lambda The Ultimate (which is coincidentally also down at the moment), Soylent, and reddit. They serve different communities.
As for hate speech, you can't call Soylent a failure there. Hate speech always comes along for the ride, and as long as it's not the generally endorsed view of the community it doesn't harm us. If you think that's the situation we're in, you haven't really looked at too many other sites.
Is there some group-think here? Sure, but it's nothing like reddit or a lot of other places. If it were, I wouldn't be here.
It's almost like you're looking at a completely different site. Cautionary tale??? I just don't see that.
We're small, people speak their mind here, and not everything is agreeable to you. That's not failure.
Appended to the end of comments you post. Max: 120 chars.
(Score: 1, Insightful) by Anonymous Coward on Friday May 21 2021, @04:20PM (1 child)
"Hate speech" is a bullshit term with no meaning, or rather, it means whatever the person who uses it wants to mean, which is simply: "Shut up!"
Our language is constantly being degraded by invented words with no meaning and also removing meaning from old words.
(Score: 3, Insightful) by PiMuNu on Friday May 21 2021, @04:24PM
Get thee to a nunnery
(Score: 0) by Anonymous Coward on Friday May 21 2021, @05:22PM
I would not feel safe in posting opinion under a constant identifier on today's internet. It won't help you, but it can come to hurt you.
(Score: 1, Interesting) by Anonymous Coward on Friday May 21 2021, @05:30PM (1 child)
Total Pilpul comment, or bot comment.
This isn't the '90's anymore. Free speech is now a conservative policy, not a "libtard" one. If you don't believe me, look at Reddit and Facebook.
I remember that discussion, and compared to the others, moderation was totally balanced there, it being a charged topic. If you're gonna complain about mod imbalances and being modded into oblivion, then try criticizing cancel culture, woke bullshit, unnecessary lockdowns/vaccinations, and BLM riots. Then, you will truly see the downmod sockpuppetry at work.
Linux: All the corporate ownership of Windows, with none of the functionality. "Stupid Windows users." "Yeah I know, Microsoft is a corporation and won't let you see the code. But with Linux there are many eyes on the code so it's secure." It isn't the year 2000 anymore, Linux is completely corporate and spook-owned. Look at its "sponsors" if you don't believe me.
Hey stupid, just because you went woke doesn't mean that everybody else in the tech industry did. Anti-wokers are the new "niggers" of the internet. When the woke fad starts to wind down in a year or three because it's alienated everybody except you, I doub't you'll take away any lesson from it.
Huh, I seem to remember TMB saving your asses, so apparently some people DO want to work with TMB -- I'd hire him over anybody with pronouns and a BLM stamp in their resume, because I know he's gonna get shit done rather than find faults with everybody else. A lot of EF's comments are crap but he's made plenty of good ones, many of which brought the crybabies out of the woodwork to whine because he bruised their fragile egos. It's like being the jock in a room full of fat White kids. I think a lot of people just don't want to admit that they were wrong about woke bullshit as their narratives crumble all around them.
(Score: 0) by Anonymous Coward on Saturday May 22 2021, @09:32PM
Hahahaha
That explains why all the conservative moderated platforms ban people for ideological violations. As usual you nutbars have it backwards but believe the lies you're told.
(Score: 2) by Joe Desertrat on Thursday June 03 2021, @08:28PM
Go with "unpopular opinions" instead of "unpopular truths" and maybe you might have a complaint. Most of the "mod-bombed" comments, if they are posted in serious discussions, are down modded because they have no real basis in fact, they are usually just rehashes of various already discredited ideas. If something gets down modded that probably shouldn't have been, it usually gets modded back to at least its starting point by those that disagree.
(Score: 5, Interesting) by bzipitidoo on Friday May 21 2021, @04:52PM
You ever done any sysadmin work? I have. Data safety takes more space than you seem to realize. There is never enough space for all the things you ought to do to keep the data safe. You have to make compromises. RAIDs protect you from hard drive failures, but little else. They aren't backups, no matter how much some people feel they are. You need to keep daily backups, real ones, actual complete copies of the data, on separate machines, preferably in separate facilities, because there are disasters that can wipe out or disable an entire server farm.
No, you DON'T throw away yesterday's backup the moment you have finished creating today's backup. You keep it for a while. What you want to do is keep 7 days worth of backups, and as each backup passes the age of 7 days, delete them, keeping only one to be your weekly backup. Keep 4 weekly backups, use one to be the monthly backup, and delete the other 3 after a month. That way, when someone finds a mistake, finds out that they accidentally deleted the wrong file, or edited out some crucial verbiage, perhaps 3 weeks later, you can go back and get a copy from before the mistake was made. At any moment, you're going to have at least a dozen or two dozen backups on hand. That takes a lot of storage space. Lot of companies, particularly small companies, can't do it, don't have the expertise or the equipment or money.
What do you do when you run out of space? I was at a company when that happened, and the person who ran out of space took matters into his own hands. He deleted one of the backups, and TURNED OFF THE BACKUP PROCESSES. He had the authority to do that. I wasn't notified, nor did I receive any warnings frmo the monitoring software that alerts people when something is wrong with the backups, because he also turned that off. Didn't tell anyone, and he should have. That was the state of affairs for a week. Then it happened. Disaster. Another developer screwed up and accidentally erased our entire production database. How'd that happen, you might wonder? The developers had demanded, for their convenience and against the database admin's strenuous protests, no password protection on the database. The admin wanted to at least have that, but he was overruled.
So this developer thought he was erasing and rebuilding the test database, and didn't realize until too late that he had his little script pointed at production. DROP TABLE on all the tables. Asking for a password would have stopped it. I was on a call with the database admin when it happened. We turned to check some little something on the website, only to discover it was gone! Was working fine just a moment before. Then there was a mad scramble to find out what the H was going on. Our first thought was that we'd somehow been hacked. But it soon became clear that if it was a hack job, it was very weird in that it left all the servers intact and running, all the login credentials unchanged. I found no evidence of intrusion anywhere. Soon the guilty developer confessed. Then it was discovered that the backups had been turned off. Our newest backup was a week old. The only thing that allowed the database admin to recover was that all the database activity had been logged on yet another machine. Took him 3 weeks to run the logs to catch up the week old backup, but he did it.
(Score: 0) by Anonymous Coward on Sunday May 23 2021, @01:35AM
Probably all the space used to warehouse Ari's BS.
(Score: 1, Touché) by Anonymous Coward on Friday May 21 2021, @12:08PM
Only if you keep the database and the backups on the same storage. In which case, they're not "backups", merely "data snapshots".
(Score: 2) by mcgrew on Saturday May 22 2021, @04:08PM
Hard disk space for a text-only site? I have a 4 terabyte drive I paid $200 for a couple of years ago, saw a 2 tb thumb drive advertised for fifty bucks recently. ALL the Star Trek series and movies are on my $200 drive, all the Star Wars movies (and hundreds of other full length movies), a dozen multi-year TV series, all the rest of my files (music, photos, books) and it's still nearly empty. Five of them would hold the entire Library of Congress.
Thanks to those who worked to fix it. I just looked, the last of my journals listed was from January. Hope the missing ones come back.
mcgrewbooks.com mcgrew.info nooze.org
(Score: 5, Funny) by Rosco P. Coltrane on Friday May 21 2021, @05:49AM (4 children)
I'll tell you what sucks: had I known this would happen, I could have trolled the living daylights out of this here forum for over a month and gotten my slate wiped clean today :)
(Score: 0) by Anonymous Coward on Friday May 21 2021, @02:09PM (3 children)
Nobody’s going to want to work with someone who is going to bitch about having to use any sort of structure in their workplace processes like version control. Or a bug tracker.
(Score: 4, Insightful) by Rosco P. Coltrane on Friday May 21 2021, @03:20PM
How much do you pay for the quality of service you seem to expect, that you sarcastically point to the lack of?
Yeah, that's what I thought.
(Score: 2) by bzipitidoo on Sunday May 23 2021, @02:25AM (1 child)
Subversion?? WTF? Do you know what you're talking about? That, and those other 2 tools you mention are software development tools, not system administration tools. A software repository is NOT a backup! Subversion isn't even the best, git is better. And git isn't even that new any more, been around for 15 years. git is distributed, cloning the repository in such a way that individual copies can act as backups. Not Subversion. Subversion is much more centralized. Lose that master repository, and you're going to have a rough time reconstructing it unless there is a backup.
Backups are done with tools such as rsync. For monitoring, we used Nagios. Scheduling can be done with cron. Many other things were done with little scripts.
(Score: 0) by Anonymous Coward on Sunday May 23 2021, @06:15AM
That just a troll trolling, better off not to feed it.
A completely unrelated point SVN == cathedral && git == bazaar
(Score: 3, Funny) by Anonymous Coward on Friday May 21 2021, @12:31AM (35 children)
Especially TMB - throwing moves as most reluctant disco dancer ever.
(Score: 5, Insightful) by Tork on Friday May 21 2021, @01:09AM (33 children)
🏳️🌈 Proud Ally 🏳️🌈
(Score: -1, Flamebait) by Anonymous Coward on Friday May 21 2021, @06:03AM (31 children)
It was his crap setup that probably caused this in the first place. AND I predicted it would probably happen in advance the next time Linode did network maintenance. But instead of listening and learning to the guy who has run high availability clusters professionally, he got all high and mighty about how I was misinformed. I just hope that whomever is in charge this time actually reads the manual and doesn't make basic mistakes like he did.
(Score: 0) by Anonymous Coward on Friday May 21 2021, @06:10AM (20 children)
*yawn*
Have you ever volunteered? No? OK then, we can safely ignore your prescience that can't be proved. Like every other medium who predicts catastrophies, you're probably just another charlatan.
(Score: 0) by Anonymous Coward on Friday May 21 2021, @06:27AM (19 children)
Except it was documented on this site and I'm not the first nor only one to notice the problem pattern with their setup. And why would I have volunteered to butt heads with someone who apparently hasn't read the docs and doesn't understand the apparent basics of HA clusters like this.
(Score: 3, Touché) by janrinok on Friday May 21 2021, @08:36AM (18 children)
I'm not about to go searching through every AC comment to find where you claim it is "documented".
And it appears that you have already reached the conclusion of what caused the problem and who is responsible. If it was a full hard drive then, I believe, somebody on the current team should have been responsible for checking for that possible occurrence and taking the appropriate action before we get to a critical stage. And in such a case that person would be responsible - not TMB who is no longer managing our hardware and therefore cannot be responsible for our day-to-day running. And when the site was first set up there were quite a few people responsible for structuring the hardware to meet the requirements of the code that we inherited. TMB was only one of those involved.
True, we can restructure the entire hardware and software configuration but that takes qualified people and there are precious few of them volunteering to give up any of their time to help this site. Fortunately we had 2 people - mechanicjay and TMB - who have given willingly and freely of their personal time over the last day or so. If you are volunteering to help, then you must understand that we cannot simply accept the word of an AC - you will have to be a little bit more approachable than that. You will be welcome in that event but, if not, then you can continue to rant on here as an AC and we will continue to take everything you say with a healthy pinch of salt.
The site will crash again - of course, I don't know when or why. Making such a claim does not qualify me to operate as a sysadmin on this site, however by making this forecast I have now got exactly the same provable credentials as yourself.
(Score: -1, Flamebait) by Anonymous Coward on Friday May 21 2021, @09:44AM (17 children)
Keep making excuses. You don’t need to search for any “documentation “ - the site crashes on a regular basis, and everyone knows it.
And as a parent poster pointed out, nobody is going to volunteer to fix it if it means continually arguing with Mr “Proud I Don’t Need An Education” Buzztard.
But stick with the current “plan”. Where you’ve built up so much technical debt that recovery is impossible. Because there’s no fool like an old fool.
(Score: 4, Informative) by janrinok on Friday May 21 2021, @10:21AM (12 children)
You've read something that I didn't say.
We ARE looking at how to improve both the system configuration and the software that we use. We are not, however, simply throwing everything away to start from scratch again. The system can be simplified which should result in a more robust site. As mechanicjay has stated elsewhere, some of the software that is installed to provide resilience is actually causing more problems than it is intended to solve. We can get rid of that straight away. A content management system should be able to work with any chosen database, and that includes MySQL, so there is nothing that I am aware of to suggest that MySQL cannot fulfil the role we are asking of it. People may have their own personal preferences but changing the database will require changes to the perl code which will all need writing and testing.
There is a problem with documentation but that is also linked to the lack of staff that we currently have, and that goes back at least 3 years. What few staff we have are currently kept busy keeping the site going and, although we are aware of areas where work needs to be done it can only be done by those who understand the system configuration. You can only write more documentation when you have people who understand what it is they are documenting. We need more sysadmins because system failures do not occur when the only active sysadmin is sat at his computer with nothing better to do.
We need more programmers who are prepared to volunteer to help support the site. It doesn't matter which language the site is written in, we will still need programmers to do that work. We always need more editors - although even with just the handful that we currently have we a looking relatively well manned compared with every other part of the support team. QA is a one-man team - MartyB again, who actually fills several more roles in the team at the same time.
You definitely have got a bug about TMB though - in case you missed it he is no longer part of the support team although he remains a member of our community and he kindly gave advice to mechanicjay during the last 24 hours. If you have had your nose put out of joint during your earlier discussions with him then that is a personal matter between you two.
So you can see that, far from your claim, we are not sticking with the current plan. With the limited resources that we have we will make progress at as fast a rate as is possible. When the site first went active there were 20-30 active participants who were all contributing to keeping the site going. I reckon that we have less than 10 available today. Rather than sitting back and criticising as an AC, wouldn't you prefer to join the team and help fix some of the problems?
(Score: -1, Redundant) by Anonymous Coward on Friday May 21 2021, @11:05AM (11 children)
But keep telling yourself that slash can be fixed. You need proper devs, something you haven’t had in years, who wouldn’t put up with wishful thinking but will speak the hard truths borne of the confidence of experience.
There are other CMS that will also do a decent job. But NONE OF THEM ARE WRITTEN IN PERL. So the language issue is entirely relevant. Because software has to be maintained. And nobody wants to use Perl for large projects any more. Not when there are better alternatives.
If your code is so great why isn’t anyone else running it? Because it’s an in maintained pile of patches over patches.
The month of corrupted backups is just a symptom, another red flag. But keep making excuses - reality will continue to bite, and bite increasingly harder. There are hard decisions to be made, and you either make them now or events will make them for you. The future waits for nobody.
(Score: 2) by janrinok on Friday May 21 2021, @11:37AM (6 children)
Rehash was not the cause of the latest crash (as far as we can ascertain) - it is not where the focus is at present. That is not to say it will never be replaced but, for the time being, it is still working as expected. Currently, we have not got the resources to replace Rehash with a different language or package. If it ain't broke, don't fix it.
You are focussing on an area that is not causing us a problem at the moment. The system configuration is where we continue to encounter problems and that is where mechanicjay is currently concentrating his efforts.
(Score: 0) by Anonymous Coward on Friday May 21 2021, @01:58PM (5 children)
More often than not, this actually means "the fix is too hard, so it cannot possibly be broken".
(Score: 2) by janrinok on Friday May 21 2021, @02:08PM (4 children)
Rehash is working today as advertised, the system configuration isn't - with very limited resources which one would you work on first?
(Score: 0) by Anonymous Coward on Friday May 21 2021, @02:34PM (3 children)
Another post mentioned using a separate process to update story counts. Someone needs to learn to code better, and brushing up on sql as well. Just because the original devs didn’t know how to do it right is no excuse to preserve shit like that. This is 2021, not 1995.
TMB fücked up by not using a LIMIT clause in SQL that would have avoided time-outs under load. Experienced devs will ALWAYS seek ways that guarantee the most efficient use of resources because they don’t want intermittent bugs. Rehash is a total hash. Either learn to code or get something that other people are maintaining because it’s widely used, in a language that is widely used for web development. But you won’t. You will continue to ignore the red flags.
Why the resistance to a clean-sheet rethink of the site? Articles, user comments , and user journals are the only essentials. The polls suck, but most CMS packages contain poll functionality, so keep pills if you must. But do you really want to waste part of your life dealing with stupid complaints about unfair moderation? What a time sink! Dump it. It’s far from essential, and keeping it didn’t preserve slashdot’s ability to generate the slashdot effect.
If you think that user moderation is the killer feature that keeps people on the site, well, it ain’t working here, same as it didn’t on the green site. Is it SO hard to grab a copy of geeklog and skin it so it looks the way you want while still allowing the essentials - stories, comments, and journals? It’s a one-day job (with breaks).
What do you have to lose at this point?
(Score: 3, Insightful) by janrinok on Friday May 21 2021, @04:39PM (1 child)
We are, at this very moment, discussing options on a private channel. And currently ALL of our resources are currently working on recovering from yesterday, or keeping the site going today.
The only thing that is causing a problem (repeatedly) is one element of the system configuration that is not providing us with any benefit whatsoever - so that is what we are currently working on removing. The rest of the site is working just as we want it to. Let me explain it in an auto analogy - which is the traditional way of doing things around here. What you are suggesting is that we currently have a flat tire but you are recommending that we also paint the car, change the upholstery and fit a new engine too.
If it can be done in a day I will await your contribution by, shall we say, Sunday evening? Show me something working to convince me - not just make ridiculous suggestions that we haven't got the resources to complete anyway.
(Score: 0) by Anonymous Coward on Friday May 21 2021, @10:46PM
Perhaps that is for the best since you all apparently don't know how to use it properly. Quite a number of people use it under higher loads with better uptimes, after all.
(Score: 0) by Anonymous Coward on Saturday May 22 2021, @05:03PM
Ah yes, but only when it comes to inferior brains
(Score: 5, Insightful) by martyb on Friday May 21 2021, @02:30PM (3 children)
One thing to keep in mind is the "heritage" of our code.
I was with /. before it even had userids! I've witnessed all kinds of attacks on the site. Page-widening trolls. Actual SPAM comments. Mod bombs. Whatever creative nerds could come up with, they threw it at /. and changes were made to mitigate them. It stood up under heavy fire.
Slashcode begat rehash which is the open-source, freely available code that powers this site. Our foundation is solid.
Also, there is MUCH MUCH more going on behind the scenes. I dare say the admin interface has AT LEAST as much going on as what is presented to the community. Quite possibly twice (or thrice) as much. Every once in a while I find yet-another setting or configuration that could be tweaked!
The foundation is solid.
Admittedly, the site would benefit from some tuning. When SoylentNews started, it was difficult to foresee what areas would grow fastest and what needed to be allocated. I mean, here is my first comment on the site: comment 255 [soylentnews.org], and here I am replying to comment number 1,137,513!
Remember, too, this site is run by volunteers in their spare time.
Sure, it would be wonderful to have paid, full-time staff monitoring the site 24/7/365 like on Reddit or the like. How much would that cost per year? 3 x 8-hour shifts per day x 365 days-per-year is 8760 hours. At $15.00 per hour (dirt cheap for these kinds of skills!) that works out to $131,400 per year! And that does not even include server hosting costs! More realistically, at just $30.00 per hour, that works out to $1,839,600 per year! And that does not even include hosing costs.
SoylentNews gets by on just $7,000 for an entire year!. And that includes the annual costs of being incorporated, filing taxes,hosting expense, everything!
.
Wit is intellect, dancing.
(Score: 4, Informative) by martyb on Friday May 21 2021, @02:38PM
Oops! His submit instead of preview.
s/hosing/hosting/
s/131,400/919,800/
There's prolly some more; it was a LONG day yesterday!
Wit is intellect, dancing.
(Score: 0) by Anonymous Coward on Friday May 21 2021, @03:07PM (1 child)
Yes, it’s expensive keeping a full-time dev on the payroll. That’s why hobby sites like soylent don’t do that - they use widely used open source CMS packages that have proper documentation, a developer community, and use a broadly used language combo - the most popular being written in php and any MySQL or PostgreSQL variant.
Not slash. Not rehash. Perl lost the race a long time ago.
What do users want? Articles, the ability to post comments, and journals. Plenty of CMS packages using php and a database server can do that without the legacy of Perl.
Grab a copy of geeklog and play around with it. You should be able to have a functional site with stories, comments , and journals. And of c, the administrative backend contains all the functionality you want to hide from users.
This is what you use when you can’t afford to keep a team of developers on staff. Php has a wide user base, so you might actually attract developers, because nobody wants to screw around with Perl. The whole “TMTOWTDI” is a bug, not a feature.
You might even want to give the site a new, fresher look.
Seriously, give it a try. Take a shitbox computer, install Linux or FreeBSD on it, and give geeklog a try. It worked for groklaw under traffic you can only dream of. Don’t be fooled by groklaw’s blah appearance. If you know HTML and CSS, and have any graphics talent , you can make it look clean and modern and spiffy as all. Icons for stories? Screw that - real images or graphics that the text wraps around. (You can still keep the topic icons if you must, but they’re really dated).
As for the whole editorial process, you’d best run a private copy for the editors to edit submissions before someone posts them to the main site. I get that the subs queue is there so people can check before submitting a story, but multiple submissions are a good thing if they contain more information. You’ll probably end up dropping ICQ if editors can see what their proposed stories and edits and included graphics look like, and other editors can cut n paste and change and tweak it, and see the changes right there in the thread in their comments.
(Score: 1, Insightful) by Anonymous Coward on Saturday May 22 2021, @11:04AM
Those php sites get pwned every now and then too.
Geeklog's security track record is crap and the types of vulnerabilities are not confidence inspiring: https://www.google.com/search?q=%22Geeklog%22+exploit [google.com]
Go get a clue. If you're not going to spend much time and money on a site you don't pick shit that needs to be patched every month.
(Score: 1) by khallow on Saturday May 22 2021, @11:35PM (3 children)
Even when TMB was in house, nobody was continually arguing with Mr. "Proud". I wonder how much else of your narrative is just as imaginary?
(Score: 0) by Anonymous Coward on Sunday May 23 2021, @06:09AM (2 children)
And then you wonder why a system that has a five nines guarantee in a 2/2/2 setup doesn't even have two.
(Score: 1) by khallow on Sunday May 23 2021, @01:14PM (1 child)
I already know of real world systems - the Space Shuttle, that failed that hard. There's no wondering over here.
(Score: 0) by Anonymous Coward on Monday May 24 2021, @12:37AM
Because operating on the edge of science and technology at the extremes of risk with single points of failure meeting Swiss Cheese model of reality is directly analogous to running an incorrectly deployed bog-standard cluster deployment that is failing to meet its uptime guarantees despite hundreds of thousands of deployments operating successfully in worse conditions when they do deploy it correctly.
Right.
(Score: 5, Insightful) by Tork on Friday May 21 2021, @06:15AM (9 children)
Whether he is blameless or not the fact is he quit. I've butted heads with him before, so it's not like I'm being charitable when I say: I can't think of a reason to fault him for lending a hand.
🏳️🌈 Proud Ally 🏳️🌈
(Score: -1, Troll) by Anonymous Coward on Friday May 21 2021, @06:20AM (8 children)
I'm not faulting him for lending a hand, it's everything else that got this site to this point.
(Score: 0, Disagree) by Anonymous Coward on Friday May 21 2021, @11:28AM (7 children)
Money talks.
(Score: 0) by Anonymous Coward on Friday May 21 2021, @02:17PM
What utter nonsense. This site is run by volunteers and the staff (including TMB) have been excellent.
(Score: 2) by janrinok on Friday May 21 2021, @02:21PM (4 children)
MartyB is responsible for recording donations, and he has been rather busy for the last year or two - not TMB. Anything you else you want to accuse TMB of while you are at it?
(Score: 1, Interesting) by Anonymous Coward on Friday May 21 2021, @03:19PM (3 children)
Plenty of people saw buzz as a large part of the problem. After a week of “playing nice” he packed it in.
If you had even ONE serious developer on tap you could have a different CMS up and running in a day, stories, comments, user journals, stupid polls, etc. ONE DAY. Take a few weeks to get user feedback, tweak things a bit, etc.
But NOBODY with a clue wants to maintain a mess of obsolete Perl that has no documentation. It’s one of those cases where you have to close your eyes and ignore the sink costs of the time and emotions tied up in rehash.
Software development is brutal. You can’t be sentimental over old code. You need to “knife the baby” on a regular basis to progress. Otherwise we’d still be stuck with GWBASIC.
(Score: 3, Informative) by janrinok on Friday May 21 2021, @04:49PM
The Perl code is working as expected. This is a system configuration problem. Stop trying to fix the wrong problem.
(Score: 5, Informative) by DECbot on Saturday May 22 2021, @03:06AM
To add to what Janrinok has said, slashcode regularly handled a traffic load that would take offline mainstream static sites and ddos ISPs with less resources then what people throw at a regular Wordpress or Drupal sites today. The perl code is proven, mysql is proven, but there is a configuration issue causing stability issues with the SN implementation of the slashcode. That is what needs to be addressed. It's not time to replace the horse cause it threw an ill fitting shoe.
cats~$ sudo chown -R us /home/base
(Score: 2) by bzipitidoo on Sunday May 23 2021, @03:08AM
One day, for one person to do a migration to another CMS? You're dreaming. I am not familiar with CMSes, but if it's anything like a database migration, you're talking weeks at the least. Yeah, one expert could migrate a little toy of a database in one day, but a real one with millions of rows and complexities such as stored procedures and non-standard SQL, no way.
(Score: 2) by Tork on Friday May 21 2021, @04:03PM
He 'shit' on everything by babbling just like I do. If I couldn't take his silliness I certainly wouldn't be able to deal with the AC-holes around here. (I'll leave it ambiguous for now whether or not I'm including you in that.)
I'm not 'in love with my abuser', my panties are simply wrinkle-resistant.
🏳️🌈 Proud Ally 🏳️🌈
(Score: 2) by Reziac on Saturday May 22 2021, @02:29AM
Third. And to whoever else stayed up all night getting the site back up. Made my day.
And there is no Alkibiades to come back and save us from ourselves.
(Score: 1, Insightful) by Anonymous Coward on Saturday May 22 2021, @08:49AM
Dude came out of retirement to help put out the fire. That speaks volumes as to his character.
(Score: 5, Insightful) by WizardFusion on Friday May 21 2021, @12:31AM (13 children)
Once you are well rested, it might be worth documenting how it's currently configured. :)
(Score: 5, Informative) by janrinok on Friday May 21 2021, @08:49AM (12 children)
It is documented - but obviously not well enough. The configuration is extremely unusual but that it in part due to the software that we inherited and the fact that many of the original team have fallen by the wayside as real world priorities interfered with their lives. Once the team gets depleted it takes all the effort of those remaining to keep the site running without taking on the tasks that, at the time, seem less important.
I think that we have all learned one lesson already - our team of volunteers is now much too small for the work involved. We need to expand in all areas in my opinion, thus freeing up those who are already overstretched to take things at a more reasonable pace whilst ensuring that there is sufficient expertise to handle critical events such as the downtime that we have just experienced.
So for anybody in our community who thinks that they have something to offer and is willing to give it a try, please get in contact via email or IRC and come and join us. Even if it is only for a limited duration, you might still be able to make a valuable contribution to the site.
(Score: 0) by Anonymous Coward on Friday May 21 2021, @09:46AM (11 children)
pipedot always looked neat and modern.
(Score: 2) by janrinok on Friday May 21 2021, @10:26AM (10 children)
It is based on our software so I don't see much of a benefit there.
(Score: 0) by Anonymous Coward on Friday May 21 2021, @11:22AM (1 child)
(Score: 1, Touché) by Anonymous Coward on Friday May 21 2021, @05:44PM
Pipedot didn't have people like Janrinok, TMB, Martyb, and Mechanicjay.
(Score: 0) by Anonymous Coward on Friday May 21 2021, @11:23AM (7 children)
always got the impression it was newer base code.
http://pipedot.org/story/2014-02-17/pipedot-status-week-1 [pipedot.org]
but the consensus here appeared to be SN landed on better PR and reader/dev group.
i bow to your stronger knowledge and also add my thanks for the SN teams efforts to keep the site going.
(Score: 0) by Anonymous Coward on Friday May 21 2021, @11:42AM
The wayback machine has a last snapshot from January 2nd 2018, with the April 2017 post being the last one. So it’s dead.
(Score: 5, Informative) by janrinok on Friday May 21 2021, @01:57PM (5 children)
As best I can recollect from over 6 years ago: Soylent and Pipedot (Bryan) cooperated together in a friendly manner to produce something to replace slashdot.beta, but following 2 different paths. The idea was that if one project should fail the other would still be around to continue. Pipedot uses a lot of new code written in PHP. Our devs upgraded Slash to the then current libs and released it as Rehash on git from where anybody is free to use it. We both went operational and for a time it seemed that we were likely to compete against each other for a community. I think we had a slight head start and Pipedot decided that it made no sense for us to be fighting for the same readership. Pipedot was put into a sleeping mode, hence the reason why the site is still live but dormant. It could be restarted if Soylent should fail.
I am a bit hazy on the details but that is roughly how I remember it.
See https://soylentnews.org/article.pl?sid=19/08/19/0814218 [soylentnews.org] and https://github.com/pipedot/pipecode [github.com].
(Score: 5, Informative) by bryan on Friday May 21 2021, @05:09PM (4 children)
Nice summary :) I was wondering what happened yesterday when I tried to click on an article and just got errors. Loosing a month of data sucks, but I see that not as a fault in Rehash but as the infrastructure being overly complex for the current need. Reverting MySQL to a simple and more common configuration will help greatly.
As for my own site: still sleeping! Occasionally, some spam messages do get posted (common in all internet forums) - but the moderation system hides those once marked (unless you choose to see them in your preferences.)
(Score: 2) by janrinok on Friday May 21 2021, @05:33PM
I was wondering if you were still around, Bryan! Good to 'see' you again.
Yeah, it is a system config problem and mechanicjay is looking at the best way to remove part of our setup completely. It was intended to give us redundancy and resilience but has proven to be the cause of most of our system downtime.
Rehash is working fine but it is getting increasingly difficult to find people to maintain it. We are looking at various options but the priority is the get the site stable again so we can approach any long-term changes in a calm and methodical manner.
(Score: 2) by Reziac on Saturday May 22 2021, @02:33AM (2 children)
Hey there! sleeping is okay; I'd be very disappointed if it were dead and gone. It seems to work well enough, and the day may come when it's needed awake again.
And there is no Alkibiades to come back and save us from ourselves.
(Score: 3, Funny) by kazzie on Sunday May 23 2021, @08:02AM (1 child)
Now I'm getting crossover between Pipedot and the legend of King Arthur...
(Score: 3, Funny) by Reziac on Sunday May 23 2021, @02:32PM
Zombie fora are everywhere. :D
And there is no Alkibiades to come back and save us from ourselves.
(Score: 5, Funny) by Anonymous Coward on Friday May 21 2021, @12:34AM (4 children)
Or just systemd?
(Score: 5, Funny) by Acabatag on Friday May 21 2021, @01:25AM (3 children)
Explain the difference.
(Score: 5, Funny) by maxwell demon on Friday May 21 2021, @05:53AM (2 children)
Ransomware has a built-in way to undo the damage after you've paid the ransom.
The Tao of math: The numbers you can count are not the real numbers.
(Score: 3, Touché) by kazzie on Friday May 21 2021, @06:39AM (1 child)
Correction: Ransomware *tells you* that is has a built-in way to undo the damage.
(Score: 1) by khallow on Saturday May 22 2021, @11:38PM
(Score: 1, Funny) by Anonymous Coward on Friday May 21 2021, @12:36AM (3 children)
I demand all my karmas back.
(Score: 0) by Anonymous Coward on Friday May 21 2021, @12:52AM (2 children)
I have 150 karma for some reason. You can have some of those.
(Score: 1) by Sulla on Friday May 21 2021, @12:54AM (1 child)
Since when has AC had positive karma?
Ceterum censeo Sinae esse delendam
(Score: 0) by Anonymous Coward on Friday May 21 2021, @01:02AM
What? You think all us ACs have negative karma?
(Score: 5, Insightful) by srobert on Friday May 21 2021, @12:49AM (4 children)
Thank You mechanicjay and TMB. And also to you martyb, and everyone else who keeps this site running.
(Score: 3, Funny) by fustakrakich on Friday May 21 2021, @01:01AM (2 children)
Site goes down, who do ya call? Ghost Buzzard! [soylentnews.org]
La politica e i criminali sono la stessa cosa..
(Score: 1) by Sulla on Friday May 21 2021, @05:49PM (1 child)
You seem to forget.
That which has no record of death may eternal lurk, and in strange eons, even buzzards may fly.
Ceterum censeo Sinae esse delendam
(Score: 2) by Reziac on Saturday May 22 2021, @02:36AM
Death shall not release you...
And there is no Alkibiades to come back and save us from ourselves.
(Score: 0) by Anonymous Coward on Friday May 21 2021, @01:15PM
A big thank you from a regular anonymous reader/lurker! :)
I read this site just about every day, and quite enjoy it. I appreciate the work of all the folks that keep this site up and running.
So .... Thank You!!
(Score: 1, Funny) by Anonymous Coward on Friday May 21 2021, @12:50AM (7 children)
is because they couldn't afford the malware ransom. The thieves kept lowering the ransom until they got to $100, but the SN admins said SN is not a goldmine.
(Score: 5, Funny) by Anonymous Coward on Friday May 21 2021, @12:53AM (1 child)
I told them to install Russian and Ukrainian locales.
(Score: 2) by The Vocal Minority on Saturday May 22 2021, @03:48PM
Ah ha I think we have found the culprits! Russian Hackers obviously hacked SN to remove that story, cunning bastards!
Seriously though thanks Mechanicjay and TMB.
(Score: 5, Funny) by janrinok on Friday May 21 2021, @08:53AM (3 children)
THIS IS NOT TRUE!
They were offering us hard cash to close the site down ourselves - but we are too thick-headed to accept the offer! So, like a phoenix, it has risen from the software ashes until the next time it falls off its perch.
(Score: 2, Funny) by Anonymous Coward on Friday May 21 2021, @12:13PM (2 children)
Nails, my friend. Nails will keep it on its perch.
(Score: 0) by Anonymous Coward on Friday May 21 2021, @04:41PM
> Nails
??
Are you sure you don't mean talons? (grin)
(Score: 1) by Sulla on Friday May 21 2021, @05:51PM
Well you have to do that, otherwise the site would be pushing up the daisies.
Ceterum censeo Sinae esse delendam
(Score: 0) by Anonymous Coward on Friday May 21 2021, @10:52AM
Soylentnews paid the $100 but the restoration attempt didn't work. Soylentnews should have first demanded that the hackers decrypt and restore some random posts of Soylentnew's choice to prove that they can actually restore the rest of it before forking over the money.