SoylentNews

State of the Backend

Right now, the backend is mostly built on an outdated version of mod_perl 2.2, and MySQL cluster, which is very much not a good place to be. Originally as envisioned, I planned this site to be able to be easily scalable, with a larger user base. That's why the infrastructure was designed to be as scalable as it was, with the downside of having a much higher overhead than a more traditional setup has. Furthermore, rehash (the code that powers this site) is, uh, to put it frankly, a beast to work on. It's a 90s era Perl code base and pretty much everything else that implies; if it wasn't for the fact that rehash is one of the main reasons to use SoylentNews, I'd argue it might be time to replace it.

Right now, I'm working on doing another round of server hardening. As it is at the moment, I've got rehash and Apache running in an AppArmor jail, and everything is pretty well sandboxed from everything else, but I still need to go through and adjust a lot of firewalls, and finish decommissioning out a bunch of the boxes. That said, the site is running faster than it has in a long while since a lot of small things got corrected as we went. Sometime this weekend, I'm going to finish adjusting the firewalls to lock it down further, and that should mostly get back to the point where I might have restful sleep again. That being said, there's still a fair bit more to do.

Moving ahead, we need to get off MySQL cluster, and either onto the current mod_perl, or, ideally, FastCGI, to end the Apache dependency entirely. Unfortunately, working on Rehash is quite difficult, and it requires a very specific setup to be viable. My current plan here is to basically get it working in Docker, so its easy to spin up and spin down instances, and return to a less cursed variant of MySQL. This is probably a few hours of work, but I'm hoping that overall it is going to be easy and straightforward to do since most of the backend is fairly well documented at this point. This also leaves me in a decent position to implement a couple of long overdue features, but modernization efforts come first. I'm hoping to livestream my efforts on this on the weeks to come, and I will make stream announcements as I go along.

Policy and Code Changes

My intent, based off the policy changes that were made to disallow ACs to post on stories is to sunlight the feature entirely, including in journals and more. The decision to have ACs on SoylentNews was made in 2014, when the Snowden leaks were only a few months old. Furthermore, we've seen from experience that the karma system doesn't go far enough at keeping bad actors from still getting a +2 status. By and large, the numbers underpinning the system need a rework. My general thought is to cap karma at either 10 or 15, and drastically decrease how far into the basement you can go, as well as uncapping posts in moderation to be able to go to -5.

As a rule, incredibly bad takes do get moderated out of existence, but because there's no real penalty for doing so, we get constant shitposts. Time to make this a bit harder to abuse. I've documented the antispam measures on the site before, but the site keeps track of IP addresses and subnets in the form of hashed /24, and /16s (/64 and /48 for IPv6), which has a karma number attached to them. If an IP range goes too far into the basement, it ends up posting at 0 or -1. By adjusting the caps, it should allow this threshold to be reached much more easily, and help bring the signal to noise ratio back to something more "positive".

Furthermore, I believe its generally in the site's interests to allow editors to delete comments. This functionality is actually built into rehash, but has been long disabled. At the time, I felt the community was best self-moderating, but I think on the whole, its better to treat this like a moderated subreddit, and have messages get a notice that they've in-fact been deleted ala reddit. This is a fairly large departure for the site as a whole, but I think one justified given the state of the Internet on 2022. I am open to discussions on all of this, but let me see what all your thoughts are like.

Final Notes

I do intend to keep livestreaming my progress with the site as we go along; and we raised another ~500 dollars towards Trevor Project during the last livestream. I've left that stream unlisted until I've had a chance to finish implementing all the hardening measures I've discussed, but I'm hoping at the end of it, I'll have a pretty good documentary on what it takes to modernize an aging website. As usual, if you want to support me directly: Ko-fi is available for one time donations, or Patreon for a recurring donation.

~ NCommander

[ If you are an AC and wish to make a constructive comment, please see my journal. janrinok ]

What Happened to NCommander

I guess I should start with what happened to me, since I was head of the project, and I essentially disappeared. For those who knew me, I went through some serious life problems from 2016-2019; I was still active on Soylent's IRC, but I was very much one degree removed from the project. I did write things up like talking about my time as a ICANN fellow, and screwing around with Windows 1.0 SDK, but it was very much a hands off thing. Even at that point, the backend was very much starting to show its age in the years since it first went online. However, there were a few things in the background that was threatening to brew over.

When we launched in 2014, SoylentNews was promised as a baston of free speech, and essentially a testament of the Internet of the early 90s. It was the Slashdot we all remembered, freed from corporate interests that had forced the beta interface. I actually spent a bunch of time documenting what I remember of the SoylentNews launch on my own wiki here, but if you want to remember what lead to the foundation of this site, gaze upon the eye of the beta interface and despair. The end result of this was the SoylentNews Manifesto, essentially our touchstone document of our core values and such. I was 26 when I wrote that document. The Snowden leaks had just happened, and well, the biggest threat on the horizon was the idea of mass censorship. I had, somewhat naively, had assumed that, given a choice, people won't willingly listen to misinformation. Then I saw the Trump presidency first hand, combined with active efforts to lie and distort the truth about a global pandemic, and leading up to the Capitol Riots on January 6th, 2021. From today, it's been eight years since I put pen to paper, and I've had a long time to think about those words.

By and large, while I don't disagree with the principles of what SN was founded on, I've been forced to admit that this has had a lot of undesirable consequences. As the Trump presidency continued, the signal to noise ratio began to drop on the site like a stone. I remember that, at times, this site sometimes felt like reading /b/ more than anything else, and was having serious concerns with the state of the community by 2017-2018. However, there were two major factors that stopped me for doing anything: first, I felt bound my own words, and second, there were members of the staff who preferred to keep things as is.

By and large, volunteer projects depend on the organization have a core set of ethics. While SoylentNews is, simply put, a success story in volunteer collaboration, I'm the one who set the direction in which staff followed. This was true when I took ownership of the project from John, and when we were all at mrcoolbp having a BBQ celebrating our road to incorporation. Even though I haven't been active on SN in years at this point, you can still see the impressions in the clay that have lasted over the better part of a decade. However, this has never been a one man show. Several members of the staff, primarily those who helped maintain the backend, felt that the status quo was better. I could have forced the issue, but I would have likely ended up alienated both staff and the community over it. As such, I just began to silently slide into the background from 2017 onward.

As I continued to move forward in my life well, I began to have the benefit of hindsight. Towards late 2015, and early 2016, I got picked up by Mixer (then called Beam), and created the Faster-Than-Light streaming protocol, redefining the standard of what was possible for video streaming. I then found my way working with whistleblowers, and doing a stint of policy work with ICANN. It's given me the benefit of hindsight, and a lot of wisdom that 2014 didn't me did have. I also struggled with constant health issues. In 2019, I made the decision to try and cross the United States by bike (which I documented on my second channel, Restless Yankee. I started that trip in March of 2020; no points guessing how that ended :)

It's hard to summarize my feelings about the site at that time. By and large, I felt like SN had been a well intended, although ultimately misguided effort to make the world better, and the state of the comments section reinforced that. However, that changed in 2020, with the pandemic. Medical misinformation was rampant on SN, and there were several large fights about this in staff-only IRC channel. I was essentially outnumbered; there were too many people who wanted the toxic waste dump. So I resigned. Or well, intended to. I wrote a fairly lengthily resignation letter, and staged it for release on SoylentNews, and looked at disentangling myself from the PBC that owns the site. For various reasons, I never put it public, although it was briefly leaked.

In the end, the result was the same. I left SoylentNews, and focused on my own YouTube channel, and started building my own community built around the lessons learned from the SN era of my life. By and large, I've succeeded beyond my wildest dreams.

But while I was off doing this, things were changing here.

What Happened In the Interim

Well, as it turns out, I left some very large shoes to fill, and my frustrations about the site didn't fall on deaf ears. I won't go through the full details; for one, I wasn't here, but problems relating to spam, misinformation, and more continued to grow. As before, the people who actually understood how the very legacy Slashdot.org derived codebase worked in were in favor of letting the site continue as is, while those, like janrinok, were beginning to reach a breaking point. I'll skip ahead to the punchline, the staff ended anonymous posting on the main page, which was a major departure from our initial promises

To say there was backlash is very much an understatement. When the dust settled, there were a small number of staff, mostly editors, left, who continued valiantly on for the next two years as the site slowly began to fall apart without maintenance. It's a testament to how much effort was put in both by the original Slashdot crew, and those of us who worked on rehash that it basically stayed up despite no one watching it, but it's starting to reach a breaking point. I took a look at the state of things earlier today. It's *bad*. To put it bluntly, SN's backend was always maintained on what could be described as "best effort", and there were some deferred maintenance issues piling up when I stepped away.

It's now two years later ...

Fortunately, simply because when we launched SN, we were forced on an obsolete stack, I did an obscene amount of machine hardening; the entire thing is running in an AppArmored bubble, and I left a stupid amount of notes on how it all worked. Add to that Slashdot was very the tech site of the 1990s and 2000s, and you have a pretty tough nut to crack. The pile of Perl running this site is old enough to drink. However, we've piled up a large amount of technical debt. By and large, this is not a sustainable situation, but at least for the time being its livable if someone put in the necessary TLC to make it keep going.

So the question is, what comes next?

By and large, SoylentNews has been drifting without a direction for years at this point. I couldn't force a direction without alienating staff, and I didn't have the energy to maintain SN indefinitely without hope things would get better. However, by sheer dint of staying up (probably of spite), you have one of the last time capsules to an Internet that has mostly disappeared. One without JavaScript, and a tribute to everything awesome about the 90s and 2000s. Watching SN die out of apathy hurt would hurt too much. This is where we get back to "now". Although I had departed staff, I did occasionally check in on things. I did see the problems with trolls, and I had at least read through parts of the comments when anonymous posting finally got turned off. I actually did feel like SN had a chance of turning itself around. What I didn't know is how bad things had been internally.

Since moving off from SN, I've been working to actively preserve pieces of history; for example, through a community effort taking place mostly on my Discord, we've seen AIX for Itanium get archived, and even restored to the point we have a working copy of GCC. This was one of the centerpieces of the SCO v. IBM lawsuites of the early 90s, and was a huge focus point for Groklaw. The legendary Project Monterey; preserved for future generations, and I was there when it happened. We've also worked to save versions of Banyan VINES, and I'm even hosting a "Slow Computing Speedrun" as we speak, which, among other things, has someone with a genuine PDP-10 being livestreamed at this moment! You could say it's been an interesting time ...

Recently, I was talking about SoylentNews, and what I could remember relating it being forked from Slashdot, which I was urged to write in a public channel. The discussion basically boiled down to "how often do you get to siphon a large amount of the Slashdot old guard". This lead to a bunch of volunteers to write an entire page relating to this sites history; and as many notable things as I could remember over the last eight years. The full log has been saved here, but I'll give you some samples:

On Slashdot Beta

• N: So, some context, Slashdot was formed in the very early 90s by CmdrTaco, and CowboyNeal, pretty much the first tech news of the early Internet
• N: was in the days of the dotcom boom, and it got handed over probably a dozen times; it was an Internet landmark, like SourceForge, or Tucows was
• N: Eventually Slashdot got acquired by DICE Enterainment, after a very long period of changing hands
• N: Like all the original editors were gone
• BCD: I wanna say it was right-leaning? Hated microsoft and loved open-source
• N: DICE had made a lot of unpopular changes, but the straw that broke the camels back was "Beta"
• N: Like it was this horrid whitespace meme; it turned a very information dense site into a wordpress block...It was like if every page of the print New York Times was double spaced.

On soylentnews.org being held hostage

• N: John [the original founder] had actually blackmailed us. And sold the soylentnews.org domain without giving it to staff
• N: Matt Angel bought it for 3k (https://soylentnews.org/article.pl?sid=14/03/10/1129232/)
• N: Right, [I] forced him out, and then he sold the domain
• N: fucking hell this is a story
• N: Anyway, Matt basically was our venture captilist, and I was the CTO
• N: And I basically said "we're going to incorporate this, and we're going to do it right"
• N: and we did
• N: Fitting, that we got our incorporation papers back on July 4th

On upgrading the code to mod_perl 2

• N: This is not something you do lightly. Like, this is a fucking effort
• N: This was the itch I couldn't let stand
• N: I said at the beginning "we're going to do that"
• N: and that was the last major thing I did in that code base

After finishing that interview, I ended up taking a much closer look at the state of the site, and found myself immensely depressed. This is how it ends? With a wimper? Nah. I've made an entire career out of doing the fucking impossible. This site is proof of it. So, here I am again, preparing to step once again into the breach ...

The Road To Dawn

Just because of how much time I spent on it, I refuse to let SoylentNews die a slow painful death, but I also don't want to be in the position where I'm going to have to maintain it as a one man show indefinitely. 8 years ago, we built this community ex nilihio in a week. I can find the time to at least get us back to serviceable. Right now, I'm probably looking at many hours of work ahead of me, but there's a silver lining. I do a lot of charity livestreaming on cursed and vintage technology.

So, why not livestream it? The honest truth is a lot of people like to watch me suffer excessively, and this is going to be a test of patience beyond anything else. It will also (hopefully) bring a lot of fresh eyes looking towards SoylentNews, and perhaps teach some people on what it takes to keep a 30 year old codebase going. It's literally an artifact of a bygone era, one that is, rather depressingly, disappearing before our eyes. However, there's a bigger thing I need to look at.

February will mark our ninth anniversary, and frankly, I think we can make it to a decade with just a bit of effort. At which point, I'm going to need to decide what we're going to do. I don't want to see a long slow painful death. If anything, I rather do an orderly shutdown, archive everything for posterity, and leave soylentnews.org as a marker that says we were here, and let me stress this point, if nothing changes. If we can find new staff, and folks who are willing to maintain the site, then SoylentNews will go on, for as long as there are people there are to tend for it. If not, then it will be left as a testament of what is actually possible when people get together.

Assuming the community is willing to go along with this plan, I'll likely host a stream this weekend doing a fairly through inspection of the site infrastructure, and fixing any critical issues I can find live. I'll put a stream announcement up on SoylentNews, on my personal Twitter, and on Mastodon (@ncommander@restless.systems), as well as on my YouTube channel.

At least some of these streams will be fundraising for charity. Currently, I've fundraising for The Trevor Project, while in the past, I've fundraised for both National Network of Abortion Funds, and Planned Parenthood of America. I'm pretty sure the comments section might have thing to say, and frankly? I don't really care. If you don't like it, you can put in the effort yourself to save this place.

I think, at this point, this NCommander novel has reached its final form, so I will drop a link to my Ko-fi, and Patreon for one time and recurring support for me directly.

Until next time, this is NCommander, signing off, wishing you all a pleasant day ...

[This has been recovered from the WayBack machine from a link (https://web.archive.org/web/20221111011540/https://soylentnews.org/meta/article.pl?sid=22%2F11%2F10%2F1428220) very kindly provided by an Anonymous Coward. I cannot reproduce the original comments - there is no mechanism in the software to do such a thing - nor should it ever have been necessary. If you want to copy the comments that you made and resubmit them you can, of course, do so but they are unlikely to appear in the same order as they were made in the original posts. JR]