SoylentNews

This is the post I never thought I would have to make. I am also writing this post on behalf of SoylentNews PBC, the legal owner of SoylentNews, and not as a member of the staff or the community.

SoylentNews is going to shut down operations on June 30th.

This wasn't an easy decision to come to, and it's ultimately the culmination of a lot of factors, some which were in my control, and some that weren't. A large part boils down to critical maintenance to the site not properly being performed for a very long time. To pay back the mountain of technical debt we've built up, it would require relaunching the site from scratch.

I'll discuss this more in depth below, but I can't personally justify the time any more, especially due to the negative impact that SN is having on my personal life.

Before we shut down, at least for the foreseeable future, I'm going to outline the situation as I see it, my own personal responsibility, and what happens next.

Technical Reasons

Let's start with the technical nitty gritty. SoylentNews was, in November 2022, at the point where it was about to have a fatal database crash. The database cluster was wedged in an invalid state. Backups weren't properly being done. As it was, we lost several days of postings after a hard crash. On top of that, we had multiple public facing machines running outdated versions of CentOS and Ubuntu running net facing services.

There's two distinct problems here, both of which have to be addressed.

The first is the site itself, and specifically what we can or can't do with it. At this point, rehash, the backend that runs SoylentNews, is nearly 30 years old, and it was written in what can be generously described as angry and especially esoteric Perl. Perl was already going extinct when we launched in 2014, and at this point is mostly relegated to legacy backend code which is slowly but surely disappearing.

Complicating matters is that rehash is specifically tied into Apache 2.2 via mod_perl, a version that is well past end of life and significantly out of date. While the website is well sandboxed and battle hardened, running obsolete code on the public Internet is not a smart move especially when combined with many of the other factors listed below.

To just keep up with patched software, we would either need to port the site to Apache 2.4 and a recent version of mod_perl, or break the Apache dependency with an alternative like FastCGI. This would also require updating the base version of Perl5 to something more recent and hoping all the necessary CPAN modules have either been updated, or can be reasonably replaced, which is doubtful at best

As the person who actually did the base port of rehash from Apache 1.3 to Apache 2, this is a massive project regardless of which way we would go. This would require a full rebuild of the /srv/soylentnews.org directory which alone could easily take weeks or months of work. That doesn't take into account all the other bits of infrastructure and software that would need to be reworked, rebuilt, or replaced.

When we migrated to rehash, we had more staff who could QA the site and quite a bit went wrong trying to do that relatively simpler migration.

As we are now?

I don't see how it's possible anymore.

After everything that has played out, I'm having trouble working up sufficient motivation to work on the site and bring it up to a serviceable state when combined with the amount of friction I've experienced just getting us here.

I had hoped to hire outside help or at least raise enough through livestreaming other SN related work to offset the costs. At this point though I believe that is a lost cause as well. If this was the only major problem, it would be bad enough. Unfortunately, it's just the tip of a very large and very ugly iceberg.

The deeper problem is that everything else has bitrotted over time.

SN's backend is something of a jigsaw puzzle which is documented in one of three places: on the site, on the internal technical wiki (which is currently down), and on the old public facing wiki which is also down. None of that documentation was or is consistent with the actual state of reality, and quite a few parts, like the MySQL cluster, were somewhat esoteric.

In practice, if you want to know how anything was plugged into anything, it was a matter of pulling cables and figuring out what broke. It also doesn't help that the backend is notoriously noisy. That makes it hard to sort out real errors from the chaff. This was a large part of why the Zoo plugin, which does the sidebars, was broken for most of December. It also didn't help that we had three different OSes (Ubuntu, CentOS, Gentoo) which complicated system administration.

Furthermore, there have been major disagreements among the sysops on actually doing any major upgrades. Someone would complain that we should do something. There would be a lot of arguments about it. In most cases, nothing got done. Because of this ongoing friction, it became increasingly more common for no one to install updates. This is why we never upgraded from Ubuntu 14.04 to 16.04 back in 2016. I eventually said we should just go to Gentoo, since there was a widespread belief that upgrading the distribution would break everything. This suggestion ultimately just ended up with only our development machine on Gentoo, and that too was woefully out of date.

When I finally checked in in November 2022, after two years, the site had finally reached a breaking point. I talked with some people in #chillax, and I got a state of affairs from mechanicjay, and I decided to do what should have been done long ago.

Clean house.

I didn't ask for permission. I didn't wait for people to answer DMs. I just did it because we had done this go around one too many times in the past.

I will let the community decide if I was justified or not in doing so.

A lot of this involved installing over a decade of upgrades. Setting up and configuring firewalls and removing unneeded services. Backing up and decommissioning old boxes. Given the extended period of time without updates, you can imagine that I at least have some concern at the number of potentially vulnerable backend services that were exposed to the Internet.

I found no evidence of breach, but given the period of time, and general lack of maintenance, I am at best uneasy.

I could have done better.

In the end, I finally installed almost a decade of upgrades in December of 2022, but that only postponed the inevitable. I also trimmed the number of machines and services in an effort to be at least slightly more secure on the Internet. However, ultimately, without a way to bring in new users, SN is slowly going to attrition itself to death.

Some might argue that I simply let it be, or should have let it be in November, but I really did hope that I could pull it out of this death spiral. Over the last few months, it has become clear that the only way work is going to be done is if I do it or if a miracle happens.

The problem is: as part-owner, where do you go from here?

There's also the matter of liability.

Ultimately speaking, if something happened with SN, Matt and I would be jointly responsible since our names are on the legal documents. I tried to find someone to take my place, and failed. I am legally attached to something that is barely being maintained, and frankly, I can't carry this cross any further.

My Role In This Outcome

I guess this falls down to a lot of my personal responsibility. While SN is at its heart a community project, it is also a business, one for which I have served as its president for its entire life. I really had no idea what I was doing when we started, and this had long term effects on SN as a whole. Part of this was that we only had subscriptions as a revenue stream.

Without a more solid revenue stream, the PBC was essentially hostage to the small trickle of money subscriptions. In a volunteer organization, it's a matter of "who shows up to do the work" dictating the direction of the site.

In the early days this wasn't a problem, I had plenty of free time, and people were often willing to help. That's largely how the site got ported to Apache 2, and why we were able to stay up for more than a year. Meanwhile, solving UTF-8 support was one of TMB's and MartyB's projects. As the early enthusiasm died off and staff began to leave, essential tasks were becoming less and less likely to be done.

That ultimately created a negative feedback loop in which technical debt continued to pile up.

SoylentNews also doesn't have a growing community, partially because we have very few inbound links, and are fairly low in search results. In our early days, folks followed us from Slashdot, and some viral posts on places like reddit and HackerNew did help to build the community, but this has largely evaporated.

Growth of some sort is important because communities have a natural attrition rate. People leave, die, or otherwise go inactive. Year over year, the community has shrunk, primarily because we don't bring in a lot of new blood.

Furthermore, the Internet as a whole has changed. When we started, GamerGate was yet to happen. The world couldn't even imagine the rise of the Trump presidency. In theory, the moderation system should have been able to handle disinformation, but the mod system requires a certain critical mass to work. Slashdot's mod system could only work as it does on a large community, and we found at least one critical flaw with its base assumption:

People rarely if ever downvote.

This, combined with ineffective anti-spam meant that it was relatively easy to game the system, and allowing bad actors outweighed good. My perception was SN's signal to noise ratio was becoming more noise year over year, and there were many conversations on this, which ultimately went nowhere. For me, personally, it finally reached a head with COVID. The amount of medical misinformation and similar such disinformation got to the point that I felt we needed to drastically overhaul the site.

This lead to some very bitter arguments.

Ultimately, I was overruled, and I attempted to resign after bitter arguments in the staff channel. My resignation was written, but ultimately never posted, and I left on bad terms with the staff at the time. Consequently, I remained President of the PBC. At that time, I requested Matt remove me from the position, but we never formalized this, primarily because there was no one to replace me. It should also be noted that we were missing a secretary and unable to find a replacement after mrcoolbp withdrew due to personal life reasons.

I could have, and perhaps should have, forced the issue then, but I could still remember how the domain was hijacked in our early days, and didn't want this to be a case of sour grapes. I also had a reasonable belief that SN would still be maintained by the active staff. I turned my attention towards my other endeavors such as my YouTube channel and tried to put it behind me.

Two years passed.

That was not the end of the infighting, and that ultimately led up to TMB leaving in 2021. The site was now running with the bare minimum of maintenance mechanicjay supported by audioguy could give it with no hope of a long term solution in sight. Had I not checked in, and decided to do emergency maintenance, the odds are that it would have been a matter of weeks or months before a severe system crash would have irreparably corrupted the database.

As it was, we had two hard crashes that lost weeks of posts. There were no functioning backups that I could find.

I did two emergency rounds of maintenance that saw the backend database replaced with a standard vanilla MySQL instance and drastically downsized the number of machines, cutting the monthly bill more than half.

However, it's become clear to me that this was too little, too late.

Many of the issues that were present when I resigned were still here. At the end of the day, I found myself caught between my responsibility towards my site and my own frustrations for what it had become. This combined with a personal disaster in my life starting in December meant that I had very little time for SN.

This was also combined with the dawning realization of how difficult it would be to get new sysops and devs to replace myself and those that had left. While I was willing at least to put some of the legwork in, no one really wanted to sit down and help with the business side of things. It felt like everyone else decided we should all hum loudly. While we had some volunteers for sysops, my lack of time, combined with the relatively arcane nature of our backend mostly nothing being done.

I honestly don't know if there was one specific misstep that led to this outcome. However, the need for sites like Slashdot and SoylentNews was already passing when we launched. Slashdot is a shell of itself, and most of the role of news aggregator is taken up with sites like reddit and HackerNews. The need for something like SN has largely disappeared.

That means for SN to exist, it has to exist for itself, and well, that's the rub of it. SN stopped being maintained while I was absent. It wasn't being well maintained well before that point. It's not going to be maintained now simply because I can't justify the time and effort anymore and no one else is putting time or effort into this either.

Suggestions like running ads to try and pay for some of the maintenance costs have either been rejected or at least treated with enough skepticism that makes me doubtful it would help.

Finally, I'm tired of fighting over every single issue which in the end leads to nothing being done and everyone just walking away unhappy.

What Would Have Been Needed To Save SN?

As before, I'll break this into two sections, involving the technical, and the non-technical. To summarize, it essentially required people to take responsibility and pledge to fix it as well as relieve me from my position from the PBC.

Technically speaking, we'd need to be able to refresh the site infrastructure as well as the site's backend dependencies. You're essentially dealing with a legacy Linux install that has been upgraded from Ubuntu 12.04 to Ubuntu 22.04 that at least a dozen of sysops have worked on.

To reduce site admin burden, we'd probably end up migrating email, and most services beside IRC and the website to third party hosting providers. This would have solved many of the email and registration issues that have plagued the site since GMail made their spam barriers extremely hostile to external SMTP hosted mail.

We would also need a development environment that properly tracked with production to allow changes to be done incrementally and rolled back, something that was a continuous problem throughout every major site upgrade. This would let us test each aspect of the overhaul and deploy it piecemeal instead of having the site be broken for weeks or months as happened with the much smaller November upgrade.

Ideally, we would use an automation deployment solution such as GitHub Actions which would make sure the machine state was always in sync with the build files, and allow for easy and rapid deployment of backend patches and security updates.

With this all done, it would have allowed site maintenance to easily be done en masse to all machines and without risk of the site breaking in new and arcane ways.

I did talk to Matt about the possibility of either fundraising or selling stock in an effort to finance it.

I also made multiple efforts to find someone who was willing to seriously take over the site and take over the PBC. There were a few email discussions that went ultimately nowhere.

What it boils down to is that to do anything with the site, I would have to put in legwork that, after everything that has been said and done, I am no longer willing to do nor is anyone truly stepping in to try and take over for me.

It doesn't help that nearly every single thing I've laid out here was shot down by at least one other member of staff while at the same time no realistic alternatives were worked on or even proposed.

What Happens Now?

At this point, we need to get the expenses of the PBC to zero. We have about $1,500 USD in the bank, most of which will go to handle our shutdown fees. I want to give a window for people to exchange contact info and write goodbyes. Subscriptions will be disabled on the site by time this post goes live. SN doesn't have a robust infrastructure to process refunds, and TMB wrote most of the code involving that. I am discussing with Matt what our options here are, but in the worst case scenario, any leftover will be donated to the EFF.

A final backup of the VMs and site database will be taken and soylentnews.org will be redirected to a static page. Everything representing the site be archived and taken offline. I'm going to hold the domain name and backups in trust in the hope that circumstances in the future may allow for the site to return in some form.

I wish I did not need to say such a thing might happen, but all things must end.

Until we meet again, ~ NCommander