SoylentNews

We talked about the state of things to come for over an hour, and then I had one on one calls with Matt and kolie. The conversation was fairly high-level, and mostly consisted of a recap of the last few weeks, current progress, as well as what actually rebuilding the site is going to look like. Right now, we have a commitment to finishing the infrastructure overhaul, and upgrade it as is practical.

That would basically bring us to the "ok for now" status quo ...

However, the status quo does not address the dwindling signal to noise ratios or shrinking community. It does not address the lack of moderation or content standards. It does not address our problems relating to SEO or anything resembling modern human interface guidelines. It also does not resolve the long standing problems that lead to here.

We also had a fairly long conversation on the circumstances that lead to here and how we avoid a repeat of ending back on the brink.

We still have the question of what we will do in the immediate future.

I have no illusion that staff are happy to work with me, but I am hoping we can at least define some sort of formal truce in the name of the future of the site. However, members of the staff have been demanding for me to step down and simply get out.

I have been given the impression that if I was to step down, they would simply continue the site as is. I am unaware of any defined plan relating to fixing any of the problems plaguing the site, both as it is now, and those that have persisted long before this point.

I have also been given the impression that if I do not simply GTFO soon, they're simply going to quit.

That's fair and understandable given everything that has happened.

However, I am not the only stakeholder, as currently, Matt as co-owner, and kolie, as an interested outside party who is actively helping to fix the site, have both indicated that they want me to stay involved at this time.

Ultimately, dealing with the deferred maintenance is going to take priority. Given everything, I don't see how I can have a revised plan for SoylentNews any earlier than July, and realistically I expect it to take longer to have a solid agreement hashed out by all stakeholders.

The staff should be a part of this discussion. However, at this point, they have at this point only made demands, and have made no attempts that I am aware of to negotiate anything. Unilateral demands is not a negotiation.

This is part of a larger problem that, at the end of the day, the staff had no true stake if the site succeeded or failed. This is compounded by the fact that no one has been willing to put themselves forward to join as a member of the board of the directors since mrcoolbp disappeared, and since none are apparently willing to work with me, the person who is both president and 50% owner, well, it leaves us at an impasse.

The easiest thing to do is to allow them to present their case as to why I shouldn't be involved and show to everyone that they can lay out a realistic plan to deal with the issues that have long plagued this website. After all, if they're so insistent that I shouldn't be involved, then as a matter of due diligence, I need them to show a plan that involves fixing the site, and actual work being doing towards it.

For my part, I will do the same in the form of a new business plan for the PBC that will specifically explain why we are taking money in, how we will use it, how we are going to deal with raising capital in the future, and everything else that is involved in keeping SN going for another decade. Such a plan is going to require both kolie and I to discuss the specifics of what replacing rehash is going to require.

At the end of the day, regardless of who was responsible, SN decayed to the point that the database was suffering from corrupted tables. That has to be addressed, and ultimately, until some else steps up to the plate, and presents a workable plan, that falls to me

Infrastructure Rebuild

Of course, talk is cheap, so here's what I have actually done in the last two weeks with help from kolie.

We have been making slow but steady progress on this. The plan is to convert the entire site to an ansible playbook, which now exists. We have successfully started deploying site services on a fresh set of Linode accounts. So far, kolie has got the public wiki rendering, bringing it from MediaWiki 1.18 to a currently supported version.

Meanwhile, I've been digging deeper into rehash. In any scenario that involves the site continuing, we are going to need to be able to deploy code changes. Fortunately, when I did the work to get the site running on Apache 2, I left myself a lot of good notes on how it all works, as well as the "make build-production-environment" target which handles a lot of the worst parts of how to deal with the mountain of legacy Perl.

This is slowly coming together in building a Dockerfile that's on the public rehash repo.

As of writing, I have gotten it to the point it can successfully run the install target. I wrote some rather hacky code that handles shoving connection database information into the system DBIx::Password module, because that's what rehash requires, and I was reminded again why this is the codebase from hell.

We have also had a longer discussion towards implementing new services like status monitoring. A member of the community submitted a long and detailed plan to implement Prometheus for service monitoring. We're not quite ready for that, but we also need to talk about specifically access to the backend.

Infrastructure Access

When I formally announced the shutdown for SoylentNews PBC, I locked out all shell accounts to the backend as well as limited access to the Linode panel. This was done both to protect the site and as a matter of liability. When the circumstances changed, access to the backend was not restored. Access to rehash's administration panel however remained available, which is how janirirok and the other editors have been posting articles. The #chillax channel remains up, although I'm no longer in it.

I didn't restore access after the situation changed. There are quite a few reasons I could give, but given the sheer amount of hostility I have received from certain individual members of the staff, I could not and cannot rule out the possibility that someone would simply "rm -rf /" the production environment out of sheer spite. Also, these machines are going away. There isn't going to be shell access beyond this point aside from a control node.

Right now, the current plan is to simply get the site back to the level of functionality it had back in November, with as many parts of infrastructure being fully up to date as-is possible. This includes going through the configs, removing obsolete bits, and basically reviewing every aspect of the underlying nuts and bolts.

I am going to be taking a solid look at getting rehash ported to the current versions of Apache 2.4 and mod_perl. It mostly depends how much of the stack can be built on mod_perl 2.4 easily. At least with the site on deployable infrastructure, it drastically simplifies what it will take to move forward.

Timeline Moving Forward

Fixing SN is going to require people to be involved and dedicated to rebuilding and essentially relaunching the site as well as fixing many of the problems that have led to here.

To ask that without providing some sort of compensation is folly but without a defined plan, well, we end up in a catch-22 situation.

So, here's what I'm going to do and what I am going to ask of anyone who stays involved.

We will have the site migrated to less broken infrastructure by the end of June at the latest, and likely well before that point.

After that migration is complete, kolie and I are going to negotiate a contract that will handle either overhauling or outright replacement of rehash. This agreement is going to define any new functionality that will be built into what will essentially be version 2 of this website.

We also need to define what specifically is the role of various volunteers in the upkeep of this site, as well as defining our options in case we ever end in another situation like this five or ten years down the road.

If there's one thing I have surmised, no one is happy with this situation, but a big part of moving forward is having an agreement on how it will be maintained and have some flexibility going forward.

A major point of this is growing SoylentNews's cash flow to the point that it can reasonably afford to have at least one paid staff member and provide some actual incentives to keep people involved in its upkeep and overall maintenance.

I have quite a few ideas on how to approach this, but I rather have a chance to write them out in-depth and explore them.

As part of this, I also want to deal with raising the overall signal to noise ratio, increasing the number of comments per article, and hopefully bringing in new blood to the site. It's very hard to have any idea as of the state of the health of the community as we don't have any analytics, and have historically only ever run PiWik for a short period of time.

I think we need to consider doing that again, if only to have an idea of where we are in terms of actual readership and engagement.

Finally, there needs to be a reason why someone might post inbound links to SoylentNews, and once someone clicks on them, sticks around longer than a few seconds. That also makes finding such content easier for both humans and search engines.

These are just some ideas off the top of my head. Realistically, I'm going to need to sit with a pen and paper, sound out which ones are practical with kolie and Matt, and then start taking steps to implement them, with whomever is willing to join us going forward.

I will keep you all posted,

~ NCommander

Technical Reasons

Let's start with the technical nitty gritty. SoylentNews was, in November 2022, at the point where it was about to have a fatal database crash. The database cluster was wedged in an invalid state. Backups weren't properly being done. As it was, we lost several days of postings after a hard crash. On top of that, we had multiple public facing machines running outdated versions of CentOS and Ubuntu running net facing services.

There's two distinct problems here, both of which have to be addressed.

The first is the site itself, and specifically what we can or can't do with it. At this point, rehash, the backend that runs SoylentNews, is nearly 30 years old, and it was written in what can be generously described as angry and especially esoteric Perl. Perl was already going extinct when we launched in 2014, and at this point is mostly relegated to legacy backend code which is slowly but surely disappearing.

Complicating matters is that rehash is specifically tied into Apache 2.2 via mod_perl, a version that is well past end of life and significantly out of date. While the website is well sandboxed and battle hardened, running obsolete code on the public Internet is not a smart move especially when combined with many of the other factors listed below.

To just keep up with patched software, we would either need to port the site to Apache 2.4 and a recent version of mod_perl, or break the Apache dependency with an alternative like FastCGI. This would also require updating the base version of Perl5 to something more recent and hoping all the necessary CPAN modules have either been updated, or can be reasonably replaced, which is doubtful at best

As the person who actually did the base port of rehash from Apache 1.3 to Apache 2, this is a massive project regardless of which way we would go. This would require a full rebuild of the /srv/soylentnews.org directory which alone could easily take weeks or months of work. That doesn't take into account all the other bits of infrastructure and software that would need to be reworked, rebuilt, or replaced.

When we migrated to rehash, we had more staff who could QA the site and quite a bit went wrong trying to do that relatively simpler migration.

As we are now?

I don't see how it's possible anymore.

After everything that has played out, I'm having trouble working up sufficient motivation to work on the site and bring it up to a serviceable state when combined with the amount of friction I've experienced just getting us here.

I had hoped to hire outside help or at least raise enough through livestreaming other SN related work to offset the costs. At this point though I believe that is a lost cause as well. If this was the only major problem, it would be bad enough. Unfortunately, it's just the tip of a very large and very ugly iceberg.

The deeper problem is that everything else has bitrotted over time.

SN's backend is something of a jigsaw puzzle which is documented in one of three places: on the site, on the internal technical wiki (which is currently down), and on the old public facing wiki which is also down. None of that documentation was or is consistent with the actual state of reality, and quite a few parts, like the MySQL cluster, were somewhat esoteric.

In practice, if you want to know how anything was plugged into anything, it was a matter of pulling cables and figuring out what broke. It also doesn't help that the backend is notoriously noisy. That makes it hard to sort out real errors from the chaff. This was a large part of why the Zoo plugin, which does the sidebars, was broken for most of December. It also didn't help that we had three different OSes (Ubuntu, CentOS, Gentoo) which complicated system administration.

Furthermore, there have been major disagreements among the sysops on actually doing any major upgrades. Someone would complain that we should do something. There would be a lot of arguments about it. In most cases, nothing got done. Because of this ongoing friction, it became increasingly more common for no one to install updates. This is why we never upgraded from Ubuntu 14.04 to 16.04 back in 2016. I eventually said we should just go to Gentoo, since there was a widespread belief that upgrading the distribution would break everything. This suggestion ultimately just ended up with only our development machine on Gentoo, and that too was woefully out of date.

When I finally checked in in November 2022, after two years, the site had finally reached a breaking point. I talked with some people in #chillax, and I got a state of affairs from mechanicjay, and I decided to do what should have been done long ago.

Clean house.

I didn't ask for permission. I didn't wait for people to answer DMs. I just did it because we had done this go around one too many times in the past.

I will let the community decide if I was justified or not in doing so.

A lot of this involved installing over a decade of upgrades. Setting up and configuring firewalls and removing unneeded services. Backing up and decommissioning old boxes. Given the extended period of time without updates, you can imagine that I at least have some concern at the number of potentially vulnerable backend services that were exposed to the Internet.

I found no evidence of breach, but given the period of time, and general lack of maintenance, I am at best uneasy.

I could have done better.

In the end, I finally installed almost a decade of upgrades in December of 2022, but that only postponed the inevitable. I also trimmed the number of machines and services in an effort to be at least slightly more secure on the Internet. However, ultimately, without a way to bring in new users, SN is slowly going to attrition itself to death.

Some might argue that I simply let it be, or should have let it be in November, but I really did hope that I could pull it out of this death spiral. Over the last few months, it has become clear that the only way work is going to be done is if I do it or if a miracle happens.

The problem is: as part-owner, where do you go from here?

There's also the matter of liability.

Ultimately speaking, if something happened with SN, Matt and I would be jointly responsible since our names are on the legal documents. I tried to find someone to take my place, and failed. I am legally attached to something that is barely being maintained, and frankly, I can't carry this cross any further.

My Role In This Outcome

I guess this falls down to a lot of my personal responsibility. While SN is at its heart a community project, it is also a business, one for which I have served as its president for its entire life. I really had no idea what I was doing when we started, and this had long term effects on SN as a whole. Part of this was that we only had subscriptions as a revenue stream.

Without a more solid revenue stream, the PBC was essentially hostage to the small trickle of money subscriptions. In a volunteer organization, it's a matter of "who shows up to do the work" dictating the direction of the site.

In the early days this wasn't a problem, I had plenty of free time, and people were often willing to help. That's largely how the site got ported to Apache 2, and why we were able to stay up for more than a year. Meanwhile, solving UTF-8 support was one of TMB's and MartyB's projects. As the early enthusiasm died off and staff began to leave, essential tasks were becoming less and less likely to be done.

That ultimately created a negative feedback loop in which technical debt continued to pile up.

SoylentNews also doesn't have a growing community, partially because we have very few inbound links, and are fairly low in search results. In our early days, folks followed us from Slashdot, and some viral posts on places like reddit and HackerNew did help to build the community, but this has largely evaporated.

Growth of some sort is important because communities have a natural attrition rate. People leave, die, or otherwise go inactive. Year over year, the community has shrunk, primarily because we don't bring in a lot of new blood.

Furthermore, the Internet as a whole has changed. When we started, GamerGate was yet to happen. The world couldn't even imagine the rise of the Trump presidency. In theory, the moderation system should have been able to handle disinformation, but the mod system requires a certain critical mass to work. Slashdot's mod system could only work as it does on a large community, and we found at least one critical flaw with its base assumption:

People rarely if ever downvote.

This, combined with ineffective anti-spam meant that it was relatively easy to game the system, and allowing bad actors outweighed good. My perception was SN's signal to noise ratio was becoming more noise year over year, and there were many conversations on this, which ultimately went nowhere. For me, personally, it finally reached a head with COVID. The amount of medical misinformation and similar such disinformation got to the point that I felt we needed to drastically overhaul the site.

This lead to some very bitter arguments.

Ultimately, I was overruled, and I attempted to resign after bitter arguments in the staff channel. My resignation was written, but ultimately never posted, and I left on bad terms with the staff at the time. Consequently, I remained President of the PBC. At that time, I requested Matt remove me from the position, but we never formalized this, primarily because there was no one to replace me. It should also be noted that we were missing a secretary and unable to find a replacement after mrcoolbp withdrew due to personal life reasons.

I could have, and perhaps should have, forced the issue then, but I could still remember how the domain was hijacked in our early days, and didn't want this to be a case of sour grapes. I also had a reasonable belief that SN would still be maintained by the active staff. I turned my attention towards my other endeavors such as my YouTube channel and tried to put it behind me.

Two years passed.

That was not the end of the infighting, and that ultimately led up to TMB leaving in 2021. The site was now running with the bare minimum of maintenance mechanicjay supported by audioguy could give it with no hope of a long term solution in sight. Had I not checked in, and decided to do emergency maintenance, the odds are that it would have been a matter of weeks or months before a severe system crash would have irreparably corrupted the database.

As it was, we had two hard crashes that lost weeks of posts. There were no functioning backups that I could find.

I did two emergency rounds of maintenance that saw the backend database replaced with a standard vanilla MySQL instance and drastically downsized the number of machines, cutting the monthly bill more than half.

However, it's become clear to me that this was too little, too late.

Many of the issues that were present when I resigned were still here. At the end of the day, I found myself caught between my responsibility towards my site and my own frustrations for what it had become. This combined with a personal disaster in my life starting in December meant that I had very little time for SN.

This was also combined with the dawning realization of how difficult it would be to get new sysops and devs to replace myself and those that had left. While I was willing at least to put some of the legwork in, no one really wanted to sit down and help with the business side of things. It felt like everyone else decided we should all hum loudly. While we had some volunteers for sysops, my lack of time, combined with the relatively arcane nature of our backend mostly nothing being done.

I honestly don't know if there was one specific misstep that led to this outcome. However, the need for sites like Slashdot and SoylentNews was already passing when we launched. Slashdot is a shell of itself, and most of the role of news aggregator is taken up with sites like reddit and HackerNews. The need for something like SN has largely disappeared.

That means for SN to exist, it has to exist for itself, and well, that's the rub of it. SN stopped being maintained while I was absent. It wasn't being well maintained well before that point. It's not going to be maintained now simply because I can't justify the time and effort anymore and no one else is putting time or effort into this either.

Suggestions like running ads to try and pay for some of the maintenance costs have either been rejected or at least treated with enough skepticism that makes me doubtful it would help.

Finally, I'm tired of fighting over every single issue which in the end leads to nothing being done and everyone just walking away unhappy.

What Would Have Been Needed To Save SN?

As before, I'll break this into two sections, involving the technical, and the non-technical. To summarize, it essentially required people to take responsibility and pledge to fix it as well as relieve me from my position from the PBC.

Technically speaking, we'd need to be able to refresh the site infrastructure as well as the site's backend dependencies. You're essentially dealing with a legacy Linux install that has been upgraded from Ubuntu 12.04 to Ubuntu 22.04 that at least a dozen of sysops have worked on.

To reduce site admin burden, we'd probably end up migrating email, and most services beside IRC and the website to third party hosting providers. This would have solved many of the email and registration issues that have plagued the site since GMail made their spam barriers extremely hostile to external SMTP hosted mail.

We would also need a development environment that properly tracked with production to allow changes to be done incrementally and rolled back, something that was a continuous problem throughout every major site upgrade. This would let us test each aspect of the overhaul and deploy it piecemeal instead of having the site be broken for weeks or months as happened with the much smaller November upgrade.

Ideally, we would use an automation deployment solution such as GitHub Actions which would make sure the machine state was always in sync with the build files, and allow for easy and rapid deployment of backend patches and security updates.

With this all done, it would have allowed site maintenance to easily be done en masse to all machines and without risk of the site breaking in new and arcane ways.

I did talk to Matt about the possibility of either fundraising or selling stock in an effort to finance it.

I also made multiple efforts to find someone who was willing to seriously take over the site and take over the PBC. There were a few email discussions that went ultimately nowhere.

What it boils down to is that to do anything with the site, I would have to put in legwork that, after everything that has been said and done, I am no longer willing to do nor is anyone truly stepping in to try and take over for me.

It doesn't help that nearly every single thing I've laid out here was shot down by at least one other member of staff while at the same time no realistic alternatives were worked on or even proposed.

What Happens Now?

At this point, we need to get the expenses of the PBC to zero. We have about $1,500 USD in the bank, most of which will go to handle our shutdown fees. I want to give a window for people to exchange contact info and write goodbyes. Subscriptions will be disabled on the site by time this post goes live. SN doesn't have a robust infrastructure to process refunds, and TMB wrote most of the code involving that. I am discussing with Matt what our options here are, but in the worst case scenario, any leftover will be donated to the EFF.

A final backup of the VMs and site database will be taken and soylentnews.org will be redirected to a static page. Everything representing the site be archived and taken offline. I'm going to hold the domain name and backups in trust in the hope that circumstances in the future may allow for the site to return in some form.

I wish I did not need to say such a thing might happen, but all things must end.

Until we meet again, ~ NCommander