SoylentNews

That is just as true today as it was then. We are in a similar position to that in which we found ourselves 9 years ago. Software is having to be rewritten and repackaged. It is still based on software written 10 years before we even went live. It works but has a specific set of requirements that are becoming increasingly more difficult to meet. Hardware is being restructured necessitated by the changing economic circumstances that we as a site, individuals, and a community are having to learn to live with.

To say that it has not all been smooth sailing would be an understatement. The world and our community have changed far more than anyone could have envisaged in those first few weeks and months. The site has had to change also - sometimes to enhance what we offer each other as a community and at other times to prevent all of the good work being undone by a small number of people, one result of their actions being to drive away a significant proportion of our community. But here we are - a bit battered and bruised but preparing to move forward while meeting both legal requirements and community expectations as we do so.

Being in the same situation that we found ourselves in 2014 means that NCommander's words are just as relevant today as they were back then. We rely on the community to help us in exactly the same way. We would like to rebuild the community and we can do that with your help by word of mouth. We still rely on your submissions, comments and moderations so please continue to make them. And we are also in need of people who are willing to give a small amount of their time to a wide range of tasks that are necessary to keep the site operational. There is no specific commitment of time or effort - just to contribute whatever you are comfortable with. Get in touch and let us know what interests you, and we will look for a role that will meet those interests wherever possible. If we find a match then we will both benefit but if not there is no commitment on you to join the team.

As I write this we are trying to introduce 2 new volunteers to the sys-ops team, and possibly another editor in the next few weeks - more on that once they have found their seats!

However, if nothing else, please keep doing what you have been doing to support the site - your comments and views are what makes this site what it is. The site is nothing without the community.

Jan

The biggest problem is that most of the backend is undocumented. I wrote some documents in the early days of the site, but by and large, the site was mostly maintained by individuals who are no longer active on staff. The internal TechOps wiki was woefully out of date, and even I find myself struggling to know how the entire site is put together. Considering it's been online for over 9 years, and was a bit of a rush job out the gate, well, you know, it happens. I think at some point at the decade mark, I will want to chronicle more about SN's history, but let's first make sure we've got a site when we get there.

By and large, I'm not involved in the day to day operations. janrirok has been, and is, at this point the de facto project leader. My role with SoylentNews these days is kinda vague and undefined, since I stepped down privately in 2020, and then stepped back last November. I also find myself very uncertain if I want to even be involved at all, but, ultimately, I was here at the start, and while SoylentNews was always a collaborative project, I left a mark on both what this site is and will be that has persisted over the better part of a decade.

As such, I feel personally obligated to get SoylentNews to the best shape I can possibly get it, and give it the best chance of success I can give it. However, we're in the uncomfortable situation that we have a dated Perl codebase running on undocumented infrastructure that has been creaking along with no major reworks in almost all that time. You can imagine I've been having a fun time of this. Most of the relevant information mostly exists in my head, since I was the one who got Slashcode running all those years ago.

Right now, my biggest victory is I managed to get us off MySQL Cluster, and onto a more normal version of MySQL which drastically reduces memory and disk load in favor of slower load performance.

Moving forward, the solution is to have a reproducible deployment system, likely based around Docker, or possibly even Kubernetes, with all aspects of rehash (the site software) documented. We use GitHub to handle site development, and I think it would be in our best interests to integrate a full CI pipeline for both development and production environments. While implementing this, I also intend to entirely redo every aspect of the backend, complete with proper documentation, so something beside me can actually maintain it. After that, it will actually be practical for SoylentNews to survive past a single person, and we can have a more serious discussion on what the road forward looks like.

I do realize that the last few months have had a lot of ups and down, mixed with excitement and disappointment. I can't really say for sure where we're going, but you know? I want us to reach that decade mark together, and then we'll figure out where we're going beyond that.

Until next time,

~ N

What's In A Name?

There are significant challenges happening behind the scenes that even I was unaware of until an hour or two ago. One is a problem with our domain name itself. You may recall that in the very initial stages of the site the domain was taken by Barrabas who then only relinquished it for the sum of several thousand dollars. I do not have the details but is appears that there is some form of new dispute regarding the name, which NCommander and another board member are currently trying to resolve. If I have understood it correctly, it seems that another entity is trying to grab the name but all that I have to go on are a couple of cryptic comments on IRC. If and when I have more information I will pass it on.

Addendum: After a discussion with another member of staff I believe that our own domain name is safe. NCommander is having an issue with a domain elsewhere. I will keep you informed.

Bugs

NCommander has a professional life away from this site (as do all the staff with myself being the only exception). There has been significant upheaval in his work over recent weeks and, as a result, he has not had the time that he had expected to have to work on the site. Others are contributing where they can but any resolution of the current problems is unlikely to happen before Christmas. He has also made commitments that he has to honour and he is burning the candle at both ends for the moment. At the end of the day he has to pay the rent and put food on the table. The site is an entirely voluntary and unpaid activity for all of us.

The three major bugs (lack of mod points, the failure to display new journal entries, and the search facility not finding recent stories) are all slashd issues. Some community members are currently scouring the logs/code to try to pinpoint the precise error so that, when NCommander does have some time, the necessary software changes can be carried out quickly.

Site Statistics

There has been a bit of a feeling of pessimism over the site recently - so I thought I would do some analysis of the figures to see exactly what the situation is.

Since September we have had a total of 465 unique accounts contributing with comments - and most contribute many times. To be honest that figure is higher than I thought it might be. There are more who have been moderating but not commenting. Figures for how many people actually read our site regularly are difficult to calculate accurately because we cannot count those who do not log in. But by comparing current active participants and page hits with historical data gives a rough estimate of a community of around 700-800 people but that potentially has a wide margin of error. There has also been a reappearance of quite a few accounts that have been dormant for the last couple of years who have returned to the site. I hope that this trend continues.

I was also concerned about the fall in the number of comments since non-logged-in AC posting was disabled on the main pages. Again, it turns out that it is significant but it is not as bad as the figures first suggest. When I account for the number of off-topic, troll, and down-modded posts the comment figures are lower than they were a few months ago, but only marginally so. The quality of discourse is much higher now and is, I think, more acceptable to community as a whole. I am not suggesting that ACs do not make a significant and valuable contribution to the discussions. In fact, it is clear that the majority of Anonymous Cowards do act sensibly and contribute just the same as most other members do. But to enable them to return to the main stories we have to have a way of differentiating between each AC rather than having to control them all in a single account. A recent proposal on how to achieve this was rejected so, for the time being, I am concentrating on other tasks.

New Accounts

It has been claimed that we are not accepting new accounts. This is simply untrue and it is a statement being repeatedly made by an Anonymous Coward. As at the time of writing we have a total of 21836 accounts, but over the last few weeks a 'fake account' bot has been running and has created well over a thousand account entries which simply do not exist. However, there have been 77 new accounts created recently which have been active on the site. These figures do not include the many sock-puppet accounts that have been created; over 600 have been identified and disabled. As far as I can tell, and contrary again to many AC claims, only 3 or 4 of them ever belonged to Runaway1956 and none have been active since last year.

Subscriptions

Without a doubt one of our biggest problems currently is funding. NCommander has managed to reduce our server count significantly and has therefore reduced our running costs. People everywhere are struggling to make ends meet and many have to make a difficult decisions regarding how to spend their money. If you are able to make a subscription - no matter how big or how small - it would be appreciated by the entire community.

Holiday Period

Over the holiday period we will be reducing the story output to 5 stories a day (the 'weekend' rate). I will be travelling to visit my own family many of whom I have not seen since the start of the Covid pandemic. I will only have periodic internet access, and we will try very hard to find current newsworthy stories to publish. Good submissions are always welcome and we try to select those which have a submitter's identity over those that are created by some of our alternative collection bots.

I read somewhere that there are actually 11 different holidays being celebrated around this time of year. So on behalf of all of the team here at SoylentNews may I wish you, your family, and those close to you, the very best wishes for the holiday period and into the New Year.

State of the Backend

Right now, the backend is mostly built on an outdated version of mod_perl 2.2, and MySQL cluster, which is very much not a good place to be. Originally as envisioned, I planned this site to be able to be easily scalable, with a larger user base. That's why the infrastructure was designed to be as scalable as it was, with the downside of having a much higher overhead than a more traditional setup has. Furthermore, rehash (the code that powers this site) is, uh, to put it frankly, a beast to work on. It's a 90s era Perl code base and pretty much everything else that implies; if it wasn't for the fact that rehash is one of the main reasons to use SoylentNews, I'd argue it might be time to replace it.

Right now, I'm working on doing another round of server hardening. As it is at the moment, I've got rehash and Apache running in an AppArmor jail, and everything is pretty well sandboxed from everything else, but I still need to go through and adjust a lot of firewalls, and finish decommissioning out a bunch of the boxes. That said, the site is running faster than it has in a long while since a lot of small things got corrected as we went. Sometime this weekend, I'm going to finish adjusting the firewalls to lock it down further, and that should mostly get back to the point where I might have restful sleep again. That being said, there's still a fair bit more to do.

Moving ahead, we need to get off MySQL cluster, and either onto the current mod_perl, or, ideally, FastCGI, to end the Apache dependency entirely. Unfortunately, working on Rehash is quite difficult, and it requires a very specific setup to be viable. My current plan here is to basically get it working in Docker, so its easy to spin up and spin down instances, and return to a less cursed variant of MySQL. This is probably a few hours of work, but I'm hoping that overall it is going to be easy and straightforward to do since most of the backend is fairly well documented at this point. This also leaves me in a decent position to implement a couple of long overdue features, but modernization efforts come first. I'm hoping to livestream my efforts on this on the weeks to come, and I will make stream announcements as I go along.

Policy and Code Changes

My intent, based off the policy changes that were made to disallow ACs to post on stories is to sunlight the feature entirely, including in journals and more. The decision to have ACs on SoylentNews was made in 2014, when the Snowden leaks were only a few months old. Furthermore, we've seen from experience that the karma system doesn't go far enough at keeping bad actors from still getting a +2 status. By and large, the numbers underpinning the system need a rework. My general thought is to cap karma at either 10 or 15, and drastically decrease how far into the basement you can go, as well as uncapping posts in moderation to be able to go to -5.

As a rule, incredibly bad takes do get moderated out of existence, but because there's no real penalty for doing so, we get constant shitposts. Time to make this a bit harder to abuse. I've documented the antispam measures on the site before, but the site keeps track of IP addresses and subnets in the form of hashed /24, and /16s (/64 and /48 for IPv6), which has a karma number attached to them. If an IP range goes too far into the basement, it ends up posting at 0 or -1. By adjusting the caps, it should allow this threshold to be reached much more easily, and help bring the signal to noise ratio back to something more "positive".

Furthermore, I believe its generally in the site's interests to allow editors to delete comments. This functionality is actually built into rehash, but has been long disabled. At the time, I felt the community was best self-moderating, but I think on the whole, its better to treat this like a moderated subreddit, and have messages get a notice that they've in-fact been deleted ala reddit. This is a fairly large departure for the site as a whole, but I think one justified given the state of the Internet on 2022. I am open to discussions on all of this, but let me see what all your thoughts are like.

Final Notes

I do intend to keep livestreaming my progress with the site as we go along; and we raised another ~500 dollars towards Trevor Project during the last livestream. I've left that stream unlisted until I've had a chance to finish implementing all the hardening measures I've discussed, but I'm hoping at the end of it, I'll have a pretty good documentary on what it takes to modernize an aging website. As usual, if you want to support me directly: Ko-fi is available for one time donations, or Patreon for a recurring donation.

~ NCommander

[ If you are an AC and wish to make a constructive comment, please see my journal. janrinok ]