SoylentNews is people
SoylentNews
So it's been awhile since I last wrote, and I'm a bit overdue for a status update. So, let me give you all the short version on what's been going on.
First, I've been doing a lot of backend work to drastically reduce the size of the SoylentNews bill month to month. We had a lot of infrastructure that was either unnecessary, or have gotten so many free tier upgrades that they were being vastly underutilized. Along the way, I've given a lot of fine tuning to bits, although I won't say its been problem free, since we went a few weeks without working sidebars. I'm truly sorry for the delays in getting up and running. My personal life chose to become very exciting in December, and I'm still dealing with the fallout of that entire mess. As such, what I had planned went a bit pear-shaped, and I went unexpectedly radio silent. ...
More past the break ...
The biggest problem is that most of the backend is undocumented. I wrote some documents in the early days of the site, but by and large, the site was mostly maintained by individuals who are no longer active on staff. The internal TechOps wiki was woefully out of date, and even I find myself struggling to know how the entire site is put together. Considering it's been online for over 9 years, and was a bit of a rush job out the gate, well, you know, it happens. I think at some point at the decade mark, I will want to chronicle more about SN's history, but let's first make sure we've got a site when we get there.
By and large, I'm not involved in the day to day operations. janrirok has been, and is, at this point the de facto project leader. My role with SoylentNews these days is kinda vague and undefined, since I stepped down privately in 2020, and then stepped back last November. I also find myself very uncertain if I want to even be involved at all, but, ultimately, I was here at the start, and while SoylentNews was always a collaborative project, I left a mark on both what this site is and will be that has persisted over the better part of a decade.
As such, I feel personally obligated to get SoylentNews to the best shape I can possibly get it, and give it the best chance of success I can give it. However, we're in the uncomfortable situation that we have a dated Perl codebase running on undocumented infrastructure that has been creaking along with no major reworks in almost all that time. You can imagine I've been having a fun time of this. Most of the relevant information mostly exists in my head, since I was the one who got Slashcode running all those years ago.
Right now, my biggest victory is I managed to get us off MySQL Cluster, and onto a more normal version of MySQL which drastically reduces memory and disk load in favor of slower load performance.
Moving forward, the solution is to have a reproducible deployment system, likely based around Docker, or possibly even Kubernetes, with all aspects of rehash (the site software) documented. We use GitHub to handle site development, and I think it would be in our best interests to integrate a full CI pipeline for both development and production environments. While implementing this, I also intend to entirely redo every aspect of the backend, complete with proper documentation, so something beside me can actually maintain it. After that, it will actually be practical for SoylentNews to survive past a single person, and we can have a more serious discussion on what the road forward looks like.
I do realize that the last few months have had a lot of ups and down, mixed with excitement and disappointment. I can't really say for sure where we're going, but you know? I want us to reach that decade mark together, and then we'll figure out where we're going beyond that.
Until next time,
~ N
(Score: 5, Insightful) by pTamok on Wednesday February 01, @07:32PM (3 children)
I wouldn't move from perl unless it is utterly broken. The first law of rural motor mechanics applies: "If it ain't broke, don't fix it."
It might be sub-optimal, but if it works, it has more going for it than unwritten and untested code. As you astutely point out, a higher priority is to document everything, so that you are no longer a SPOF. Rewrites and optimisation can come later.
There are many things that SoylentNews gets right: no javascript, and a moderation system that is pretty darn good. And no adverts!
Thank you again, and I hope the personal stuff is less overwhelming soon.
(Score: 3, Interesting) by DeathMonkey on Thursday February 02, @09:21PM (2 children)
Define "broken." If you can't find Perl developers that sounds pretty broken to me!
Every time this topic comes up there are several people saying they are willing to help but they just don't know the tech.
Having such an incredibly feature-stable product to work from does make platform upgrades easier. Maybe it's time to just copy this exact same design onto something people know how to work with?
(just a thought, great job however you're doing it!)
(Score: 1, Interesting) by Anonymous Coward on Friday February 03, @09:52AM
This project has three intersecting problems. 1) It is written in perl, which not a lot of people use and less use regularly everyday. 2) This website has a relatively small user base. 3) Some of the code and architecture is a beast and not the best to learn on. All three of those would make on-boarding difficult. There are things that can be done to make that easier, some of which NCommander is addressing, but it will be hard to overcome the big 3.
However, a total rewrite isn't exactly easy either. Before you begin, there is some serious questions to ask, such as "what language?" and "what architecture?" And I'm not sure those questions would even meet a consensus. Then you get the fun part of actually implementing it. The first 50% will probably fly by and feel great. But then you quickly start to realize why the 90-90 rule exists and that Hofstadter's law is always true. I'm not saying it can't be done, especially with a lot of existing work directly translatable, but there will be plenty of difficulty once you get to all the weird corner cases.
But if this site did decide on a rewrite, how sure could they even be that they would get developers for the new version? Two of the three main problems it has would still seem to apply.
(Score: 3, Interesting) by pTamok on Friday February 03, @10:16AM
There is no lack of Perl developers.
However, there is a lack of Perl developers willing to work for free on a website where thanks are in short supply, and where there is plenty of kibbitzing, and a range of politically sensitive opinions. The problem is not Perl. Any Turing-complete language could be used, the problem is in finding people with the inclination and resources to follow 'the job' through and not give up in disgust, while keeping the website and community going. I'm not that person, for a variety of reasons, even though SoylentNews is one of the very few places on the Internet I choose to contribute to.
I have a great deal of respect for the people that do provide time and resources to keep the place going, despite the difficulties. Managing technical debt while providing a continuous service is remarkably difficult.