SoylentNews is people
SoylentNews
I'm going to simply write this quickly now. I have had very long discussions with a member of the community known as kolie who has been negotiating to try and keep SN operational, and help provide a realistic plan for both rebuilding the site, and migration. I was approached after the shutdown post was put up via public contact information. He has offered help in the form of hosting, capital, and helping coding a replacement for rehash. He has convinced me that there are enough people in the community that it might be possible to pay down the technical debt.
I was asked to formally take the gun off SN's head, since it doesn't help recruit volunteers if there's a death sentence.
I am more than a little reluctance to do this, simply on the basis that there has been a long history on this site of saying "we'll do X", and then X never happens. The situation was also discussed prior with Matt, and quite a few other people before I finally made the decision after it became clear to me that the situation had become completely untenable. I spent weeks looking for an alternative before I finally resided myself that there were no other viable options. But sometimes you can be wrong, and sometimes you can get outside help.
One of my cited reasons for shutting down SN was that calls for help were left unanswered. However, said call finally got answered and came at the 11th hour, and as an unsolicited DM by someone who wanted to see the site go on. We have been discussing this at length since Monday, in a conversation that at this point has been longer than everything said in a private, staff channel for the last six months. So, I accept the possibility I can be wrong. More specifically, I hope I am wrong.
So, ultimately, I will put my faith in someone I have never met before. It might be absurd sounding, but that is ultimately how SN started. A bunch of people who never met coming together to make a replacement for Slashdot. I will take steps to keep SN going past the 30th. This may involve the legal entity changing, as the PBC already voted to dissolve itself. I will write more on this next week, since frankly, I need time to sit back and reflect. I also need to write some emails.
The staff have told me that they will not work with me going forward. For my part, the feeling is mutual.
There are also the facts that I listed in the shutdown letter. SN's codebase is effectively unmaintained since the departure of TMB. I've already discussed the state of infrastructure to death, but there's an objective truth here: SN's VMs were exposed to the open Internet on end-of-life operating systems for years and the database cluster had been in an extended failure with corrupted log tables. As I see it, the staff allowed SN to degrade to the point that it was about to entirely fail. As I understand it, they see me as acting rashly and irresponsibly in attempting to address the situation. I freely admitted I could have done better.
At the end of the day, the only worse outcome than a volunteer shutdown is one where the site is either compromised, or lost in a crash. SN was one hard shutdown from an unrecoverable cluster failure.
That is not a viable state of affairs. That is a liability nightmare that at the end of the day the PBC is responsible for, which was the basis on which I intervened.
Finally, I'm only still here because SN has never been able to accommodate people leaving, especially as no one has historically been willing to take over legal responsibility for the operation of the site. I resigned three years ago. SN needs actual governance by people who can ultimately say that Z, Y, and X need to be done, and have the ability to either have it done, or can help raise the money to help get it done.
So, I guess we'll see if miracles happen twice.
~ NCommander
(Score: 5, Interesting) by janrinok on Saturday May 27 2023, @07:05AM (6 children)
You wrote in an earlier statement than you would le the community decide if what you did was right. I am a member of out community. But I hold an almost unique position; firstly, I am on the inside and can see what is being said, written and done, and secondly, I am not directly involved with the sys-admin side of the house. I have stood and watched things develop over the last 6 months. My view of what has just happened has some significant differences to what you write here.
First, let me acknowledge what you have achieved. You have updated the software so that it is more secure and maintainable. You have fixed, apparently, the corruption of the database (more on that later). You have significantly reduced the monthly cost of the hardware that we run such that we have saved around 50% of our costs. These are all worthwhile achievements and I am very pleased that you have done them. Thank you.
Before I continue, I would like to leave a few quotes here that you have recently written. I will put them here because I will refer to them several times in what will follow.
Communication
You have acknowledged elsewhere that, within a team such as ours, communication is key. So I have a few points to make on the matter:
Who did you prepare to do any supporting actions that your work would require? If we look back at the quotations that I have presented the answer is clear - no-one. There was no communication of any significance before you started. The start deadline was set by a live stream that you had promised to do on Youtube, which was also to raise money for a charity that you were supporting. Not for the benefit of SN, but for something admirable but of a private interest to you.
How did you give advance warning to the community about the risks of loss of data, or where to seek information if the site and/or our own IRC were down? This might get a bit repetitive, but it is the same answer. You simply didn't. You knew what you were planning to do but you felt that everyone else would just know too.
Planning
Where is your plan of action? I have not seen a written plan of action anywhere. If I had I would probably given the advice that perhaps you should rethink it, but we will cover that in more detail down the page.
Who did you discuss your plan with? You mentioned that you had briefly spoken to mechanicjay. I had also been speaking to him. We were aware that there were problems in the database. Like you and everyone else on this site, we have lives away from SoylentNews. The same is true of mechanicJay. He was trying to identify the problem and its solution so that it could be rectified and also to find the time to do that task. He was correct in saying that he wasn't 'doing' anything particular at the time you asked him. But I would rather have someone who thinks before acting.
Whose advice did you seek? Again, the answer appears to be no-one.
Execution
Backups. Every schoolboy who works on computers knows that you should backup your data regularly, and particularly before working on a computer system. You have mentioned (in the quotes) that you couldn't find any working backups. Why, where had you put them? The answer is simple. You didn't make any any, but then you blamed other staff for not having done what you yourself should have done.
Working on the Live system. Another common rule is that you do not work on the live system. You should have started with Dev - that is what it is there for. Once you had that up and running as a fully fledged server, you could swap them over and update the real server. You would have only affected part of the site, and you would have discovered any problems with the the database at that point.
Loss of connectivity. You seem completely oblivious to the fact that you have stopped some members of staff and the community from accessing the site. We have lost community members. When you updated the software it automatically updated the encryption algorithms. This is normally a good thing, but many who connect to the site using cell/mobile phones don't receive much support from their provider. They do not have the latest encryption. I know this because TMB had discovered the same problem several years ago and he had to restore the old algorithms alongside the newer ones. This would have become apparent during the testing phase - but you didn't have one in your action plan, or even any action plan.
Recovery
Where is your recovery plan? I know, it is with the action plan. But when things started getting sticky (around late December) you suddenly went missing. I know that you had real world problems to deal with - that is why a recovery plan is necessary. It tells everyone else what needs to be done to bring the site up to a working condition. But this became a 5 month period of insignificant progress and continuous excuses about why you would have to return later than the last promise that you made.
Where is the Dev server? The Dev server is an essential part of our system. We haven't got one any more. We cannot attempt to reproduce bugs so that we can identify what needs to be fixed. The community have lots of problems. Try searching for c0lo's inability to moderate anything - there is no obvious ban in place. We cannot access the wikis for training new staff, nor can the community get answers to many of their questions. When were you going to restore the Dev server?
Where are the wikis? When are you going to restore the wikis? This does not require money or outside help.
In what state did you leave IRC? We had been forced to switch to the Libera IRC node that we have for emergencies because our IRC was dead. I know that the updated software required a new version of IRC to be installed. Eventually, Deucalion stepped in. He hadn't been told in advance that you would need him to do the work. He had to search for the old files because you hadn't documented where you had put them. You don't want to consult anybody but you expect them to know what you will require of them.
Who do you expect to dig you out of this hole? The team that has been here from the very beginning along with some new staff are prepared to solve the problems that you have created. Just as they have done continuously since 2014. They haven't taken a couple of years away from the site. The sys-admins have been available on the end of a phone and have responded to everything we asked of them, subject to their own real-world obligations both professional and personal.
Documentation
Are the changes you have made fully documented? You have complained (again, it is in the quotes) that things aren't properly documented. I have looked. I cannot find your documentation. What have you changed? What service is on which server? I would normally look in the tech wiki - but that doesn't exist any more. So where have you put your documentation? Does our github repo reflect exactly what the Perl code contains?
Handover
I am pleased that kolie (who has a site nickname and a uid in the 2000s) has volunteered to replace you. I do not know him either. But he is enthusiastic and keen to do what is necessary. So why are you imposing a future plan of action upon him that you were not prepared to do yourself? You had decided to simply close the site down. Make a clean break and let him give it a go. He actually wants to do what you are asking of him. He may be content to accept your conditions - but he shouldn't have to. Stop pulling the strings.
Summary
This is NOT your personal train set. Being at the head of the tree implies more responsibility to do things properly and communicate your intentions, rather than less. We have a team and a community. Involve them both or end up with a considerable amount of egg on your face.
You made a commitment to the community to keep the site going until Feb 2024 at the earliest. You indicated that you were not gong to keep to that commitment. This is simply a trust issue.
You are undoubtedly intelligent and in some aspects regarding programming almost gifted. But you didn't think things through before you started. You dug this hole, and it is deep. The team is willing to try and fix it - when you have no longer any access to the hardware. There is still significant community support for this site despite what you have just done. We have been here before - we believe that we can do it again.
You have made a huge contribution to this site - indeed we would not be here today without that contribution. There is much that you should be proud of. I am genuinely grateful and when I say 'Thank You' I mean it.
What has happened cannot be undone. It is a pity that what you have done recently, however well intentioned you thought it to be, was ill-conceived and badly executed. It has tarnished what you have previously achieved. However, I hope that you remain in the community. Drop by and let us know what you are doing. Let us know what other streams you are putting out - we might bring you a few more viewers.
Now, let us all get back together and start working together on making a better site and restoring the community.
I am not interested in knowing who people are or where they live. My interest starts and stops at our servers.
(Score: 2) by janrinok on Saturday May 27 2023, @07:27AM
Sorry for the typos - but it was early in the morning (around 06:30) when I started writing this and my brain/finger interface wasn't working too well. It seems that eyesight wasn't up to much either....
I am not interested in knowing who people are or where they live. My interest starts and stops at our servers.
(Score: 4, Insightful) by pkrasimirov on Saturday May 27 2023, @08:30AM (2 children)
Thank you too for all work you put for all of us! I didn't know about these problems above, thank you for sharing. Don't mind the typos, if any.
(Score: 4, Insightful) by janrinok on Saturday May 27 2023, @08:58AM (1 child)
You shouldn't have to know really. What you should see is a site running smoothly and keeping the stories flowing.
This has been a long time coming, but I bear no grudges. Everybody did what they thought was best - rightly or wrongly. But one must own one's mistakes - not start throwing the blame around to others who also think they are doing the best. Everybody makes mistakes, it is part of life.
I am not interested in knowing who people are or where they live. My interest starts and stops at our servers.
(Score: 3, Touché) by Reziac on Sunday May 28 2023, @02:34AM
Thank you for laying it out. Approximately what I'd deduced from the outside, but it's good to know, rather than wonder.
SN is a more complicated endeavor than most of us knew.
And there is no Alkibiades to come back and save us from ourselves.
(Score: 0) by Anonymous Coward on Saturday May 27 2023, @10:27AM
Needs to be moderated to leventy-leven or higher. Thank you, janrinok.
(Score: 4, Touché) by mechanicjay on Sunday May 28 2023, @12:35AM
My VMS box beat up your Windows box.