from the changing-of-the-guard dept.
As many of you have already read, John Barrabas resigned as head of SoylentNews, and I've taken over in his place. Many people who are not involved in Staff were likely blindsided on this, and the community itself deserves to have an understanding of the reasons and events leading up to this. This post exists to set the record straight.
I would like to make it clear, especially in hindsight, that the events leading up to this were not pretty, and that no one involved came out smelling like roses. Mistakes were made all around, tempers were lost, and to be frank, at times, I've conducted myself in a way that was not professional.
In the end, the changeover was amicable, and John and I are still on speaking terms with each other. This isn't intended as a bashfest, but rather as explanation to the community (along with those staff who were not directly involved) of why and how this change came about.
NCommander Adds: Staff logs and copies of the email have been posted to my journal. Links included below.
At this point, several people had already been experimenting with the last public release of Slashcode, which was hosted in a semi-broken git repository on SourceForge. A couple of independent efforts had managed to get a slightly working Slashcode setup working. However, these instances were broken to the point that we got a bunch of jumbled HTML, and nothing resembling how we did during the private alpha. I offered to provide my knowledge and expertise in trying to get this mess working, while others looked at the possibility of a clean rewrite. In this I had the advantage of knowing where potential problems lay from a previous attempt at running Slash, and my experience at other Slashcode sites, such as Macslash.
After many hours of hacking, I was able get "install-slashcode" to run through with no errors, and bend the theming engine into something resembling a functioning pile of perl. The initial screenshot of Slash that was posted to the wiki was from this very early dev instance. However, the system we were on was a relatively dated instance of CentOS 5. Slash has somewhat unique running requirements, and is something of a four-letter-word to actually get started, let alone working.
Once it became clear that we could get Slash working, our attention turned towards figuring out what our production environment would look like. John began to solicit names for the site, and SoylentNews was picked out of those submissions. In hindsight, this was where we began to run into problems.
Originally, the plan was to be hosted on Bluehost (since that's where the wiki was), and work was done in parallel to try and get Slash up and running while myself and others would try and continue to bend slash to our wills. It became immediately clear that BlueHost wouldn't meet our needs (the initial attempt at installation hosed the VPS to the point it required re-imaging). Several other VPSes were tried (at once) and were unsatisfactory. This is in part why our initial bringup costs were so high.
Around this time, our dev system went and vanished on us, and its owner was unreachable. At this point, I was pretty frustrated with the state of things, so I set out to solve the problem. I had been a Linode customer for sometime at this point, and I knew for it fact it could meet Slash's demands. Without consulting anyone, I broke out my credit card, purchased two Linode 2048s, imaged them with Ubuntu 12.04, and proceeded to build Slashcode's dependencies, configuring them for Slash's particular needs. Furthermore, I was determined that to the extent possible, we would run on modern software.
My choice of Ubuntu for an operating system wasn't a matter of favoritism, but of practicality. I am an active Ubuntu Core Developer, and I knew the fundamentals of the system well enough to make sure we could setup Slash, and not break it by routine updates to the system. With these steps, the bring-up plan was moved from an unorganized bunch of people, and into a dictatorship. While this was not my preference, it was what was necessary to get us out the door.
As we got Slash closer-and-closer to usable, I started assigning tasks, and we got things done. We began to open the pre-launch site to more and more people, and an informal goal was to launch by the end of the Slashcott. An interim moderation algorithm was written, the missing parts of the theme were either recovered from CVS, or rewritten from scratch. Login and account creation was repaired, memcache support was fixed, and I located the last remaining problems with varnish and got that fully operational. We worked well as a team, and on February 16th, we opened our doors to the public.
What I didn't know at the time was that my take charge attitude had ruffled some feathers internally. Furthermore, I had made some decisions on how our development site should work, and that further annoyed people. After launch, John and I had a long one-on-one talk on how he saw things, and that he had gotten some complaints on my conduct. I will readily admit that I am not an experienced manager, nor do I have any formal management training, and conveyed that to John at the time. In the end, I stepped to the side, and John became the formal head of the site, while I continued to head-up dev. Furthermore, the stress and time put during golive left me very close to burn out. With the site up and running, I authored our end of day one post, and proceeded to rest for a week.
Under John, the staff was organized into various teams who in turn were managed by "Overlords," and the staff was loosely organized into this. Quoting from the wiki:
We're have overlords that manage various features. An overlord is responsible for granting access - it's intended to be a no work position, so that it can be held for long periods without requiring much time.
So for example, Applesmasher is the overlord of forums. He grants access to people and recovers access when people leave, but he doesn't have to do any work himself (although he can if he wants). He ensures that the people with access are reliable.
If you need a forum for your group, ask the overlord of forums. If you want a set of wiki pages, ask the overlord of the wiki, and so on.
The current overlords originally agreed to hold the position until Mar 1, that date has already passed. Going forward we can choose overlords via some formal process. (And the current overlords might ask to continue.)
John didn't wish to micro-manage, and felt that everything should be run by consensus. Decisions would be made by the group, and only by the group, but the saying goes, "Ask ten engineers for an opinion, get ten opinions." It quickly became apparent that we couldn't function as a cohesive unit. An Overlord had no authority to actually settle things in case of disagreement, and we had no way of resolving inter-team disputes. I'm not sure who coined it originally, but the term "management silos" quickly became common in describing the problem.
This was further compounded by an inability to communicate efficiently. We had no staff-wide mailing lists, nor a formal list of who was even on Staff (one of the things we're trying to determine post-handover). The only reason we even have staff email addresses is due to mrcoolbp collecting them all independently. mattie_p was made a "manager of everything" in an attempt to try and solve our communication issues, but even he had no authority to actually do anything, and had to defer to John on any matter of import. Staff morale was quickly sinking. In addition, we had far too many disparate venues of communication with little or no integration between them. Besides IRC, we had forums, the wiki, this site, a journal or two, and probably other ways that were setup in an attempt to address this issue, and some of us were only on one of all those methods.
Despite all of this, some teams were able to more or less run. As the Overlord of dev, I was able to build a group of good and active contributors, and managed to build the current dev VM which is architecturally similar to the production boxes. It should be noted that during bringup (as a factual error pointed out to me during drafting of this post), we were working on a CentOS based VM whose host committed suicide from load.
The editorial team was similarly able to run under these constraints, but only just barely. No one was happy with it. It was for instance, impossible to define a general "style and formatting" guideline, which is why the formatting of stories have been somewhat inconsistent.
Finally, many on the staff (myself included) felt conflicted on how some decisions were being made. For instance, John had us setup our own IRCd instead of remaining on Freenode, and to date, we've yet to have a public referendum on what the site name should be, as had been promised prior to launch. We launched under the premise that we (the staff) need to be a part of a community, and decisions impacting everyone needs to be discussed with the world at large. I know we lost people when we moved to SoylentIRC, and many of us never understood why we moved.
Issues started coming to a head about two weeks ago, due to an internal dispute on the operating system of choice for being run on what would be the final production systems (to date, we're still on the two Linode 2048s I setup at launch), and what the development systems would be. The decision was made without consulting all of the sys team, nor were other teams such as dev consulted at all (and as a member of both teams, I at the very least would have liked to been informed). I will not rehash this argument publicly, as it is only tangentially related to what ultimately happened. At this point, I grabbed John, and had another long one-on-one about both this, and the fact that we had a fundamental communication problem.
What happened was that I was slapped hard, my concerns dismissed as upset over not getting my way, and the communication issue was not addresses. Now, I'm going to be blunt. I nearly walked right then and there, and I fired off a reply that in hindsight I'm not proud of. My problem wasn't due the choice of OS per-se, but that a decision made by essentially one individual, and then backed even after the lack of discussion was brought to light. It wasn't clear that even within the sys team it had majority support (ultimately, I forced a vote on this matter at the staff meeting, and it was 2 for, 2 against).
Due to (pre-planned) traveling to Macau for a conference, there was little I could do at the time, but I was convinced that John needed to be forced to see what was going on. Failing that, a vote of no-confidence would be needed, since I felt that unless something gave, SoylentNews would fail due to mismanagement.
Now, mattie_p had managed to convince John to host an all-hands staff meeting for that Sunday. Although I debating calling for a no-confidence vote at the time, I ultimately settled on standing my soapbox, and forcing the communication issue into the staff as a whole, highlighting the problems with our system, and that the distro issue had prevented us from even setting up a development cluster. The dev VM exists mostly so I would have a way to test changes without unleashing them to the public before even being able to smoke test them. None of these issues were resolved in the course of the meeting.
In the days that followed, I finally managed to recover from jetlag, and starting speaking to staff 1:1, to get their feeling on things. At roughly the same time, John's QA went live. While much of the comments were positive, the staff in general felt ...
Well, felt that it was a bunch of hot air and marketing speak. Or at least that's how I'd put it. None of us had seen the answers beforehand aside from Mattie, and most of us were hoping for more substantive material. I, for one, was hoping that we were close to getting the not-for-profit setup, or at least get a discussion framed around that with concrete information.
I began going around to the staff in turn one-on-one, to try and get a feeling for the site, to see how things are, and what they're feelings on John were. For the most part, there was a sense of "gloom."
Ultimately, it boiled down to three major points:
- For the most part, no one was really happy with our communication issues, and that they were hamstringing us everywhere.
- No one felt that they could say anything or complain, especially since John was financing the site.
- We had no rudder, no visible vision we could work towards, especially since it wasn't clear what John was doing after almost a solid month since we began the project. The QA was just a slap in the face in that light.
With each person, I worked toward collecting support for a no-confidence vote. My plan was that at the next staff meeting, I'd ask John to step aside, and show that our issues had not been addressed, that we were choking on ourself, and that we essentially were becoming everything we left the other site to escape. Should diplomacy fail, I intended to force the issue via a vote of no-confidence, and let the chips fall where they may.
Furthermore, five of us were going to resign should the vote fail. I had talked to roughly half the staff and had already secured a majority, but wanted to make sure everyone knew what was coming, and had a chance to voice concerns BEFORE any changes in management. Fate intervened before that could happen.
John had formally gone on vacation for a few days, but we bumped each other in the staff chat, and myself and a few members of the staff who were up (this was approximately, 4 AM my local time) started began asking questions about progress of the not-for-profit setup (there was none) and seeking redress of the communications problem. What followed was one of the most disturbing chat sessions I've ever been apart of, but after going around on the major points several times, John conceded that he did not have the time necessary to run the site properly, and conceded it over to me.
As a direct result of this impromptu meeting, the head of the sys team resigned. Although no one had asked it of him, John authored a formal resignation in his journal a few hours later, which was re-posted to the main page.
Which brings us to now. As the final meeting was at a time where much of the staff was sleeping, many woke up and found that the world they left yesterday was fairly different than the world of today. There's been a lot of rumors and FUD flying around, but this writeup took considerable time to publish, as I feel that without the full story, people will always be questioning what happened here: was it an unwarranted power grab, was money involved, or was it something more? Furthermore, internally there has been a lot of uncertainty about recent events.
So, now what?
Well, now we move forward. For me, personally, writing this has been a catharsis, allowing me to deal with everything that has happened, and put it behind me. While staff morale has improved, I suspect the majority will need to come to terms with this in their own ways. None of us are happy with out this played out, or the fact that this was necessary (myself included). I'm inviting all to post below with their comments, staff to feel free to post their own version of events, and speak candidly with how things played out. As part of our commitment to open governance, this needs to be aired out and needs to be public.
There will be no reprisal from me for any members who wish to criticize me personally, or my handling of this entire sordid affair. Furthermore, I have an 'open-door' policy on both this and all matters, simply send me a PM or email with any concerns you may have. This has been a black page in our history, and we need to move forward as a group to find our future. For members of the staff who haven't seen the IRC logs or the email transcripts, please contact me in private for a copy.
However, I ask that the #staff logs and the emails remain private. While I do not personally have an issue with my own words (even the 'nuclear letter') going public, I do not have any desire to see John dragged through the mud, especially while he's not here to personally defend himself. Please keep it professional.
As for the site itself, I'll be personally shouldering all fiscal responsibilities until the point we are self-sufficient. For John's stake in this, I have discussed the matter privately with him, and I will compensate him of his costs once I return to the United States and am in a position to send him payment. He has mailed me all his written notes and various passwords which I'm in the process of evaluating.
I will have a formal write-up of our plans for the future in the next few days, as well as how the staff have been re-organized to prevent the management silos problem and communication issues we had before.
For the community who's been affected by the strife in staff, I can only offer you a humble apology, and strive to do better. I know that some have offered to volunteer to join our staff, and have either fallen through the cracks, or got lost in the recent strife. I know there are some who've wanted to join the dev team to whom I haven't been able to respond to properly due to recent events. If you're still interested, I ask that you get in touch with mrcoolbp on the Soylent IRC server.
So this brings me to what is likely your final question is, how do I plan to make it all right?
The answer is I plan to make sure we are and continue to be transparent about the goings-on that happen behind the scenes, and continue to practice full disclosure on matters involving our community. As for recent decisions, I plan to open the floor to get feedback and see if things need to be reversed, or improved.
For the staff, we've got a mailing list (finally) setup, and you should have received an email from mrcoolbp about it. The old organization has been discarded wholesale, and I've implemented a modified version of the Incident Command System. Without going into too many details here, the basic takeaway is that I am the chief officer, and that you either report to me, or to a line officer who acts as a liaison. No one person has more than 7-12 people reporting to them tops. If you have too many people, you make a separate group reporting to you. It is the responsibility of the line officers to try and discuss options within their group, and make the final call on any decision that matters. I'd like to have all staff (but especially the officers) in the staff take the free NIMS classes available online (its about 1-2 hours long self-study course) to learn the basics of ICS, understand how we're organized, and how we change to meet demands.
While its not a traditional management structure, it applies itself well to the "get it done" model, and is something I'm experienced in from my time as a firefighter. We're not a traditional organization, and it doesn't help us to think like it. I will have a more in-depth email on this topic drafted up in a few days, and posted to the wiki that goes into the deeper specifics.
Furthermore, I also plan to make sure that as an organization, we're as transparent as possible, with a guiding manifesto and mission plan, and regular updates (probably biweekly) as to our status. I plan to get the formal name discussion post rolling as quickly as possible once I have a moment to breathe, and make sure that we are made whole again. It might been a bumpy ride from here, but I'd like to think we're looking for a lot of smoother air in our future.
NCommander adds: John has posted comments from the IRC logs from the night of the discussion. In the interest of preventing anything from being removed from context, I've posted both the nuclear letter, and IRC transcripts from the night in question. I leave them here as a record of the truth, and I shall stand to be judged accordingly.
My one request is that people please remember that these were originally private conversations, posted to set the record fully straight. Tempers were flaring, and at least in the nuclear email, it had been prefixed with several days worth of calls and discussion, and where I ultimately lost my cool.
So, as I write this, day one has officially come to an end. I'm still somewhat in shock over it. Last night when I was editing the database to change over hostnames and such, I was thinking, man, it would be great if we got 100 regular users by tomorrow. Turns out I was wrong. By a factor of ten. Holy cow, people. I'm still in a state of disbelief, partially due to the epic turnout, but also because our very modest server hardware hasn't soiled itself from the influx (the numbers are, well, "impressive" is a way to put it). Anyway, I wanted to do a bit of a writeup of where we stand now, what works, and what doesn't. Check it out (and some raw numbers) after the break! Warning, it is a bit lengthy.
Based on your questions from a little over 10 days ago, we have our first batch of responses. I want to apologize for the delay, as everything has been moving so quickly that we're still getting our act together in so many ways. Remember that we were still in the very early stages a month ago, so bear with us in our growing pains. We chose a large number of questions, mostly highly rated ones from the community, with some additional questions that I felt deserved answers. Read on for answers from the site leadership.
I can no longer sustain the level of activity needed to run SoylentNews, and so I have resigned as leader.
In the end, I was literally begging people to step back and let the site "just run" for a few days while I caught up... alas, to no effect. Issues must be resolved "right here and right now" continuously throughout my day with no end in sight.
NCommander has taken over as project lead - he has access to everything from the registrar down. I understand he has a different vision for the project, which will soon be revealed.
I'm sorry to leave like this, but there's really nothing for it. I'm going back to my AI studies, which happen at a more leisurely pace.
(And yes, this was an amicable transfer of power. There's no hard feelings on my side.)
[Ed. Note: this is a word for word article based on Barrabas's journal entry.]