2022-07-02 10:17:28 ..
2022-09-19 19:08:07 UTC
2022-09-26 12:53:59 UTC --fnord666
We always have a place for talented people, visit the Get Involved section on the wiki to see how you can make SoylentNews better.
There has been some discussion about moderation on this site leading to some misconceptions and misstatements. This story is an attempt to set things straight. It lays out the historical underpinnings for moderation, history of its implementation on Slashdot, and its later refinement on SoylentNews.
Before that, though, I am going to take this opportunity to thank fnord666 who is out Alternate Editor-in-Chief. I could not handle the load alone and his efforts have made a huge difference! Further, please join me in thanking him as he reached a new milestone: over 6,500 stories posted to the site! Many a late night or rare free moment has been generously given to the site. Teamwork++!
The code for this site is a fork of code written for Slashdot. In that site's early days, it was apparent that some comments were much more interesting and informative than others. It was just as apparent that some users would just as gleefully troll the community. Moderation was conceived as a way to sift the wheat from the chaff and help users more easily avoid the "lesser" comments and more easily find the "gems".
Further, to encourage posting "good" comments, Karma was introduced. "Good" comments earned Karma; "bad" comments lost Karma. Moderation was a mechanism by which Karma could be allocated.
Slashdot experimented with several ways to moderate comments. First, it was just the staff who could moderate. Soon, there were too many comments to keep up, so a select group of members from the community were invited to moderate comments. Again, that failed to scale up, so those who had been selected were invited to recommend still other users to moderate. And, again, there were scaling issues.
Solution: make Mod Points (modpoints) available to every registered user in good standing and who indicated in their preferences that they were willing to moderate.
Originally, mod points were handed out randomly and expired after something like 6 hours: "Use 'em or lose 'em".
For the most part, that seemed to work. But there were some perceived issues and meta-moderation was implemented and introduced — moderate the moderations. Unfortunately, it experienced many of the same issues that it was supposed to rectify with comments, just one level abstracted. Further, it was unwieldy and when all was said and done, didn't work all that well, anyway.
Such was the state of things when SoylentNews started. Well sort of. The code base we started with was not current and the meta-moderation code was broken. So much so, that meta-moderation was ripped out of the code just so regular moderation could be made to work. With that behind us, we finally we had a working moderation system on our site. Yay!
That worked okay for a while, but we found ourselves with complaints from many users that they wanted to moderate and lacked mod points. Nice problem to have, right? This was combined with many more comments than moderations. It was thought that we needed more mod points made available to the community. So, after unsuccessfully tweaking the mod point allocation algorithm, it was decided to just not expire mod points until day's end. Every user in good standing got 5 mod points each morning (00:10 UTC) and those were available until day's end whereupon any remaining modpoints were reset and a new set of 5 of modpoints were allocated.
That helped! But jerks will be jerks.
We started to run into problems with "mod bombs" where one user "A" would apply all 5 of their mod points to downmod one other user "B". So code was written to allow checking for such moderations. Staff could generate a report and find such activity. It was decided that:
If you used ALL of your modpoints to downmod ONE user, that was a modbomb. IOW, 5 downmods bad; 4 downmods were permitted.
Initially, anyone who "modbombed" was manually given a "timeout". The first time earned a one month suspension of moderation privileges. A second occurrence earned a six month suspension.
Later, because there were still many more comments than moderations, the number of modpoints allocated to each registered user having good Karma was increased from 5 to 10 per day. The modbomb threshold was, however, kept the same: 4 downmods was still okay, 5 (or more) downmods to the same user was "bad".
A complication arose in that there is no easy way for users to keep track of how many downmods they had made on one other user. User "A" may do 3 downmods of user "B" in the morning and 4 down mods of other (unrelated) users. In the afternoon they might perform 2 more downmods of user "B". Purely unintentional transgression. When you only have 5 mod points it was reasonable to assume that a user could mentally track how many times they downmodded a single user in one day. With 10 daily mod points available, that became less reasonable.
So, along with the allocation of 10 modpoints per day (easy) it was intended to have code written that would kick in when processing moderations: when the threshold was exceeded, the excess downmods would be automatically rejected. And that is still the intent.
The upshot of all that is that when checking for modbombs, we no longer give a "timeout" for 5 downmods against a single user in one day. We just revert the excess mods. We do take note of repeated excesses and are fully prepared to issue a "timeout" when warranted. (e.g. 8 downmods in one day, or several days in close proximity targeting the same user. This is not done unilaterally but rather in consultation with other staff for confirmation.)
First, there some who failed to take the hint that, maybe, they should take a look at what they were posting when they received repeated downmods. We are a community, not your personal soapbox. So, they created new ("sock puppet") accounts and proceeded to upmod their own comments, aka a "sockbomb". Staff have ways to note such behavior based on the IPID and SUBNETID that is recorded with every comment and every moderation. We try to give the benefit of the doubt. But, certain patterns do become apparent and are not tolerated. Upmodding your own comment is grounds for an immediate moderation ban.
Second, just as there is a limit on how many downmods can be targeted at one user in a day, so there is a limit on upmods. The same limits apply, each user "A" is limited to 4 upmods of user "B" in a given day, just like for "modbombs". Again with the caveat of no up-mods of your own account..
Our experience is that the current system could stand some refinement, automation of transgression detection and mitigation is in plan (but it will be a while), but for the most part, what we have works well in the vast majority of cases. In short, Wheaton's Law still applies: "don't be a dick". Following that seems to work the best for the most. (With apologies to anyone named Richard. =)
Last night (actually, very early this morning) mechanicjay generated and installed new Let's Encrypt certs for our servers.
I made a quick check and everything seems to be in place. The old certs were due to expire right about now, so if you do have any issues, please pop onto IRC (preferred) or reply here and let us know!
[2021-06-14 02:24:41 UTC; update 2: We made a decision to accept Linode's offer of moving up our migration of fluorine. It appears the migration has completed successfully. YAY!]
[2021-06-14 00:25:32 UTC; update 1: hydrogen appears to have successfully migrated. We had a brief 503 on the site until I bounced varnish. The site seems to be fine, now.--Bytram]
First off please accept my sincere wish for a happy Father's Day to all our dads in the community! (It is celebrated next Sunday in 90 countries.)
Also, I am happy to report a surge in participation on the site over the past month. I've seen increases in story submissions, subscriptions[*], and participation (comments, moderations, etc.) Community++
[*] NB: I was successful in crediting users for their subscriptions on the site after the server crash. Unfortunately, that failed to account for the dollar amount of their subscriptions in our tracking database table which is used to source our progress against our funding goal. I have a plan for getting those updates in place, but want to run it past other members of staff to make sure everything is accounted for before making any changes.
Read on for the rest of the site's news, or just wait and a new story will be out before too long.
We have received word that Linode, our web-hosting provider, will be conducting maintenance on two of our servers in the next 24 hours.
Last night Linode shut down one of our servers (boron), migrated the disk image to a new physical server, and restarted it. All seems to have gone smoothly.
Later on today, two more of our servers are due to be migrated:
Also of note, we are eligible for a free storage upgrade on fluorine from 96 GB to 160 GB. It is not clear at this moment if we will also conduct the storage upgrade at this time.
We are aware and intend to have updated certs installed before then.
(NB: I may have some terminology errors in what follows, but I believe the overall process/concepts should be correct.)
I have personally installed updated certs twice before on our servers, and if need be, am prepared to do so again. It has been a couple years or so but the process should remain largely the same. The majority of the steps are automated, but historically we've preferred to handle the DNS updates manually. That way, just in case something goes sideways, we are hands-on and can take steps to mitigate problems... instead of finding we have a botched DNS and greatly restricted access the servers. (That is a bit of an overstatement, but as I understand it, it's a lot easier to make changes over SSH connections to running servers than through a console port to one server at a time.)
Also, there has been discussion about using a fully-automated Let's Encrypt cert update process, we'll keep you posted.
Behind the scenes we've been hard at work. juggs, mechanicjay, and audioguy have put in many long and thankless hours stabilizing and documenting our service infrastructure. They've made great strides and we continue to make progress. We cannot change what was done (and not done) in the past, but we can learn from it! What services "live" on what servers? How to restart each service? Monitoring of disk usage and CPU usage? All are gradually being documented and site operations knowledge is getting shared all around.
Lastly, here's a shout-out to the editorial staff who strive to keep stories coming to you 24/7. Fnord666 just posted his 6,500th story! Also, thanks to janrinok, mrpg, chromas, and FatPhil who have all pushed out stories this past month! Teamwork++!
[N.B. Let's not forget our Editor-In-Chief martyb, who just posted his 10,100th story! This is in addition to serving as our primary QA person. - Fnord]
As many of you noticed, we had a site crash today. From around 1300 until 2200 UTC (2021-05-20).
A HUGE thank you goes to mechanicjay who spent the whole time trying to get our ndb (cluster) working again. It's an uncommon configuration, which made recovery especially challenging... there's just not a lot of documentation about it on the web.
I reached out and got hold of The Mighty Buzzard on the phone. Then put him in touch with mechanicjay who got us back up and running using backups.
Unfortunately, we had to go way back until April 14 to get a working backup. (I don't know all the details, but it appears something went sideways on neon).
We're all wiped out right now. When we have rested and had a chance to discuss things, we'll post an update.
In the meantime, please join me in thanking mechanicjay and TMB for all they did to get us up and running again!
We had a few hours this evening where there were issues we didn't notice with new submissions. We had been mothballing (setting the number of matches necessary beyond what would ever match anything) some spam filters exclusive to submissions and managed to somehow confuse one of the Apache web frontends. The offending filters are now just deleted entirely and incapable of confusing even MS Paint.
Please excuse the embuggerance. We now return you to your regularly scheduled arguments.