Stories
Slash Boxes
Comments

SoylentNews is people

Meta

Log In

Log In

Create Account  |  Retrieve Password


Site News

Join our Folding@Home team:
Main F@H site
Our team page


Funding Goal
For 6-month period:
2022-07-01 to 2022-12-31
(All amounts are estimated)
Base Goal:
$3500.00

Currently:
$438.92

12.5%

Covers transactions:
2022-07-02 10:17:28 ..
2022-10-05 12:33:58 UTC
(SPIDs: [1838..1866])
Last Update:
2022-10-05 14:04:11 UTC --fnord666

Support us: Subscribe Here
and buy SoylentNews Swag


We always have a place for talented people, visit the Get Involved section on the wiki to see how you can make SoylentNews better.

posted by martyb on Friday August 20 2021, @10:00AM   Printer-friendly

Milestone #1:
First off, please join me in congratulating janrinok in posting his 5,400th story! I can attest that it represents a tremendous commitment of time and effort, all freely given to the community. Thanks JR!

Milestone #2:
Secondly, we are a few days away from our team reaching 2.8 billion points towards Folding at Home. Official Team Stats and a more informative summary. As I write this, our team is currently ranked #392 in the world. Please be aware we are up against teams such as AWS, Google, Apple, Facebook, SAP, IBM, Dell, Oracle... you get the idea. Our top contributor is Runaway1956 who has been contributing about 2.5 million points per day. Barring any surprises, he is on track to reach 1 billion points by month's end. Way to go!

Moderation:
Lastly, I need to call the community's attention to some problems with moderations.

For the most part, things have been working out well! Considering the diverse viewpoints — and strong feelings about them — I'd say things are working amazingly well. There are some, however, who are prolific, vocal, strong-willed, and are trying to push their own agenda. They are likely to be unhappy with these changes. Until notified otherwise, feel free to moderate complaints about moderation as "-1 Offtopic" and just move on.

Effective Immediately:
For the benefit of the community who have been acting in good faith all along, staff will commence issuing moderation bans on accounts that have been acting unfairly. Each ban will have been discussed among staff and no unilateral action will be taken. If you receive a ban, it's because a majority of staff are in agreement that unfair moderations have been performed and needed to be dealt with.

Tools:
Staff have been developing tools and procedures for tracking moderation abuse for some time, and we are now reaching the stage where we can take action against offenders. It is necessary for these tools to provide a high degree of confidence to avoid false positives. We now have that confidence.

What's the Point?
Moderation is intended for the community to decrease the visibility of "poor" comments and to improve the visibility of "good" comments. Note the word "comments". The user who posted the comment should have no bearing on your moderation. (If it does, you are doing it wrong.) Express your disagreement either by presenting a counter argument in a new comment, or by using a "-1 Disagree" mod — that's what it's there for. As one staff member noted: "If you find almost everything another user posts objectionable, then moderation is not the solution — simply foe him, and set your preferences for foes to a suitably-low negative value."

Goal:
We have tried to provide the tools for the community to moderate itself. This has worked out well for the most part! Here's a big thank you to those who quietly go along and try to make SoylentNews a better place each day.

Sockpuppet Accounts:
These are hereby forbidden. If staff detects collusion in moderation, that is grounds for an immediate ban on moderation. Acct #1 and acct #2 both moderating acct #3 in lockstep? Ban. Acct #1 upmoding acct #2 and acct #2 upmodding acct #1 to boost karma? Ban.

The guidelines had previously stated that more than 4 downmods per day were forbidden, excess would be reversed. We have seen cases where user "A" has downmodded user "B" exactly 4 times per day for many days in a row. This demonstrates intentional action of one user against another user. Because such actions do have negative consequences on the visibility of user "B"'s posts and journals, they are effectively an act of censorship, and are strongly frowned upon; therefore staff reserves the right to take action up to an including a mod ban, at our sole discretion.

There are more instances; this list is NOT intended to be exhaustive.

Notice:
Think you can get away with something? Just because you have not seen us act so far, does not mean we have not noticed. If you persist, you're doing so at your own risk. Please do not come crying to us or the community when your schemes backfire — you have been warned. As mentioned earlier, complaints about moderation and moderation bans are off-topic and are to be moderated that way.

Summary:
Staff has access to daily reports and ad-hoc queries. We have been holding regular discussions about moderation abuse. We will continue to do so. Cases of perceived abuse are discussed and no action is taken unless there is unassailable evidence about the facts and substantial agreement about the consequences (thus avoiding unilateral action).

For future reference, here are the moderator guidelines at the time of this being written. They are undergoing review right now and will be revised to include the preceding.

This document attempts to explain the moderation system that lies underneath this implementation of Slashcode's vast comment section. It was originally written for Slashcode years ago, so the specifics of this moderation system are outlined here. Keep in mind that as this project grows, some aspects can change to better serve the community and improve the overall experience.

Contents

  1. Purpose
  2. Goals
  3. Who
  4. How
  5. Mod Bombs
  6. Sock Bombs
  7. Spam Mod
  8. Moderation Issues
  9. FAQ

Purpose

As you might have noticed, a site like this can get a lot of comments. Some are downright terrible; others are truly gems, and hundreds of comments can be hard to sift through.

The moderation system is designed to sort the gems and the crap from the steady stream of information that flows through the pipe. And wherever possible, it tries to make the readers of the site take on the responsibility.

The goal is that each reader will be able to read the the threshold they prefer. Select "-1" and you'll see trolls and possible wrongly-modded comments, try "5" and you'll see only the top-rated comments.

Goals

  1. Promote Quality, Discourage Crap.
  2. Make SoylentNews as readable as possible for as many people as possible.
  3. Do not require a huge amount of time from any single moderator.
  4. Do not allow any single moderator a 'reign of terror' -- no 'mod bombs'.

Who

We've set up a few simple rules for determining who is eligible to moderate.

  1. Logged In User If the system can't keep track, it won't work, so you gotta log in. Sorry if you're paranoid, but this system demands a certain level of accountability.
  2. Positive Contributors Slashcode tracks your "Karma" (see the FAQ). If you have non-negative Karma, this means you have posted more good comments than bad, and are eligible to moderate. This weeds out spam accounts.
  3. No Sockpuppet Accounts Accounts newer than 1 month are not eligible to moderate. This should keep sockpuppet accounts from immediately being a problem.

So the end result is a pool of eligible users that represent (hopefully) average, positive SoylentNews contributors.

How

Each day every eligible moderator is given 5 10 mod points to play with. Each comment they moderate deducts a point. When they run out of points, they are done moderating until 00:10 UTC when mod points are regenerated.

Moderation takes place by clicking the drop down list that appears next to comments, and selecting one of the adjectives like 'Flamebait' or 'Informative'. In general, bad words will reduce a comment's score by a single point (a 'down mod'), good words increase a comment's score by a single point (an 'up mod'). All comments are scored on an absolute scale from -1 to 5. Logged in users start at 1 (although this can vary from -1 to 2 based on their overall contribution to discussions) and anonymous users start at 0.

Moderators can participate in the same discussion as both a moderator and a poster. You are only prevented from modding your own posts.

Concentrate more on promoting than on demoting. The real goal here is to find the juicy good stuff and let others read it. Do not promote personal agendas. Do not let your opinions factor in. Try to be impartial about this. Simply disagreeing with a comment is not a valid reason to 'down mod' it. Likewise, agreeing with a comment is not a valid reason to 'up mod' it. The goal here is to share ideas. To sift through the haystack and find golden, shiny needles. And to keep the children who like to spam in check.

Mod Bombs

A 'mod bomb' is simply when a user, 'A', uses all 5 or more of their moderation points to 'down mod' comments posted by a single user, 'B'. Would you want someone who has a vendetta to use all a bunch of their mod points on your comments? It works both ways -- don't use all a bunch of your mod points on a single user. When this is detected, the account performing the moderation ('A') is given a 30-day 'time out' on moderating all moderations making up the mod bomb are reversed and the mod points are not returned. We would like to make the code automatically prevent a mod bomb from occurring, but this is not yet in place. The focus is on the quality of the comments on the site, not on who posts them. Remember that there are other users on the site who have mod points. If you have used all that are permitted, do not fret as someone else will likely come along later.

Sock Bombs

Much like a 'mod bomb', a 'sock bomb' is when a user, 'A', use 4 or more of their moderation points to 'up mod' comments posted by a single user, 'B'. (The name is taken from the idea of a "sockpuppet" account.) Again, our intention is to update the code to automatically prevent this from happening. We realize that this can happen unintentionally when, say, a subject-matter expert provides supporting information in comments to a story. Excess 'up mods' beyond 4 per day are subject to being reversed. A repeated pattern of user 'A' upmodding user 'B' may be subject to further action. In short, please do not try to 'game' the system.

Spam Mod

The spam moderation (spam mod) is to be used only on comments that genuinely qualify as spam. Spam is unsolicited advertisement, undesired and offtopic filth, or possibly illegal in general. Spam can come in many forms, but it differs from a troll comment in that it will have absolutely no substance, is completely undesired, is detrimental to the site, or worse.

The spam mod is special in that is removes 10 Karma points from the user that posted the comment. This mod is meant to combat spam and not to be used to punish commenters (when in doubt, don't use this mod). Our goal is to put a spammer in Karma Hell and for them to not be able to get out of it easily. As we do not want this used against non-spamers, we monitor all spam mods to make sure moderators are not abusing the spam mod. If we find a moderator that unfairly applied the spam mod, we remove the mod giving the poster back the Karma points, and the modder is banned from modding for one month. Further bans to the same modder add increasing amounts of ban time. If you inadvertently applied a spam mod, mail the admin and we will remove the spam mod without banning you. Even though we have updated the interface to physically separate the spam mod from the other mods, unintentional modding may still be an unfortunate occurrence.

Examples

If you are unsure of whether a comment is spam or not, don't use the spam mod. Here are some examples of spam:

  • Proper spam. Anything whose primary purpose is advertisement (unless somehow relevant to the discussion/article).
  • HOSTS/GNAA/etc... type posts. Recurring, useless annoyances we're all familiar with.
  • Posts so offtopic and lacking value to even be a troll that they can't be called anything else. See here, here or here for example.
  • Repeating the same thing over and over. This includes blockquoting entire comments without adding anything substantial to them.

Moderation Issues

If you see moderation abuse (mod bombing or spam mod), please mail the admin any comments (the cid link) showing the abuse. Alternatively, mention it on the main IRC channel. We will investigate and make amends if necessary. This also applies for users who display an atypical pattern of up or down mods against another user. Moderation abuse may result in the loss of ability to moderate for a time, or in some cases, permanently. (If you find almost everything another user posts objectionable, then moderation is not the solution - simply foe them, and set your preferences for foes to a suitably low negative value.

FAQ I just got moderator access, what do I do?

The fact that you are reading this document proves that you are already on the right track.

Why can't I moderate any more?

  • Do you still have any moderator points left?
  • You can't moderate your own posts.

What is a Good Comment? A Bad Comment?

  • Good Comments are insightful. You read them and are better off having read them. They add new information to a discussion. They are clear, hopefully well written, or maybe amusing. These are the gems we're looking for, and they deserve to be promoted. (Score: 2-5)
  • Average Comments might be slightly offtopic, but still might be worth reading. They might be redundant. They might be a 'Me Too' comment. They might say something painfully obvious. They don't detract from the discussion, but they don't necessarily significantly add to it. They are the comments that require the most attention from the moderators, and they also represent the bulk of the comments. (Score: 0-1)
  • Bad Comments are flamebait, incorrect, or have nothing to do with the article. Other examples: Ad Hominem, ridicule for others with different opinion (without backing it up with anything more tangible than strong words), repeats of something said 15 times already (read previous comments before you post), use of unnecessary foul language, some are hard to read or just don't make any sense. Basically they detract from the article they are attached to. (Score: -1)

What is Karma?

Karma is the sum of all moderation activity done to a user. Karma is used to determine eligibility for moderator status and can affect your comments starting score. Every new user starts with a Karma of 0, and as long as your Karma isn't negative you are eligible to become a moderator.

Why Don't I get my points back after I post in a discussion I moderated?

We've decided to allow a moderator to moderate in a discussion, and then comment afterward without undoing their moderation.

How can I improve my Karma?

10 tips for improving your Karma:

Post intelligently:
Interesting, insightful, thought provoking comments are rated higher on a fairly consistent basis.
Post calmly:
Nobody likes a flame war. In fact, more times than not the flamer gets burned much more than their target. "Flamebait" is hit quickly and consistently with "-1" by moderators. As the bumper sticker says... "Don't be a dick."
If you can't be deep, be funny:
If you don't have something to contribute to the discussion, some humor is welcome. Humor is lacking in our lives and will continue to be promoted. Remember though, what rips your sides out may be completely inane to somebody else.
Post early:
If an article has over a certain number of posts on it already, yours is less likely to be moderated. This is less likely both statistically (there are more to choose from) and due to positioning (as a moderator I have to actually find your post way at the end of a long list.)
Post often:
If you only post once a month you can expect your karma to remain low. Also, lively discussion in an open forum is what makes SoylentNews really "Rock the Casbah."
Stay on topic:
Off topic posts are slapped quickly and consistently with "-1" by moderators.
Be original:
Avoid being redundant and just repeating what has already been said. (Did I really just say that?) Yes, being moderated as "redundant" is worth "-1" to your post and your karma. Especially to be avoided are the "what he said" and "me too" posts.
Read it before you post:
Does it say what you really want it to say? Check your own spelling and grammar. Occasionally, a perfectly beneficial post is passed over by moderators because it is completely irrelevant to content feature. This is also a good approach to checking yourself for what you're really saying. Can't tell you the number of times I've stopped myself from saying the opposite of what I meant by checking my own s&g.
Log in as a registered user:
I know, this sounds obvious but, "Anonymous Coward" does not have a karma rating. You can't reap the perceived benefits of your own accidental brilliance if you post anonymously. Have pride in your work and take credit for it.
Read SoylentNews regularly:
You can't possibly contribute to the discussion if you're not in the room. Come to the party and play.

Original Submission

posted by martyb on Monday July 05 2021, @09:00PM   Printer-friendly
from the UPDATE++ dept.

Milestone:
I doubt many people noticed, but yesterday (2021-07-04) marked a significant milestone in SoylentNews' history:

Our Independence Day: We're Officially Incorporated!

I'm pleased to announce that as of today, our articles of incorporation have been accepted and signed off by the State of Delaware, and "SoylentNews PBC" is a licensed public benefit corporation, ready to accept business, effective today.

[...] I feel that its fitting to post this today on July 4th, Independence Day, given the mission and unique history of SN.

This transformed us from a collection of like-minded nerds into an official entity and which made it possible for us to (finally!) accept funding for the site! Yes, there have been some hiccups along the way, but this was a key step in giving us a solid foundation from which we could proceed.

Server Upgrade:
It is with pleasure that I can announce that one of our database front-ends, fluorine was updated on 2021-07-02. We had been running with 96 GB of storage. On this date, with Juggs (aka Deucalion) watching over things, I snapped a quick ad-hoc backup and mechanicjay pulled the trigger to upgrade us to 160 GB of storage!

It's not that we were pressed for space. Linode (our web hosting provider) had made a free storage upgrade available to us quite a while ago. We held off for reasons I do not entirely understand except for the classic: "If it's not broke, don't fix it!". Whatever the reason, that wait is over! MJ assured us that the site would stay up (our other front-end hydrogen would carry the load) without any issues. So, with some crossed fingers, the button was pushed, fluorine was taken down, its disk was upgraded, and the server was restarted... with nary a hiccup!

It was suggested that maybe this was not worthy of mention, but we strive to be open with the community. I felt it was critically important we keep the community informed as to our activities, so here we are!

Folding@Home:
Still plugging away, the SoylentNews Folding@Home (F@H) team is still plugging along. Facing competition from the likes of Microsoft Azure, Google, IBM, Oracle, and other huge multinationals, it is a great pleasure to announce our current status.

We are still ranked within the top 400 teams in the world — at last check, we have earned over 2⅔ billion points!

New Funding Period:
According to my estimates, we missed our $3,500.00 funding goal by just 44 ¢. There were some additional subscriptions that came in after midnight UTC[*]. Since then, we have already received $211.69 (6.0%) of our $3,500.00 goal for the next half.

[*] Where to draw the line? I am not a CPA, and these amounts are all unofficial estimates, so for our purposes here I'm using Midnight UTC as the cutoff.)


Original Submission

posted by martyb on Thursday July 01 2021, @03:21PM   Printer-friendly
from the Woo-Hoo! dept.

THANK YOU!

I'm calling it a success.

Our fundraising goal for the first half of 2021 was $3,500.00 — according to my calculations, we raised $3,499.56 or 99.987% of our goal!

We missed our goal by... $0.44, yes just 44 cents!

We received 15 subscriptions on 2021-06-30 alone. When we asked the community to support[*] the site, you did!

But wait; there's more! As of the time of this writing, here were 6 more subscriptions (which netted us $113.82) that came in after the 2021-06-30 23:49:49 UTC cutoff. These will be applied towards fundraising for the 2nd half of the year.

[*] Support is more than money. A big thank you to those who submit stories, write comments, moderate comments, and freely give their time to edit stories and support the systems which, together, make this a community!

Yes, we faced some challenges, but we've met them. And, through the adversity, I'd dare say it has pulled us together and made us stronger!

NB: I'll leave up the fundraising status (which show our) end-of-first-half totals for a few days, so everyone gets a chance to see them.

Lastly, I count it a privilege and distinct honor to serve the community as its Editor-in-Chief. Thank you for your encouragement and support!

Previously:
Site Fundraising Status -- Almost There -- Today is Last Day of 2021H1 [UPDATES: 4]


Original Submission

posted by martyb on Wednesday June 30 2021, @11:09PM   Printer-friendly
from the We're-thiiiiiiiis-close! dept.

[2021-07-01 01:33:11 UTC; UPDATE 4]:

It came down to the wire, but at we ended up $0.44 shy of our $3,500.00 goal for the first half of the year.

Great job everybody!

[2021-06-30 23:05:04 UTC; UPDATE 3]
We are SO CLOSE!
We are currently at 99.1% ($3,470.73) of our goal! We need just about $30!

I've opened up the "Stretch Goal" field in the "Site News" slashbox in preparation.

We have less than an hour to go... Let's do this!


[2021-06-30 18:40:09 UTC; UPDATE 2]:
Since the last update we received SIX $20 subscriptions.
We are currently at 96.5% ($3,377.75) of our goal and need only $122.25 to reach it!

Books close at 23:59:59 UTC tonight... we can do it!


[2021-06-30 14:12:37 UTC; UPDATE 1]:
WOW! Since this story was posted, we have received TWO $100 subscriptions!
We are now at 93.2% ($3263.63) of our goal; only $236.37 to go!


Today (June 30, 2021) is the last day of the first half of 2021!

Fundraising Status:
First off, a great big THANK YOU to everyone who has started a new subscription or renewed their subscription this year!

As of this writing, we have received approximately $3,028.57 since 2021-01-01 towards our goal of $3,500.00 for the first half of the year!

We need only $471.43 to meet our goal!
We are so close! Please help us.

How to Subscribe:

Load the Subscription page and then:

select a recipient (defaults to your own logged-in account),
select a duration,
enter an amount (the values provided are suggested minimums; feel free to override with a larger amount)
click the [Continue] button which will open a new page.

NOTE: All payment processing is handled external to SoylentNews! You'll need to enable PayPal or Stripe for processing to work!

Then just click on button for the payment processor and payment type you want.

Last step is simple; a new form is displayed by the selected payment processor, enter the requested card information, and follow their instructions. That's it!

Recent Subscriptions:
In just the past month we've received these gross subscription amounts:

AMOUNTCOUNTTOTAL
400.001400.00
200.001200.00
100.001100.00
40.00140.00
25.00125.00
20.0018360.00
12.00112.00
10.00110.00
5.00210.00
4.0014.00

Original Submission

posted by martyb on Friday June 25 2021, @12:50PM   Printer-friendly
from the The-Mod-Squad dept.

There has been some discussion about moderation on this site leading to some misconceptions and misstatements. This story is an attempt to set things straight. It lays out the historical underpinnings for moderation, history of its implementation on Slashdot, and its later refinement on SoylentNews.

Before that, though, I am going to take this opportunity to thank fnord666 who is out Alternate Editor-in-Chief. I could not handle the load alone and his efforts have made a huge difference! Further, please join me in thanking him as he reached a new milestone: over 6,500 stories posted to the site! Many a late night or rare free moment has been generously given to the site. Teamwork++!

History:
The code for this site is a fork of code written for Slashdot. In that site's early days, it was apparent that some comments were much more interesting and informative than others. It was just as apparent that some users would just as gleefully troll the community. Moderation was conceived as a way to sift the wheat from the chaff and help users more easily avoid the "lesser" comments and more easily find the "gems".

Further, to encourage posting "good" comments, Karma was introduced. "Good" comments earned Karma; "bad" comments lost Karma. Moderation was a mechanism by which Karma could be allocated.

Slashdot experimented with several ways to moderate comments. First, it was just the staff who could moderate. Soon, there were too many comments to keep up, so a select group of members from the community were invited to moderate comments. Again, that failed to scale up, so those who had been selected were invited to recommend still other users to moderate. And, again, there were scaling issues.

Solution: make Mod Points (modpoints) available to every registered user in good standing and who indicated in their preferences that they were willing to moderate.

Originally, mod points were handed out randomly and expired after something like 6 hours: "Use 'em or lose 'em".

For the most part, that seemed to work. But there were some perceived issues and meta-moderation was implemented and introduced — moderate the moderations. Unfortunately, it experienced many of the same issues that it was supposed to rectify with comments, just one level abstracted. Further, it was unwieldy and when all was said and done, didn't work all that well, anyway.

Early Tweaking:
Such was the state of things when SoylentNews started. Well sort of. The code base we started with was not current and the meta-moderation code was broken. So much so, that meta-moderation was ripped out of the code just so regular moderation could be made to work. With that behind us, we finally we had a working moderation system on our site. Yay!

That worked okay for a while, but we found ourselves with complaints from many users that they wanted to moderate and lacked mod points. Nice problem to have, right? This was combined with many more comments than moderations. It was thought that we needed more mod points made available to the community. So, after unsuccessfully tweaking the mod point allocation algorithm, it was decided to just not expire mod points until day's end. Every user in good standing got 5 mod points each morning (00:10 UTC) and those were available until day's end whereupon any remaining modpoints were reset and a new set of 5 of modpoints were allocated.

That helped! But jerks will be jerks.

Mod Bombs:
We started to run into problems with "mod bombs" where one user "A" would apply all 5 of their mod points to downmod one other user "B". So code was written to allow checking for such moderations. Staff could generate a report and find such activity. It was decided that:

If you used ALL of your modpoints to downmod ONE user, that was a modbomb. IOW, 5 downmods bad; 4 downmods were permitted.

Initially, anyone who "modbombed" was manually given a "timeout". The first time earned a one month suspension of moderation privileges. A second occurrence earned a six month suspension.

Later, because there were still many more comments than moderations, the number of modpoints allocated to each registered user having good Karma was increased from 5 to 10 per day. The modbomb threshold was, however, kept the same: 4 downmods was still okay, 5 (or more) downmods to the same user was "bad".

A complication arose in that there is no easy way for users to keep track of how many downmods they had made on one other user. User "A" may do 3 downmods of user "B" in the morning and 4 down mods of other (unrelated) users. In the afternoon they might perform 2 more downmods of user "B". Purely unintentional transgression. When you only have 5 mod points it was reasonable to assume that a user could mentally track how many times they downmodded a single user in one day. With 10 daily mod points available, that became less reasonable.

So, along with the allocation of 10 modpoints per day (easy) it was intended to have code written that would kick in when processing moderations: when the threshold was exceeded, the excess downmods would be automatically rejected. And that is still the intent.

The upshot of all that is that when checking for modbombs, we no longer give a "timeout" for 5 downmods against a single user in one day. We just revert the excess mods. We do take note of repeated excesses and are fully prepared to issue a "timeout" when warranted. (e.g. 8 downmods in one day, or several days in close proximity targeting the same user. This is not done unilaterally but rather in consultation with other staff for confirmation.)

Sock Bombs:
First, there some who failed to take the hint that, maybe, they should take a look at what they were posting when they received repeated downmods. We are a community, not your personal soapbox. So, they created new ("sock puppet") accounts and proceeded to upmod their own comments, aka a "sockbomb". Staff have ways to note such behavior based on the IPID and SUBNETID that is recorded with every comment and every moderation. We try to give the benefit of the doubt. But, certain patterns do become apparent and are not tolerated. Upmodding your own comment is grounds for an immediate moderation ban.

Second, just as there is a limit on how many downmods can be targeted at one user in a day, so there is a limit on upmods. The same limits apply, each user "A" is limited to 4 upmods of user "B" in a given day, just like for "modbombs". Again with the caveat of no up-mods of your own account..

Summary:
Our experience is that the current system could stand some refinement, automation of transgression detection and mitigation is in plan (but it will be a while), but for the most part, what we have works well in the vast majority of cases. In short, Wheaton's Law still applies: "don't be a dick". Following that seems to work the best for the most. (With apologies to anyone named Richard. =)


Original Submission

posted by martyb on Thursday June 17 2021, @11:19AM   Printer-friendly

Last night (actually, very early this morning) mechanicjay generated and installed new Let's Encrypt certs for our servers.

I made a quick check and everything seems to be in place. The old certs were due to expire right about now, so if you do have any issues, please pop onto IRC (preferred) or reply here and let us know!

Thanks mechanicjay!


Original Submission

posted by martyb on Sunday June 13 2021, @04:50PM   Printer-friendly

[2021-06-14 02:24:41 UTC; update 2: We made a decision to accept Linode's offer of moving up our migration of fluorine. It appears the migration has completed successfully. YAY!]


[2021-06-14 00:25:32 UTC; update 1: hydrogen appears to have successfully migrated. We had a brief 503 on the site until I bounced varnish. The site seems to be fine, now.--Bytram]


Prologue:
First off please accept my sincere wish for a happy Father's Day to all our dads in the community! (It is celebrated next Sunday in 90 countries.)

Also, I am happy to report a surge in participation on the site over the past month. I've seen increases in story submissions, subscriptions[*], and participation (comments, moderations, etc.) Community++

[*] NB: I was successful in crediting users for their subscriptions on the site after the server crash. Unfortunately, that failed to account for the dollar amount of their subscriptions in our tracking database table which is used to source our progress against our funding goal. I have a plan for getting those updates in place, but want to run it past other members of staff to make sure everything is accounted for before making any changes.

Read on for the rest of the site's news, or just wait and a new story will be out before too long.

Server Migrations:
We have received word that Linode, our web-hosting provider, will be conducting maintenance on two of our servers in the next 24 hours.

Last night Linode shut down one of our servers (boron), migrated the disk image to a new physical server, and restarted it. All seems to have gone smoothly.

Later on today, two more of our servers are due to be migrated:

  • hydrogen (alternate database server) 4 CPU Cores, 160 GB Storage, 8 GB RAM
    This Linode's physical host will be undergoing maintenance at 2021-06-14 00:00 UTC. During this time, your Linode will be shut down, cold migrated to a new host, then returned to its last state (running or powered off)
  • fluorine (primary database server) 4 CPU Cores, 96 GB Storage, 8 GB RAM
    This Linode's physical host will be undergoing maintenance at 2021-06-14 07:00 UTC. During this time, your Linode will be shut down, cold migrated to a new host, then returned to its last state (running or powered off).

Also of note, we are eligible for a free storage upgrade on fluorine from 96 GB to 160 GB. It is not clear at this moment if we will also conduct the storage upgrade at this time.

Cert Updates:
Our certs (issued by Let's Encrypt) are due to expire June 17, 2021.

We are aware and intend to have updated certs installed before then.

(NB: I may have some terminology errors in what follows, but I believe the overall process/concepts should be correct.)

I have personally installed updated certs twice before on our servers, and if need be, am prepared to do so again. It has been a couple years or so but the process should remain largely the same. The majority of the steps are automated, but historically we've preferred to handle the DNS updates manually. That way, just in case something goes sideways, we are hands-on and can take steps to mitigate problems... instead of finding we have a botched DNS and greatly restricted access the servers. (That is a bit of an overstatement, but as I understand it, it's a lot easier to make changes over SSH connections to running servers than through a console port to one server at a time.)

Also, there has been discussion about using a fully-automated Let's Encrypt cert update process, we'll keep you posted.

Site News:
Behind the scenes we've been hard at work. juggs, mechanicjay, and audioguy have put in many long and thankless hours stabilizing and documenting our service infrastructure. They've made great strides and we continue to make progress. We cannot change what was done (and not done) in the past, but we can learn from it! What services "live" on what servers? How to restart each service? Monitoring of disk usage and CPU usage? All are gradually being documented and site operations knowledge is getting shared all around.

Lastly, here's a shout-out to the editorial staff who strive to keep stories coming to you 24/7. Fnord666 just posted his 6,500th story! Also, thanks to janrinok, mrpg, chromas, and FatPhil who have all pushed out stories this past month! Teamwork++!

[N.B. Let's not forget our Editor-In-Chief martyb, who just posted his 10,100th story! This is in addition to serving as our primary QA person. - Fnord]


Original Submission