[Update: It looks like the slashd daemon which was scheduled for 2017-06-29_00:10:00 failed to run again. We are investigating.]
We ran into a problem with the process which hands out moderation points each night.
We received a couple reports that people had no mod points. A query of our DB showed that fully 80% of our users had the full complement of 5 mod points. That seemed strange — we have a daemon that runs every night and that, among many other things, replenishes your supply each morning at 00:10 UTC. Apparently that process fell over and went toes up. Complicating matters, if you had mod points left over from the prior day, those were still available to you.
I put out the call to the devs as I tried to sleuth out what was going on. Many thanks to mrpg who played guinea pig and offered a fresh perspective as we tried to isolate the issue. With the information that was gathered, TheMightyBuzzard and paulej72 quickly figured out what happened. Further, rather than wait for tonight's process to run, TheMightyBuzzard manually updated the DB and handed out mod points to everyone.
(Debugging was complicated by the fact that there was another issue that was clogging up the logs which made it doubly hard to determine what happened. Debugging efforts are continuing on that matter.)
We anticipate things should be back to normal tomorrow. We'll check in on this story first thing in the morning so if you run into any issues with this, please post a reply with details.
P.S. I remember in the early days of this site when more than 12 hours of continuous up-time was an accomplishment. It's a credit to the staff here that we are now at a state where a system issue is a rare event, rather than an everyday occurrence.
-- martyb
(Score: 2) by kaszz on Thursday June 29 2017, @12:54AM (5 children)
What made the difference so you could stay up longer than 12 hours?
(Score: 4, Funny) by bob_super on Thursday June 29 2017, @01:29AM
Disabling Dr. Watson, which was called in every four hours.
[ducks]
(Score: 0) by Anonymous Coward on Thursday June 29 2017, @09:56AM
They migrated from Windows servers to Linux.
(Score: 2) by NCommander on Thursday June 29 2017, @11:19PM (2 children)
To put it bluntly, we got slashdotted :).
More specifically, at the time, none of us had a lot of experience on dealing with a large codebase, and Slashcode was a rather massive beast, and with a lot of obsolete dependencies. The usual problem was slashd would hang. I can't actually remember how we fixed it short of deleting a lot of code.
Still always moving
(Score: 2) by Reziac on Friday June 30 2017, @04:00AM (1 child)
Well, it's nothing compared to the Slashdot modpoints bug... twice now it's given me infinite mod points: I spend 15 points and it immediately resets to 15 more points. First time this lasted about six months, second time a couple weeks (maybe longer; this was post-Soylent and I wasn't spending so much time there anymore).
And there is no Alkibiades to come back and save us from ourselves.
(Score: 2) by NCommander on Saturday July 01 2017, @08:34AM
Are the moderations going through? There is/was a longstanding issue that when you tried to spend too many points, it would abort the transaction and return them.
Also, welcome back. I'm curious on why you left and why you came back.
Still always moving
(Score: 0) by Anonymous Coward on Thursday June 29 2017, @12:57AM
Good to know you guys take this as, if not more, seriously than that other site :)
p.s. Datestamped 'old news' pages are the best for bookmarking old stories you might want to look up years later. It is too bad every other site decided to move to the 'pages from current news' indexing instead :(
(Score: 3, Insightful) by aristarchus on Thursday June 29 2017, @01:24AM (3 children)
I am happy to have Hanlon's razor confirmed yet again. Well, only the part about it not being malice. Good work, eds and admins!
(Score: 2) by The Mighty Buzzard on Thursday June 29 2017, @01:48AM (2 children)
Oh there's a moderate likelihood it'll turn out to be stupidity causing it. I'm just hoping it was an old slashcode dev's instead of mine.
My rights don't end where your fear begins.
(Score: 2, Redundant) by c0lo on Thursday June 29 2017, @03:35AM (1 child)
It'll be a first in a long time, if you discover that the fault was in your code, right?
Question is: will you admit it (and repent)?
(grin)
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 2) by The Mighty Buzzard on Thursday June 29 2017, @09:46AM
I always admit when I'm wrong. It just happens so rarely.
My rights don't end where your fear begins.
(Score: 4, Insightful) by frojack on Thursday June 29 2017, @01:42AM (4 children)
All those complaining that I saw were ACs.
Since when did they start getting mod points?
No, you are mistaken. I've always had this sig.
(Score: 2) by The Mighty Buzzard on Thursday June 29 2017, @02:10AM (1 child)
I could give em some if you like. Wouldn't help though cause there are too many !$user->{is_anon} checks for an AC to actually use them.
My rights don't end where your fear begins.
(Score: 4, Funny) by chromas on Thursday June 29 2017, @02:36AM
As long as they show up as available. Imagine the frustration as they keep trying to mod things.
(Score: 2, Funny) by Anonymous Coward on Thursday June 29 2017, @05:02AM
Froj! These are not the ACs you are looking for! Do you see that little button, just above the "Submit" button on the comment screen? It says, "post as AC". Some ACs are not ACs, so they have mod points under their regular user account, but are just posting as AC in case the purge has started, and we all need to run for the nearest onion router portal . . . it could happen, seriously!
Signed, your favorite AC
(Score: 0) by Anonymous Coward on Thursday June 29 2017, @09:40AM
Because it was a conspiracy of satanic nazi space chickens and we didn't want them to find us.
(Score: 3, Insightful) by Phoenix666 on Thursday June 29 2017, @02:19AM (1 child)
Thanks for tracking it down, guys.
I am relieved that I don't have to keep asking myself why I haven't made myself disagreeable enough to have my mod points taken away, also. An AC told me I have a bad-boy image to maintain.
Washington DC delenda est.
(Score: 0) by Anonymous Coward on Thursday June 29 2017, @10:05AM
That AC is delusional. Your bad girl image was shattered last week when you were spotted trading your v card for a spliff in Lefty's Bar
(Score: 5, Funny) by realDonaldTrump on Thursday June 29 2017, @02:00PM (3 children)
I have 10 billion Moderator Points.
(Score: 0) by Anonymous Coward on Thursday June 29 2017, @02:39PM
-1 Triggered
(Score: 2) by turgid on Thursday June 29 2017, @04:04PM
I'm friendly with the SAS.
I refuse to engage in a battle of wits with an unarmed opponent [wikipedia.org].
(Score: 2) by DeathMonkey on Thursday June 29 2017, @05:49PM
Wow, that's way more than Obama had!
(Score: 0) by Anonymous Coward on Monday July 03 2017, @08:17AM
killswitch engage @
https://soylentnews.org/~retrodynamic/journal [soylentnews.org]
https://soylentnews.org/~centerprism/journal/ [soylentnews.org]