SoylentNews is people
SoylentNews
[Update: It looks like the slashd daemon which was scheduled for 2017-06-29_00:10:00 failed to run again. We are investigating.]
We ran into a problem with the process which hands out moderation points each night.
We received a couple reports that people had no mod points. A query of our DB showed that fully 80% of our users had the full complement of 5 mod points. That seemed strange — we have a daemon that runs every night and that, among many other things, replenishes your supply each morning at 00:10 UTC. Apparently that process fell over and went toes up. Complicating matters, if you had mod points left over from the prior day, those were still available to you.
I put out the call to the devs as I tried to sleuth out what was going on. Many thanks to mrpg who played guinea pig and offered a fresh perspective as we tried to isolate the issue. With the information that was gathered, TheMightyBuzzard and paulej72 quickly figured out what happened. Further, rather than wait for tonight's process to run, TheMightyBuzzard manually updated the DB and handed out mod points to everyone.
(Debugging was complicated by the fact that there was another issue that was clogging up the logs which made it doubly hard to determine what happened. Debugging efforts are continuing on that matter.)
We anticipate things should be back to normal tomorrow. We'll check in on this story first thing in the morning so if you run into any issues with this, please post a reply with details.
P.S. I remember in the early days of this site when more than 12 hours of continuous up-time was an accomplishment. It's a credit to the staff here that we are now at a state where a system issue is a rare event, rather than an everyday occurrence.
-- martyb
(Score: 2) by kaszz on Thursday June 29 2017, @12:54AM (5 children)
What made the difference so you could stay up longer than 12 hours?
(Score: 4, Funny) by bob_super on Thursday June 29 2017, @01:29AM
Disabling Dr. Watson, which was called in every four hours.
[ducks]
(Score: 0) by Anonymous Coward on Thursday June 29 2017, @09:56AM
They migrated from Windows servers to Linux.
(Score: 2) by NCommander on Thursday June 29 2017, @11:19PM (2 children)
To put it bluntly, we got slashdotted :).
More specifically, at the time, none of us had a lot of experience on dealing with a large codebase, and Slashcode was a rather massive beast, and with a lot of obsolete dependencies. The usual problem was slashd would hang. I can't actually remember how we fixed it short of deleting a lot of code.
Still always moving
(Score: 2) by Reziac on Friday June 30 2017, @04:00AM (1 child)
Well, it's nothing compared to the Slashdot modpoints bug... twice now it's given me infinite mod points: I spend 15 points and it immediately resets to 15 more points. First time this lasted about six months, second time a couple weeks (maybe longer; this was post-Soylent and I wasn't spending so much time there anymore).
And there is no Alkibiades to come back and save us from ourselves.
(Score: 2) by NCommander on Saturday July 01 2017, @08:34AM
Are the moderations going through? There is/was a longstanding issue that when you tried to spend too many points, it would abort the transaction and return them.
Also, welcome back. I'm curious on why you left and why you came back.
Still always moving