Summary:
It wasn't just you; SoylentNews.org was down today (Sunday, 2021-01-03) for a few hours in the mid morning to early afternoon UTC. It seems to be back up and running, but there are some minor artifacts.
Background:
First sign was some CSS and Slashbox issues appearing on Saturday night. I was editing a story and when I tried to preview it, saw that the SlashBoxes that normally appeared on the LHS (Left-Hand Side) of the page were missing. A page refresh or two later, and things looked okay, again. A bit later I went to view a story and saw the same symptoms. This time a hard reload that ignored and cache on my system (Ctrl+F5) did the trick.
I popped onto IRC (Internet Relay Chat), reported these symptoms, and asking if anyone else was seeing the same thing. Received a couple confirmations.
Oh. Joy. And TMB (The Mighty Buzzard) still seemed to be away on vacation.
Oh well. Skipped on over to boron and ran a script to bounce the apache servers on fluorine and hydrogen. Popped back onto IRC, reported what I did, and asked if things were better. Got some affirmations. Yay!
Just in case, I hung around for another half hour or so to confirm the site was staying up and running okay. Looking good! After thanking everyone for their help, I wished everybody a good night and then headed to bed.
Sunday:
Shortly after I woke and attempted to visit the site, I was greeted by a message explaining the site was down due to DB issues. When I got back onto IRC, found that TMB was already hands-on. The site had crashed early in the morning. With the site already down, and it being Sunday morning, he decided to take advantage of the opportunity to make some backups and then do some maintenance work.
Status:
Site is back up, system loads seem back-to-normal, and things seem to be pretty much as they should be. Except... the Older Stuff slashbox that appears on the RHS (Right-Hand Side) of the main page seemed to be missing some entries. The newest entry as I write this is YouTube Class Action: Same IP Address Used To Upload 'Pirate' Movies and File DMCA Notice.
I suspect the missing entries will eventually start to stream in and repopulate the list.
tl;dr:
The DB crashed and took the site with it. TMB was soon on the scene and fixed the DB and did some other work. We're back up and running.
Thanks TMB!
Related Stories
YouTube Class Action: Same IP Address Used to Upload 'Pirate' Movies & File DMCA Notices
Grammy award-winning musician Maria Schneider and Virgin Islands-based Pirate Monitor Ltd teamed up in the summer to file a class-action lawsuit against YouTube.
In an effort to gain access to YouTube's Content ID system, the complaint stated that YouTube has an allegedly lax attitude to takedown notices and repeat infringers, and discriminates against smaller creators.
Schneider told the court that a number of her songs had been posted to YouTube without her permission. Pirate Monitor Ltd argued similarly, stating that pirated copies of its works had been uploaded to the site. Both further said they had been denied access to Content ID.
In its response, YouTube focused on Pirate Monitor, alleging that the company or its agents uploaded the 'pirate' movies and then claimed mass infringement, something which disqualified them from accessing Content ID.
(Score: 3, Informative) by Runaway1956 on Monday January 04 2021, @05:39AM (10 children)
TMB is a lifesaver. Several people showed up on IRC complaining of withdrawal pains from SN. I hope we didn't lose anybody!
“I have become friends with many school shooters” - Tampon Tim Walz
(Score: -1, Flamebait) by Anonymous Coward on Monday January 04 2021, @10:14AM (7 children)
You should lose everybody.
You should lose everybody who realizes that Soystain Shit News is entirely dependent on one load bearing troll who is not only a colossal fucking liberarian asshole troll but also personally responsible for keeping the site running because every single last one of you utter scum is completely fucking incompetent.
I'd say you need to recruit some tech people. Wonder why you can't? Everybody who could help you refuses to do so beause nobody wants to touch the toxic pile of political shit that is Soystain Shit News.
Enjoy total site failure when Buzzard gets hit by a bus. You will deserve it.
(Score: 5, Informative) by The Mighty Buzzard on Monday January 04 2021, @11:18AM
No buses in small town Tennessee but I do appreciate your concern for my welfare.
My rights don't end where your fear begins.
(Score: 4, Touché) by janrinok on Monday January 04 2021, @01:17PM (5 children)
I don't think that TMB is a librarian.... but perhaps you mistyped that because your fingers cannot keep up with the crap you are trying to write.
We do have other technical staff - but you may have noticed that the last 12 months or so have not exactly been normal. Everybody has real lives (apart from yourself, obviously) and they are not able to help at the moment. From time to time we do ask for volunteers for all posts but it seems many are like you - all mouth but unable to provide any real practical support.
If you dislike this site so much - why are you still here? Don't you know how to access a different URL?
[nostyle RIP 06 May 2025]
(Score: 2) by The Mighty Buzzard on Monday January 04 2021, @02:05PM (3 children)
I do have a couple thousand hardcopy books. And maybe twice as many comic books from many longs ago. And eventually there's going to be a pretty awesome library in the church house.
I think he read a lot of Pratchett and was implying that I'm an orangutan though.
My rights don't end where your fear begins.
(Score: 0) by Anonymous Coward on Monday January 04 2021, @03:44PM (1 child)
Any theme to your library? Mine (partly inherited) is about the same size as yours, but nearly all the books and magazines are on transportation--cars, car racing and some aircraft.
(Score: 2) by The Mighty Buzzard on Wednesday January 06 2021, @06:43AM
Dark hardwood, comfortable leather seating, and bronze fixtures. The only thing going to be painted is the vaulted ceiling. It's going to be white so we can bounce and scatter the big lights off of it when lamps and wall sconces won't do. No chandeliers necessary, which will be nice since the ceiling peak is thirty feet above the floor and any wiring for them would have to be surface mounted.
My rights don't end where your fear begins.
(Score: 2) by Reziac on Tuesday January 05 2021, @02:42AM
A man after my own library... some year I'll build shelves and get 'em back outta the boxes... main function of a house is a place to keep the durn books.
When you get confused with an orangutan, might be time to strop your razor. ;)
[Thanks for all you do here, Buzzy.]
And there is no Alkibiades to come back and save us from ourselves.
(Score: 3, Funny) by RS3 on Monday January 04 2021, @03:55PM
No no, not "librarian", but "liberarian": one who liberates. You're welcome. :-}
(Score: 0) by Anonymous Coward on Monday January 04 2021, @12:56PM (1 child)
Yep, and the people that showed up on IRC traded jokes and bantered with each other and TMB to keep it all fun (at least it looked like fun from my end...)
Thanks TMB!
(Score: 4, Insightful) by The Mighty Buzzard on Monday January 04 2021, @02:02PM
Lose your db nodes, you get a little unplanned work. Lose your sense of humor, you're just completely screwed.
My rights don't end where your fear begins.
(Score: 2, Interesting) by pTamok on Monday January 04 2021, @08:47AM (8 children)
Thanks TMB!
I would love to know your diagnosis and steps to resolve. Just because I'm unreasonably curious. In whatever format you are most comfortable with and wastes the least of your time.
(Score: 2) by fakefuck39 on Monday January 04 2021, @08:56AM (1 child)
I second this. This would actually be a very interesting "article" to post and discuss. What was wrong? Just dump some data, don't spend any time making it into a pretty article. We're unemployed and bored over here. speaking in plural of myself, because I'm the king, because I'm sitting on the royal throne right now. flush.
(Score: -1, Offtopic) by Anonymous Coward on Monday January 04 2021, @10:16AM
Check these logs: https://i.redd.it/hxcae8esv8961.png [i.redd.it]
(Score: 5, Informative) by The Mighty Buzzard on Monday January 04 2021, @11:46AM (5 children)
Wish I knew. The NDB data nodes both went down not long before I woke up. MySQL-Cluster being the flaming hunk of shit that it is diagnostics-wise, there were no messages in the log files or management nodes' event logs that were even slightly informative, even though they're set to be about as verbose as they can be. They weren't out of sync or even in an unhappy state, just run of the mill log entries followed by a single log entry each that they were disconnecting. Not why they were disconnecting, just that they were. So after the 20-30 minutes later that it takes to restart them one at a time, they were both back up and chugging along like normal with no complaints from either node during the process.
The rest of the time was taken up by making and testing a mysqldump backup (did not initially work, needed a config change and restart in initial mode for each node to allow the redo logs to grow enough for our whole db to be restored) and running a couple of ALTER TABLEs that had to be done in the db's single user mode. The ALTER TABLEs were supposed to be done as part of a site update quite a while ago but they wouldn't complete in normal mode and it didn't really matter much if they did unless you were an editor wanting to paste War and Peace into the editors' notes bit of the stories editing pages (VARCHAR to TEXT).
My rights don't end where your fear begins.
(Score: 1) by pTamok on Monday January 04 2021, @03:21PM
Thank you for taking the time to provide the summary. I appreciate it.
(Score: 0) by Anonymous Coward on Monday January 04 2021, @09:04PM
Did you try turning it off and then on again?
(Score: 0) by Anonymous Coward on Tuesday January 05 2021, @01:21AM
It sounds like a restart of the whole system would have been useful as a preventative measure?
(Score: 0) by Anonymous Coward on Tuesday January 05 2021, @02:00AM (1 child)
Did they disconnect from each other, the mgmt nodes, the API nodes, or all of the above? What message did they give as to why they refused to reconnect? Did you have to roll the whole cluster? What did the interleaving of the logs look like? Your ndbinfo tables and SHOW output should have been more than enough to diagnose the various issues you have.
That said, based on what little information you've given so far in this and the past few weeks post, it sounds like your various nodes are having trouble communicating with each other cleanly. It would roughly explain the problems over the past few weeks as much as they were described. You may want to monitor your traffic between systems for drops, jitter, and the rest. With that in hand, it might be helpful to escalate it to your VPS host if you record those logs as the true problem may be on their end.
(Score: 2) by The Mighty Buzzard on Wednesday January 06 2021, @06:58AM
The data nodes disconnected and either shut down or crashed with no messages other than one each telling me the node was disconnecting (within a minute of each other). Started them back up and they connected like they hadn't had any problems. The management servers kept chugging right along the whole time.
The reason I gave such little information is because there was such little information. There was literally nothing else out of the ordinary in the logs aside from those two messages. When I have time I'll be spending many a morning poking everything with a stick but we're still a little ways off from being moved into the church. Thankfully the drywall guys are moving right along and we're going to be able to start painting this weekend. Hot water tank, building some temporary kitchen counters that'll be later repurposed into workbenches, outlets/switches/fixtures, trim, and hanging a few doors should just about do it after that.
My rights don't end where your fear begins.
(Score: 3, Funny) by kazzie on Monday January 04 2021, @10:05AM
I first read the above as "needed since 1726". That's a lot of uptime!
(Score: 2) by acid andy on Monday January 04 2021, @01:02PM (1 child)
Thanks for getting it back both of you.
After the site came back up yesterday, searches had stopped working and seemed to return no results. They're back now. Not sure if the DB had to rebuild some indices or what.
error count exceeds 100; stopping compilation
(Score: 3, Informative) by The Mighty Buzzard on Monday January 04 2021, @01:59PM
Nah, I just forgot to turn the search daemon back on. Downed everything using the db when I put it in single user mode.
My rights don't end where your fear begins.
(Score: 2) by iWantToKeepAnon on Monday January 04 2021, @10:22PM
"Happy families are all alike; every unhappy family is unhappy in its own way." -- Anna Karenina by Leo Tolstoy