Stories
Slash Boxes
Comments

SoylentNews is people

Meta
posted by martyb on Monday January 04 2021, @05:10AM   Printer-friendly [Skip to comment(s)]

Summary:
It wasn't just you; SoylentNews.org was down today (Sunday, 2021-01-03) for a few hours in the mid morning to early afternoon UTC. It seems to be back up and running, but there are some minor artifacts.

Background:
First sign was some CSS and Slashbox issues appearing on Saturday night. I was editing a story and when I tried to preview it, saw that the SlashBoxes that normally appeared on the LHS (Left-Hand Side) of the page were missing. A page refresh or two later, and things looked okay, again. A bit later I went to view a story and saw the same symptoms. This time a hard reload that ignored and cache on my system (Ctrl+F5) did the trick.

I popped onto IRC (Internet Relay Chat), reported these symptoms, and asking if anyone else was seeing the same thing. Received a couple confirmations.

Oh. Joy. And TMB (The Mighty Buzzard) still seemed to be away on vacation.

Oh well. Skipped on over to boron and ran a script to bounce the apache servers on fluorine and hydrogen. Popped back onto IRC, reported what I did, and asked if things were better. Got some affirmations. Yay!

Just in case, I hung around for another half hour or so to confirm the site was staying up and running okay. Looking good! After thanking everyone for their help, I wished everybody a good night and then headed to bed.

Sunday:
Shortly after I woke and attempted to visit the site, I was greeted by a message explaining the site was down due to DB issues. When I got back onto IRC, found that TMB was already hands-on. The site had crashed early in the morning. With the site already down, and it being Sunday morning, he decided to take advantage of the opportunity to make some backups and then do some maintenance work.

Status:
Site is back up, system loads seem back-to-normal, and things seem to be pretty much as they should be. Except... the Older Stuff slashbox that appears on the RHS (Right-Hand Side) of the main page seemed to be missing some entries. The newest entry as I write this is YouTube Class Action: Same IP Address Used To Upload 'Pirate' Movies and File DMCA Notice.

I suspect the missing entries will eventually start to stream in and repopulate the list.

tl;dr:
The DB crashed and took the site with it. TMB was soon on the scene and fixed the DB and did some other work. We're back up and running.

Thanks TMB!


Original Submission

Related Stories

YouTube Class Action: Same IP Address Used To Upload 'Pirate' Movies and File DMCA Notices 43 comments

YouTube Class Action: Same IP Address Used to Upload 'Pirate' Movies & File DMCA Notices

Grammy award-winning musician Maria Schneider and Virgin Islands-based Pirate Monitor Ltd teamed up in the summer to file a class-action lawsuit against YouTube.

In an effort to gain access to YouTube's Content ID system, the complaint stated that YouTube has an allegedly lax attitude to takedown notices and repeat infringers, and discriminates against smaller creators.

Schneider told the court that a number of her songs had been posted to YouTube without her permission. Pirate Monitor Ltd argued similarly, stating that pirated copies of its works had been uploaded to the site. Both further said they had been denied access to Content ID.

In its response, YouTube focused on Pirate Monitor, alleging that the company or its agents uploaded the 'pirate' movies and then claimed mass infringement, something which disqualified them from accessing Content ID.


Original Submission

This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
(1)
  • (Score: 3, Informative) by Runaway1956 on Monday January 04 2021, @05:39AM (10 children)

    by Runaway1956 (2926) Subscriber Badge on Monday January 04 2021, @05:39AM (#1094373) Homepage Journal

    TMB is a lifesaver. Several people showed up on IRC complaining of withdrawal pains from SN. I hope we didn't lose anybody!

    --
    Let's go Brandon!
    • (Score: -1, Flamebait) by Anonymous Coward on Monday January 04 2021, @10:14AM (7 children)

      by Anonymous Coward on Monday January 04 2021, @10:14AM (#1094420)

      You should lose everybody.

      You should lose everybody who realizes that Soystain Shit News is entirely dependent on one load bearing troll who is not only a colossal fucking liberarian asshole troll but also personally responsible for keeping the site running because every single last one of you utter scum is completely fucking incompetent.

      I'd say you need to recruit some tech people. Wonder why you can't? Everybody who could help you refuses to do so beause nobody wants to touch the toxic pile of political shit that is Soystain Shit News.

      Enjoy total site failure when Buzzard gets hit by a bus. You will deserve it.

      • (Score: 5, Informative) by The Mighty Buzzard on Monday January 04 2021, @11:18AM

        No buses in small town Tennessee but I do appreciate your concern for my welfare.

        --
        My rights don't end where your fear begins.
      • (Score: 4, Touché) by janrinok on Monday January 04 2021, @01:17PM (5 children)

        by janrinok (52) Subscriber Badge on Monday January 04 2021, @01:17PM (#1094452) Journal

        I don't think that TMB is a librarian.... but perhaps you mistyped that because your fingers cannot keep up with the crap you are trying to write.

        We do have other technical staff - but you may have noticed that the last 12 months or so have not exactly been normal. Everybody has real lives (apart from yourself, obviously) and they are not able to help at the moment. From time to time we do ask for volunteers for all posts but it seems many are like you - all mouth but unable to provide any real practical support.

        If you dislike this site so much - why are you still here? Don't you know how to access a different URL?

        --
        It's always my fault...
        • (Score: 2) by The Mighty Buzzard on Monday January 04 2021, @02:05PM (3 children)

          I do have a couple thousand hardcopy books. And maybe twice as many comic books from many longs ago. And eventually there's going to be a pretty awesome library in the church house.

          I think he read a lot of Pratchett and was implying that I'm an orangutan though.

          --
          My rights don't end where your fear begins.
          • (Score: 0) by Anonymous Coward on Monday January 04 2021, @03:44PM (1 child)

            by Anonymous Coward on Monday January 04 2021, @03:44PM (#1094504)

            Any theme to your library? Mine (partly inherited) is about the same size as yours, but nearly all the books and magazines are on transportation--cars, car racing and some aircraft.

            • (Score: 2) by The Mighty Buzzard on Wednesday January 06 2021, @06:43AM

              Dark hardwood, comfortable leather seating, and bronze fixtures. The only thing going to be painted is the vaulted ceiling. It's going to be white so we can bounce and scatter the big lights off of it when lamps and wall sconces won't do. No chandeliers necessary, which will be nice since the ceiling peak is thirty feet above the floor and any wiring for them would have to be surface mounted.

              --
              My rights don't end where your fear begins.
          • (Score: 2) by Reziac on Tuesday January 05 2021, @02:42AM

            by Reziac (2489) on Tuesday January 05 2021, @02:42AM (#1094813) Homepage

            A man after my own library... some year I'll build shelves and get 'em back outta the boxes... main function of a house is a place to keep the durn books.

            When you get confused with an orangutan, might be time to strop your razor. ;)

            [Thanks for all you do here, Buzzy.]

        • (Score: 3, Funny) by RS3 on Monday January 04 2021, @03:55PM

          by RS3 (6367) on Monday January 04 2021, @03:55PM (#1094507)

          No no, not "librarian", but "liberarian": one who liberates. You're welcome. :-}

    • (Score: 0) by Anonymous Coward on Monday January 04 2021, @12:56PM (1 child)

      by Anonymous Coward on Monday January 04 2021, @12:56PM (#1094447)

      Yep, and the people that showed up on IRC traded jokes and bantered with each other and TMB to keep it all fun (at least it looked like fun from my end...)

      Thanks TMB!

  • (Score: 2, Interesting) by pTamok on Monday January 04 2021, @08:47AM (8 children)

    by pTamok (3042) on Monday January 04 2021, @08:47AM (#1094404)

    Thanks TMB!

    I would love to know your diagnosis and steps to resolve. Just because I'm unreasonably curious. In whatever format you are most comfortable with and wastes the least of your time.

    • (Score: 2) by fakefuck39 on Monday January 04 2021, @08:56AM (1 child)

      by fakefuck39 (6620) on Monday January 04 2021, @08:56AM (#1094406)

      I second this. This would actually be a very interesting "article" to post and discuss. What was wrong? Just dump some data, don't spend any time making it into a pretty article. We're unemployed and bored over here. speaking in plural of myself, because I'm the king, because I'm sitting on the royal throne right now. flush.

      • (Score: -1, Offtopic) by Anonymous Coward on Monday January 04 2021, @10:16AM

        by Anonymous Coward on Monday January 04 2021, @10:16AM (#1094421)

        Check these logs: https://i.redd.it/hxcae8esv8961.png [i.redd.it]

    • (Score: 5, Informative) by The Mighty Buzzard on Monday January 04 2021, @11:46AM (5 children)

      Wish I knew. The NDB data nodes both went down not long before I woke up. MySQL-Cluster being the flaming hunk of shit that it is diagnostics-wise, there were no messages in the log files or management nodes' event logs that were even slightly informative, even though they're set to be about as verbose as they can be. They weren't out of sync or even in an unhappy state, just run of the mill log entries followed by a single log entry each that they were disconnecting. Not why they were disconnecting, just that they were. So after the 20-30 minutes later that it takes to restart them one at a time, they were both back up and chugging along like normal with no complaints from either node during the process.

      The rest of the time was taken up by making and testing a mysqldump backup (did not initially work, needed a config change and restart in initial mode for each node to allow the redo logs to grow enough for our whole db to be restored) and running a couple of ALTER TABLEs that had to be done in the db's single user mode. The ALTER TABLEs were supposed to be done as part of a site update quite a while ago but they wouldn't complete in normal mode and it didn't really matter much if they did unless you were an editor wanting to paste War and Peace into the editors' notes bit of the stories editing pages (VARCHAR to TEXT).

      --
      My rights don't end where your fear begins.
      • (Score: 1) by pTamok on Monday January 04 2021, @03:21PM

        by pTamok (3042) on Monday January 04 2021, @03:21PM (#1094494)

        Thank you for taking the time to provide the summary. I appreciate it.

      • (Score: 0) by Anonymous Coward on Monday January 04 2021, @09:04PM

        by Anonymous Coward on Monday January 04 2021, @09:04PM (#1094670)

        Did you try turning it off and then on again?

      • (Score: 0) by Anonymous Coward on Tuesday January 05 2021, @01:21AM

        by Anonymous Coward on Tuesday January 05 2021, @01:21AM (#1094792)

        It sounds like a restart of the whole system would have been useful as a preventative measure?

      • (Score: 0) by Anonymous Coward on Tuesday January 05 2021, @02:00AM (1 child)

        by Anonymous Coward on Tuesday January 05 2021, @02:00AM (#1094805)

        Did they disconnect from each other, the mgmt nodes, the API nodes, or all of the above? What message did they give as to why they refused to reconnect? Did you have to roll the whole cluster? What did the interleaving of the logs look like? Your ndbinfo tables and SHOW output should have been more than enough to diagnose the various issues you have.

        That said, based on what little information you've given so far in this and the past few weeks post, it sounds like your various nodes are having trouble communicating with each other cleanly. It would roughly explain the problems over the past few weeks as much as they were described. You may want to monitor your traffic between systems for drops, jitter, and the rest. With that in hand, it might be helpful to escalate it to your VPS host if you record those logs as the true problem may be on their end.

        • (Score: 2) by The Mighty Buzzard on Wednesday January 06 2021, @06:58AM

          The data nodes disconnected and either shut down or crashed with no messages other than one each telling me the node was disconnecting (within a minute of each other). Started them back up and they connected like they hadn't had any problems. The management servers kept chugging right along the whole time.

          The reason I gave such little information is because there was such little information. There was literally nothing else out of the ordinary in the logs aside from those two messages. When I have time I'll be spending many a morning poking everything with a stick but we're still a little ways off from being moved into the church. Thankfully the drywall guys are moving right along and we're going to be able to start painting this weekend. Hot water tank, building some temporary kitchen counters that'll be later repurposed into workbenches, outlets/switches/fixtures, trim, and hanging a few doors should just about do it after that.

          --
          My rights don't end where your fear begins.
  • (Score: 3, Funny) by kazzie on Monday January 04 2021, @10:05AM

    by kazzie (5309) Subscriber Badge on Monday January 04 2021, @10:05AM (#1094418)

    We're down for database maintenance that's been needed since 12/26.

    I first read the above as "needed since 1726". That's a lot of uptime!

  • (Score: 2) by acid andy on Monday January 04 2021, @01:02PM (1 child)

    by acid andy (1683) on Monday January 04 2021, @01:02PM (#1094449) Homepage Journal

    Thanks for getting it back both of you.

    After the site came back up yesterday, searches had stopped working and seemed to return no results. They're back now. Not sure if the DB had to rebuild some indices or what.

    --
    Where did that thought come from? And that one? What about this one? Woah, man...
  • (Score: 2) by iWantToKeepAnon on Monday January 04 2021, @10:22PM

    by iWantToKeepAnon (686) Subscriber Badge on Monday January 04 2021, @10:22PM (#1094704) Homepage Journal
    Thanks to all the site's support, editors, contributors, etc... Y'all do a heroic and oft thankless job.
    --
    "Happy families are all alike; every unhappy family is unhappy in its own way." -- Anna Karenina by Leo Tolstoy
(1)