Stories
Slash Boxes
Comments

SoylentNews is people

Meta
posted by on Sunday August 09 2020, @05:51AM   Printer-friendly
from the SNAFU dept.

The Mighty Buzzard writes:

Yeah, so, failure to babysit the db node that was scheduled for a reboot on the 5th resulted in a bit of database FUBAR that left us temporarily losing everything from then to now. Fortunately we had a backup less than six hours old, restored from it, and appear to be copacetic now. Except for the missing five hours and change.

I'd usually make some sort of dumb joke here but it was already four hours past my bedtime when I found out about the problem. My brain is no work good anymore. Fill in whatever dad joke or snark about getting a do-over for a change strikes your fancy.

This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
(1) 2
  • (Score: 5, Informative) by Runaway1956 on Sunday August 09 2020, @06:15AM (17 children)

    by Runaway1956 (2926) Subscriber Badge on Sunday August 09 2020, @06:15AM (#1033675) Journal

    Personally, it's a disappointment that things happened when they did. But, it looks good now! Only lost a few hours, as opposed to 3 days!!

    • (Score: 5, Interesting) by martyb on Sunday August 09 2020, @08:48AM (16 children)

      by martyb (76) Subscriber Badge on Sunday August 09 2020, @08:48AM (#1033715) Journal

      Personally, it's a disappointment that things happened when they did. But, it looks good now! Only lost a few hours, as opposed to 3 days!!

      My sentiments, exactly.

      It's hard enough to get things going again. Doubly so when it's late at night and your body is screaming for sleep and concentration is required to avoid an even greater FUBAR.

      Thanks Buzz!!!!

      From my (editorial) perspective, there were a couple stories that I had pushed out that got lost. I know this thanks to "stale" browser tabs, I still had open. I don't know (yet)if any other stories got pushed out after those two and were subsequently lost.

      My "lost" stories have been recreated and pushed back out. Let's hope that was it.

      I do want to acknowledge lost comments, moderations, and potentially journal postings... that's a pain. Don't know what else to say. Truly sorry it happened but that won't bring things back. Ugh.

      It's way too early in the morning for me to be up and my brain is rebelling at being awake at this hour. /me wanders back to bed.

      --
      Wit is intellect, dancing.
      • (Score: 2, Interesting) by Anonymous Coward on Sunday August 09 2020, @09:43AM (10 children)

        by Anonymous Coward on Sunday August 09 2020, @09:43AM (#1033724)

        I don't know if this is confirmation bias, y'all being more public about this, or an actual increase, but you guys seem to keep having problems related the database processes as of late. Perhaps you should think about adding a watchdog daemon to your system, giving the database itself some maintenance and optimization, making sure everything is up to date, and checking your logs for some sort of attack on your system.

        • (Score: 5, Interesting) by The Mighty Buzzard on Sunday August 09 2020, @01:45PM (9 children)

          Funny how the db clustering system that's supposed to save us headaches has caused significant data loss twice now when boring old master/slave replication never did, ain't it? I'd have to do the math to see if occasionally restoring from backups has cost us more downtime than actually having to down the site when maintenance was required but I know for sure it's more annoying.

          --
          My rights don't end where your fear begins.
          • (Score: 0) by Anonymous Coward on Sunday August 09 2020, @07:18PM (4 children)

            by Anonymous Coward on Sunday August 09 2020, @07:18PM (#1033964)

            is there a post or posts that describe how everything is set up for SN? would make for an interesting read and other admins could weigh in with their 2 cents/$denomination.

            • (Score: 4, Funny) by The Mighty Buzzard on Monday August 10 2020, @04:31AM

              The other admins have better sense than to talk to users. I'm the dumb one.

              --
              My rights don't end where your fear begins.
            • (Score: 2) by The Mighty Buzzard on Monday August 10 2020, @05:13AM (2 children)

              Oh, if you really want to know the detailed network setup, drop me an email to remind me (I don't care if it's a real address. Throwaway is fine.) and I'll post it up as a journal entry when I get time. I've been running on busy days and four hours or so of sleep a night for what seems like about thirty years though, so don't go thinking I've forgotten about it until it doesn't show up within a week.

              --
              My rights don't end where your fear begins.
              • (Score: 2) by martyb on Monday August 10 2020, @10:14PM (1 child)

                by martyb (76) Subscriber Badge on Monday August 10 2020, @10:14PM (#1034556) Journal

                Consider me interested. :)

                If I may suggest, it you follow through in writing up something... put it up on the Wiki and then link to that in your journal. (There's probably some stuff up there to start from, anyway!)

                /me wishes there were a way to auto-explore and document (textually and graphically) connections between servers and the processes that run on each one.

                --
                Wit is intellect, dancing.
          • (Score: 0) by Anonymous Coward on Sunday August 09 2020, @10:13PM (1 child)

            by Anonymous Coward on Sunday August 09 2020, @10:13PM (#1034046)

            Are you anywhere close to the load limit on a replication setup? And a two node cluster is basically worthless because you can't get a quorum with only two nodes. Another benefit of a replication scheme for you seems to be that in the current setup, failure requires manual intervention anyway. So you can STONITH with a watchdog and degrade read-only to the replica at the first sign of trouble until you sort it out or when under maintenance.

            • (Score: 2) by The Mighty Buzzard on Monday August 10 2020, @04:56AM

              Two nodes is plenty for our purposes. Our network load vs. the bandwidth between our boxes makes replication essentially instant unless you have to completely restore a node, so mostly what we need is for the web frontends to not have to give a shit what db server they're dealing with in the event that one of them crashes. If we were looking to fail to read-only, we'd have stuck with master/slave. We consider read-only to be failure though.

              --
              My rights don't end where your fear begins.
          • (Score: 2) by gawdonblue on Monday August 10 2020, @02:40AM (1 child)

            by gawdonblue (412) on Monday August 10 2020, @02:40AM (#1034153)

            Yeah, in the last 3 years we've had to restart the DB at work twice because of "high-availability" clustering getting out of sync. These are the only fatal DB software failures that we have had.
            Seems the more dependencies you add the more brittle things become.

      • (Score: 0) by Anonymous Coward on Monday August 10 2020, @08:02PM (4 children)

        by Anonymous Coward on Monday August 10 2020, @08:02PM (#1034486)

        Boy, won't aristarchus be annoyed to find out that three of his stories were approved, as-is with no edits to them, but unfortunately were released in that window!

        • (Score: 2) by aristarchus on Tuesday August 11 2020, @01:53AM (3 children)

          by aristarchus (2645) on Tuesday August 11 2020, @01:53AM (#1034677) Journal

          Wouldn't be the first time! Eds want deniable plausibility.

          • (Score: 2) by The Mighty Buzzard on Tuesday August 11 2020, @02:48AM (2 children)

            by The Mighty Buzzard (18) Subscriber Badge <themightybuzzard@proton.me> on Tuesday August 11 2020, @02:48AM (#1034694) Homepage Journal

            No, they want potable decantibility.

            --
            My rights don't end where your fear begins.
            • (Score: 2) by aristarchus on Tuesday August 11 2020, @05:54AM (1 child)

              by aristarchus (2645) on Tuesday August 11 2020, @05:54AM (#1034762) Journal

              No, they want aristarchus vacuity! Unfortunately, I have been here since the beginning, and contrary to khallow's wishes, will probably be here until the end. That is how it is, with us near immortals, having to live through humanity committing the same stupid mistakes again, and again, until in the school of ages, some one like the TMB comes to a realization of a cosmic perspective. Think, TMB, if you were not a coder, and not a former assassin in the US Army. How would things appear, and what would reality, and fundamental truths be? Just a question.

  • (Score: 2) by sonamchauhan on Sunday August 09 2020, @06:25AM (1 child)

    by sonamchauhan (6546) Subscriber Badge on Sunday August 09 2020, @06:25AM (#1033679)

    Thank you for your work!

    • (Score: 3, Funny) by MostCynical on Sunday August 09 2020, @09:47AM

      by MostCynical (2589) on Sunday August 09 2020, @09:47AM (#1033725) Journal

      They weren't very interesting hours anyway.

      --
      "I guess once you start doubting, there's no end to it." -Batou, Ghost in the Shell: Stand Alone Complex
  • (Score: 3, Funny) by Kell on Sunday August 09 2020, @06:47AM (16 children)

    by Kell (292) on Sunday August 09 2020, @06:47AM (#1033686)

    Do you really believe Project Bluebook revealed EVERYTHING the Army Air Corps knew?
    Do you know what REALLY happened at Roswell, New Mexico in the late 1940's?
    Have you even heard about the Men in Black?
    Do you have any friends who have `missing days'?
    Do you sleep with your window unlocked?
    Be seeing you...

    --
    Scientists ask questions. Engineers solve problems.
    • (Score: 1, Funny) by Anonymous Coward on Sunday August 09 2020, @07:53AM

      by Anonymous Coward on Sunday August 09 2020, @07:53AM (#1033698)

      > Do you really believe Project Bluebook revealed EVERYTHING the Army Air Corps knew?
      Of course not. The Bluebook price of many used Yugo autos is unknowable.
      > Do you know what REALLY happened at Roswell, New Mexico in the late 1940's?
      Bill Griffith created "Zippy The Pinhead"?
      > Have you even heard about the Men in Black?
      Yes. Their lives matter.
      > Do you have any friends who have `missing days'?
      Yes. I blame Burning Man, Jack Daniels and shrooms.
      > Do you sleep with your window unlocked?
      Of course, the bars make it superfluous.
      > Be seeing you....
      I didn't resign, I was laid off due to Covid, you insensitive clod!

    • (Score: 2) by Kell on Sunday August 09 2020, @10:06AM (5 children)

      by Kell (292) on Sunday August 09 2020, @10:06AM (#1033728)

      Alas, no Star Control fans here, it seems (and yes, it's on-topic).

      --
      Scientists ask questions. Engineers solve problems.
      • (Score: 4, Funny) by RS3 on Sunday August 09 2020, @11:20AM (1 child)

        by RS3 (6367) on Sunday August 09 2020, @11:20AM (#1033741)

        I did all I can to help (with the improper downmods...)

        • (Score: 3, Funny) by Kell on Sunday August 09 2020, @11:51AM

          by Kell (292) on Sunday August 09 2020, @11:51AM (#1033749)

          Thank you for your service, brave soul. :D

          --
          Scientists ask questions. Engineers solve problems.
      • (Score: 2) by janrinok on Sunday August 09 2020, @01:55PM (1 child)

        by janrinok (52) Subscriber Badge on Sunday August 09 2020, @01:55PM (#1033807) Journal
        Never heard of it.... And now that I've read the wiki page, I don't think that I have missed much. But apologies for not knowing about a 1990s game.
        • (Score: 2) by Kell on Sunday August 09 2020, @11:17PM

          by Kell (292) on Sunday August 09 2020, @11:17PM (#1034069)

          Alas, to those of us native to the 90s, this was was good as it got. Many fond memories, often shared by those of a technical background of a certain vintage. If you've any taste for retrogaming then I recommend it whole-heartedly (and the entirety of several other organs, as well).

          --
          Scientists ask questions. Engineers solve problems.
      • (Score: 2) by Freeman on Monday August 10 2020, @04:56AM

        by Freeman (732) on Monday August 10 2020, @04:56AM (#1034204) Journal

        I don't seem to recall the actual adventure part of the gameplay. Just the space battles. Maybe I only played Urquan Masters? Beats me, all I remember doing were the skirmishes. I liked the Earth Vessel, whatever it was called. (Earthling Cruiser according to duckduckgo-fu.) Though, I seem to recall enjoying using this bad boy: "Ur-Quan Dreadnought".

        I do remember playing Starflight a good bit, but I think it was on a console? Ah, yeah, I played this one a good bit: https://www.ign.com/games/starflight [ign.com]

        Though, I think I eventually found it on PC, it was so out-dated, I wasn't terribly interested. Also, had already played that sucker a ton on Sega Genesis.

        Space Engineers is a really awesome game too, but it doesn't have much in the way of NPCs. Also, very little in the way of storyline and pretty much nothing with regards to having a space encounter with dialogue. https://www.spaceengineersgame.com/ [spaceengineersgame.com] Still, it's #4 on my games with most play time on Steam. Below Dungeon Defenders, 7 Days to Die, and Terraria. 3 of those 4 allow you to build a base / home and are sandbox style games. Dungeon Defenders is the exception (also at the #1 spot), but it's an exceptional Tower Defense / First Person / Third Person / Hack'n'Slash'RPG / Shop Keeper Sim / Online Multiplayer / Online Co-Op / Local Co-Op (Sit on the couch.) game.

        --
        Joshua 1:9 "Be strong and of a good courage; be not afraid, neither be thou dismayed: for the Lord thy God is with thee"
    • (Score: 0) by Anonymous Coward on Sunday August 09 2020, @01:12PM

      by Anonymous Coward on Sunday August 09 2020, @01:12PM (#1033781)

      Ha, see someone else is enjoying the X-files reruns on BBC

    • (Score: 2) by sjames on Sunday August 09 2020, @05:10PM (3 children)

      by sjames (2882) on Sunday August 09 2020, @05:10PM (#1033884) Journal

      So you're saying Johnny Cash had a double like Paul McCartney? And the guy with missing days might not be drinking too much?

      • (Score: 2) by Reziac on Monday August 10 2020, @01:28PM (2 children)

        by Reziac (2489) on Monday August 10 2020, @01:28PM (#1034307) Homepage

        Johnny Cash DID have a double. I met him. Maybe??

        April, 1982. Was driving through Colorado Springs. Stopped for gas. Had trouble with the pump. (This was back when self-serve was still somewhat newfangled and nothing like standardized.) Guy comes out of the shack to help. He's wearing coveralls, but no company logo. And he's the bloody spitting image of Johnny Cash.

        So after a suitable period of baffled staring, I said to him, "You look just like Johnny Cash!"

        And he replies, "Why, thank you," .... in Johnny Cash's voice.

        Many moons later, when der interwebs made it possible, I looked up Cash's itinerary for the period. And... he was touring in Colorado that week.

        Was it him, or a double? To this day, I just don't know.

        [By further weird coincidence, I'm typing this on a PC (that doesn't normally have everyday duties, but is doing emergency service) named "Cash" -- because it was acquired "one piece at a time, and it didn't cost me a dime..." and earlier I looked up a Cash tune on YT for Another Forum... I tell you, he's everywhere....]

        --
        And there is no Alkibiades to come back and save us from ourselves.
        • (Score: 3, Interesting) by sjames on Monday August 24 2020, @07:22AM (1 child)

          by sjames (2882) on Monday August 24 2020, @07:22AM (#1041068) Journal

          What is it about gas stations? I once had a good discussion about quantum physics' implications for philosophy with a gas station attendant who looked exactly like Roy Orbison.

          • (Score: 2) by Reziac on Monday August 24 2020, @12:44PM

            by Reziac (2489) on Monday August 24 2020, @12:44PM (#1041108) Homepage

            Maybe it's that gas stations have excellent taste in music? or it's their way of expressing resentment over the crud piped in to hapless users of self-serve pumps??

            --
            And there is no Alkibiades to come back and save us from ourselves.
    • (Score: 0) by Anonymous Coward on Sunday August 09 2020, @11:42PM (1 child)

      by Anonymous Coward on Sunday August 09 2020, @11:42PM (#1034075)

      Look, well all well know that any missing time TMB may, or may not, have experienced is due to high quality hooch.

    • (Score: 2) by Reziac on Monday August 10 2020, @05:09AM (1 child)

      by Reziac (2489) on Monday August 10 2020, @05:09AM (#1034211) Homepage

      80 years it takes the little grey guys to come back and steal a few hours of database? they need to tune up their spaceship. And upgrade their storage.

      --
      And there is no Alkibiades to come back and save us from ourselves.
      • (Score: 3, Insightful) by Kell on Monday August 10 2020, @08:19AM

        by Kell (292) on Monday August 10 2020, @08:19AM (#1034255)

        The ways of the Arilou are obscure.

        --
        Scientists ask questions. Engineers solve problems.
  • (Score: 3, Informative) by Barenflimski on Sunday August 09 2020, @06:54AM (4 children)

    by Barenflimski (6836) on Sunday August 09 2020, @06:54AM (#1033687)

    And thanks as well. Sounds like a thing that happens to computers.

    • (Score: -1, Flamebait) by Anonymous Coward on Sunday August 09 2020, @06:57AM (3 children)

      by Anonymous Coward on Sunday August 09 2020, @06:57AM (#1033688)

      Sounds like some RUBE GOLDBERG incompetence. Blame TMB!

      • (Score: 2) by kazzie on Sunday August 09 2020, @07:39AM

        by kazzie (5309) Subscriber Badge on Sunday August 09 2020, @07:39AM (#1033695)

        How Rube!
        [/Jar-Jar]

      • (Score: 3, Funny) by The Mighty Buzzard on Sunday August 09 2020, @01:39PM (1 child)

        Well of course. You pay for incompetence, we're going to give you your money's worth. No boring old garden variety incompetence that you can find anywhere around these parts.

        --
        My rights don't end where your fear begins.
        • (Score: 2) by Reziac on Monday August 10 2020, @05:16AM

          by Reziac (2489) on Monday August 10 2020, @05:16AM (#1034214) Homepage

          Yeah, but we didn't pay for-- wait, what am I saying??

          --
          And there is no Alkibiades to come back and save us from ourselves.
  • (Score: -1, Flamebait) by Anonymous Coward on Sunday August 09 2020, @08:02AM (1 child)

    by Anonymous Coward on Sunday August 09 2020, @08:02AM (#1033701)

    Just Runaway666 trolling about the Nazis.

    • (Score: 2, Funny) by Anonymous Coward on Sunday August 09 2020, @10:12AM

      by Anonymous Coward on Sunday August 09 2020, @10:12AM (#1033730)

      Many aristarchus submissions were roasted in the depths of a Sloar that day I can tell you.

  • (Score: 1) by ealbers on Sunday August 09 2020, @10:43AM

    by ealbers (5715) on Sunday August 09 2020, @10:43AM (#1033734)

    Thanks from all of us who have stood/sleepwalked in those shoes, or worse the shoes you almost filled...or even worse.
    We really do appreciate you.

  • (Score: 3, Interesting) by RS3 on Sunday August 09 2020, @11:24AM (4 children)

    by RS3 (6367) on Sunday August 09 2020, @11:24AM (#1033742)

    Firstly thank you (all), and always, for all of your good hard work and time.

    Secondly, why does the db node need reboots? Something eating RAM?

    • (Score: 3, Interesting) by The Mighty Buzzard on Sunday August 09 2020, @01:18PM

      I expect something on the hardware at Linode that was running the VMs that got rebooted needed updated or otherwise poked with a stick.

      --
      My rights don't end where your fear begins.
    • (Score: 2, Informative) by Anonymous Coward on Monday August 10 2020, @04:11AM (2 children)

      by Anonymous Coward on Monday August 10 2020, @04:11AM (#1034184)

      We have identified and developed a number of critical stability and performance improvements for our infrastructure. These include fixes for a potential crash, general performance tweaks, and enhancements to networking throughput. After implementing, testing, deploying, and gaining confidence in these changes, we are now ready to roll updates out to the remaining fleet. This is expected to significantly decrease the amount of unplanned maintenance.

      To complete this, we will be performing reboots on a subset of Linode’s host machines. This maintenance will update the underlying infrastructure that Linodes reside on and will not affect the data stored within them.

      If you are on an affected host, your maintenance window will be communicated to you via a Support ticket within the next few days. You can prepare your Linode for this maintenance by following our Reboot Survival Guide.

      During the actual maintenance window, your Linode will be cleanly shut down and will be unavailable while we perform the updates. To minimize impact, maintenance will be limited to each respective datacenter’s local off-hours time. A 2-hour window is allocated, however the actual downtime should be around 45-60 minutes. After the maintenance has concluded, each Linode will be returned to its last state (running or powered off).

      This status page will be updated once maintenance is complete.

      https://status.linode.com/incidents/y9t56cp18c6g [linode.com]

      • (Score: 2) by RS3 on Monday August 10 2020, @04:33PM (1 child)

        by RS3 (6367) on Monday August 10 2020, @04:33PM (#1034380)

        Thanks for the info. Semi-educated but otherwise pure speculation here: I wonder if they could have a spare machine, build it up with the new hypervisor updates, and migrate customer's VMs to the new machine, minimizing downtime. Then update the machine that was just migrated from, and move VMs from another machine, update that one, etc.

        Maybe that's what they do...

        • (Score: 1, Informative) by Anonymous Coward on Tuesday August 11 2020, @09:37AM

          by Anonymous Coward on Tuesday August 11 2020, @09:37AM (#1034819)

          Depending on what exactly they are changing, they can literally transfer you from one hypervisor to another without "turning off" the VM. As far as the VM is concerned, you just get a split second of added latency to everything, some lost packets, maybe a page fault or twenty, and your clock is off by a bit until your time daemon fixes it.

          The issue is that depending on what exactly needs changing, you can't do that sort of thing for everything. Plus, they will usually charge a lot more for those kinds of up times since it takes a little bit more expertise, time, and attention to pull off in the first place. In many situations, they will do an offline transfer of machines like you suggest, but it isn't always quick or easy in itself. Often, it is fastest, easiest, and cheapest to just shut down all the VMs, do what you need to do power cycle the hypervisor, test the changes and bring it all back up. It's not like you have large nines in your uptime guarantees.

  • (Score: 2, Insightful) by tobiasdavis on Sunday August 09 2020, @11:37AM (2 children)

    by tobiasdavis (4260) on Sunday August 09 2020, @11:37AM (#1033743)

    Is being able to get production back in hours.

    Good job fellas get some sleep

    • (Score: 3, Funny) by The Mighty Buzzard on Sunday August 09 2020, @01:27PM (1 child)

      Test our backups? He's a witch! Burn him!

      Seriously though, Linode had the most recent (and easy to restore from as well) backup and it was a snapshot of the entire DB server VM that's made daily, so testing them really isn't viable unless you folks want to fork over the money for another server for backup testing and to pay someone for work as tedious as the testing of the three VMs we have them make daily backups for.

      --
      My rights don't end where your fear begins.
      • (Score: 2) by Reziac on Monday August 24 2020, @12:47PM

        by Reziac (2489) on Monday August 24 2020, @12:47PM (#1041110) Homepage

        "I'm not a production environment, I'm a basement."
            -- TechHole Productions

        --
        And there is no Alkibiades to come back and save us from ourselves.
  • (Score: 0) by Anonymous Coward on Sunday August 09 2020, @12:36PM (3 children)

    by Anonymous Coward on Sunday August 09 2020, @12:36PM (#1033765)

    Fuck... I thought it was 4 days ago and I started posting comments to the deja vu stories again.

  • (Score: 0, Troll) by jmichaelhudsondotnet on Sunday August 09 2020, @01:45PM (53 children)

    by jmichaelhudsondotnet (8122) on Sunday August 09 2020, @01:45PM (#1033804) Journal

    "the database broke" just doesnt cut it.

    We are big boys who can actually understand log entries, why would you not post them?

    Hearing about how things break and why is one of the funnest parts of being a sysadmin, the noobs get the "the mcguffin broke and i fixed it" response, not us.

    We should all learn from how things like this happen to prevent it within our own zones of responsibility.

    I for one love hearing about chain reaction catostrophic failure, in hindsight, this has happened before and you always leave out the fun stuff.

    Also if this were not a legit site and the code was being rejiggered to censor something or prevent something from going viral in real time, this is what it would look like. I have seen this happen at numerous other sites, when things get hot, just shut it down, declare anomalous event, restart when it has cooled.

    If everything is on the up and up, what is there to hide?
    (predicted response: im lazy, tired, you are paranoid, it doesnt matter, or other cop outs, prove me wrong )

    In the big picture we are facing a global network stucture where any idea can be interdicted in real time, at the dns/ip level, and where the only options we have on what to believe are state propaganda, so this discussion is not trivial from any perspective that cares about idk truth, history, and decentralized systems resistant to censorship.

    • (Score: 5, Insightful) by janrinok on Sunday August 09 2020, @02:13PM (13 children)

      by janrinok (52) Subscriber Badge on Sunday August 09 2020, @02:13PM (#1033815) Journal

      When guys are giving freely of their own time you get whatever you are given.

      The data was backed up, it was recovered up to that backup, and martyb has reissued the handful of lost stories. If you want to read the logs then why don't you volunteer to join the team?

      • (Score: 2) by jmichaelhudsondotnet on Sunday August 09 2020, @02:50PM (12 children)

        by jmichaelhudsondotnet (8122) on Sunday August 09 2020, @02:50PM (#1033833) Journal

        Would you have me? Where do I sign up?

        • (Score: 2) by janrinok on Sunday August 09 2020, @05:37PM (10 children)

          by janrinok (52) Subscriber Badge on Sunday August 09 2020, @05:37PM (#1033900) Journal

          I have no idea if any of the systems guys would want you - if you have to ask then maybe you are not the sort of person who can do the job. Join us on IRC and chat to one of the systems guys [soylentnews.org].

          • (Score: 1) by jmichaelhudsondotnet on Tuesday August 11 2020, @06:27AM (7 children)

            by jmichaelhudsondotnet (8122) on Tuesday August 11 2020, @06:27AM (#1034778) Journal

            First question, do you know what the "you cannot post to this page" error is? Would thee be a way to get this message improved so that it says some instructions for solving it or justification? I had 25 karma, this was around 9pm utc last night.

            After some thought, the real reason I would not join the admin team is that I do not trust buzzard. I have spent a lot of time trying to have a reasonable discussion with him and he simply does not deliver.

            It goes like this:

            Me: "I think A, B, and C are critical facets of this issue, and evidence D, E, and F are important to understanding what is going on. What do you think about that?"

            Buzzard: "Your problem is you are going about it all wrong and if you werent so X then maybe people would take you seriously"

            Over, and over and over. Every single issue he makes it about me and/or some problem I have, never, ever will he talk about an issue. It is so exhausting and pitiful. I dont read his comments anymore.

            And I have an absolute 100% trust requirement for sharing credentials so we are not anywhere near that threshold. I have gladly lost jobs and lots of money over this, and been thrown under the bus when I disregarded my better judgement.

            Consider even in this case, what should be a simple af issue. I say A the incident report is significant B we are all admins here C it should be fun and D a lot of sites I have been at use downtime as a form of censorship. At least 3 people agree with upvotes, and 3 people say *this is trolling when it in no way is anything near a troll post* Disagree all you want but this is not the correct use of the word.

            And frankly it happens a lot here and when I complained about it to buzzard?

            "Dont be so butthurt" and that was the last time I bothered with IRC. Can you see why?

            Buzzards response to my simple, non acrimonious request, "how dare you disrespect my free volunteer labor and then ask for people to buy your book or contribute to your projects"

            MartyB Response "I am clutching my pearls at how you dare to insult buzzards free labor, we have been toiling here every day since before you were born"

            Do you see? The main question, "what happened and why" is just out the window, the entire issue as far as they are concerned is a personal problem with me.

            And so I cant take them seriously, and frankly your response to this comment will tell me quite a bit about how serious you are about the actual values of this site.

            I cannot stress this enough *constantly redirecting security questions back on the asker is a primary tactic of cop infiltrators and psych trolls*.

            Full stop.

            See my journal post or response to marty for full explanation. Read this if you are actually a civilian and dont just want to be a controlled opposition stooge: https://archive.is/u4d69 [archive.is]

            I have bled to learn these things and wasted much of my life trying to advance my economic interests or political beliefs in complete futility because no one the united states had a secret political repression police force that was worse than the stasi or kgb.

            Disbelieve me at your peril, or alternately if you are in on it or condone it, you had best hope there is no afterlife.

            • (Score: 5, Informative) by janrinok on Tuesday August 11 2020, @12:34PM (5 children)

              by janrinok (52) Subscriber Badge on Tuesday August 11 2020, @12:34PM (#1034865) Journal

              First question, do you know what the "you cannot post to this page" error is?

              It would help if you told us where you got this response, please. However, it is telling you that you cannot write to that particular page. Until you provide more information I cannot provide you with a better answer.

              After some thought, the real reason I would not join the admin team is that I do not trust buzzard.

              That is your prerogative. However, you might be making the same mistake that others have made. Everybody in this community - including the staff - are entitled to express their personal opinions in comments or on IRC. Do not mistake that as someone representing the site as a whole unless they are speaking on behalf of the site. We normally make it clear when we speak for the site. Nevertheless, our personal view and opinions have absolutely no bearing on how professionally we act in maintaining this site. If any member of staff started to force his views onto the running of the site he would soon have the rest of the team at his throat. And, although it isn't visible to you, disagreements happen and are always resolved by discussion or common sense.

              Buzzards response to my simple, non acrimonious request, "how dare you disrespect my free volunteer labor and then ask for people to buy your book or contribute to your projects"

              No it wasn't. That comment [soylentnews.org] was made by an AC.

              Do you see? The main question, "what happened and why" is just out the window,

              Can you read? Try here [soylentnews.org].

              If you seriously believe that anything that happens on this site is because of our nefarious activities then you should most certainly leave - if I thought it were true I would go. The last time somebody accused us of maliciously removing his comments there was a 2-week long investigation and a full report was published on this site. We could prove beyond reasonable doubt that the missing comments were never received by this site - not that we had subsequently deleted them - but they never even reached our server. That is how seriously we take such accusations. We are not trying to control what is said, censor anybody who says something with which we disagree, or support any political view. If you think that we are then please leave.

              You can express any viewpoints you wish - but you cannot demand that the community must agree with you.

              Now I suggest that you go back and think about what has been written here and in the rest of this thread. If you don't like it just leave - we cannot nor will not stop you. But you will treated like any other member of our community if you decide to stay - 'Thems the rules'.

              • (Score: 1) by jmichaelhudsondotnet on Wednesday August 12 2020, @08:42AM (4 children)

                by jmichaelhudsondotnet (8122) on Wednesday August 12 2020, @08:42AM (#1035449) Journal

                Yawn.

                Me: just want to know why site went down, this time, again, and learn more about linux, and confirm everything is on the up and up, and it seems I am getting quite a lot of malicious downvotes that abuse the voting mechanism offensively, and wth with this "you cant post to this page" error

                You and several people: the problem is all with you, see we can nitpick you here, and nitpick you there, and nitpick you everywhere and whatabout the whataboutism and how dare you and who do you think you are, and this one time we really did provide an incident report but it turned out to be nothing so it applies to this case, and if you are so cool why dont you do our work for us even though we are lying to your face and being dicks, and you are selfish even though most of what you are writing is in defense of the rights of other people and warning of actual danger, and this stack of logical errors and assholery at the end of the day means you are the one being the asshole.

                Anything else to add to that stack of logical fallacies and passive aggressive douchebaggery and diversion from the topic, and insinuation that I deserve abuse?

                I was hoping for more than an extended "muddy the water and fling more shit we would love it if you left" response. Or that you could even say what the sites error message means, when you are the admin. Or admit that the error message is itself incompetent microsoft level error handling.

                But here we are, with you and the admin team just sinking lower into my estimation. smh

                • (Score: 2) by janrinok on Wednesday August 12 2020, @06:46PM (2 children)

                  by janrinok (52) Subscriber Badge on Wednesday August 12 2020, @06:46PM (#1035707) Journal

                  you could even say what the sites error message means

                  If you are claiming that you cannot post to this thread, then how did you just post the previous comment?

                  If you cannot post somewhere else - tell me on which page you are seeing the problem. We cannot help you if you don't tell us what you are trying to do, what you are actually seeing and which page you are seeing it on.

                  You and several people: the problem is all with you, see we can nitpick you here, and nitpick you there, and nitpick you everywhere and whatabout the whataboutism and how dare you and who do you think you are, and this one time we really did provide an incident report but it turned out to be nothing so it applies to this case, and if you are so cool why dont you do our work for us even though we are lying to your face and being dicks, and you are selfish even though most of what you are writing is in defense of the rights of other people and warning of actual danger, and this stack of logical errors and assholery at the end of the day means you are the one being the asshole.

                  Anything else to add to that stack of logical fallacies and passive aggressive douchebaggery and diversion from the topic, and insinuation that I deserve abuse?

                  I suspect that ranting like this is why you are being moderated down. You claim to have written books - write something comprehensible here and perhaps we can converse intelligently with you.

                  You wanted to know what had caused the site outage - if you read the link that I provided it answers the question that you asked.

                  just want to know why site went down, this time, again, and learn more about linux

                  Then stop rambling on and read what people are saying. The fault has been explained. If you don't understand the explanation then say so and ask (politely) for someone to explain it again to you.

                  But here we are, with you and the admin team just sinking lower into my estimation.

                  I hope that you eventually find a site that suits you better.

                  • (Score: 1) by jmichaelhudsondotnet on Thursday August 13 2020, @02:09PM (1 child)

                    by jmichaelhudsondotnet (8122) on Thursday August 13 2020, @02:09PM (#1036137) Journal

                    I went back and saw buzzards explanation, wow, that is all I wanted. It should ideally be placed in the original post about the event, not nested way down here.

                    The "You cant post to this page" error happened, surprise, in this very thread, on monday night UTC. That is the best I can do, but it has happened before, usually after I have replied to one or two comments.

                    Next time I see it I will give you the exact incident report and I can see the irony in this, on monday night I was frustrated and just had to set it down. I had written a few paragraphs I had to throw out.

                    Did you know ranting is an actual art form, and that in the course of history entire famous books have been published and enjoyed that uset his form? As well as comedy and drama?

                    If you cant see my point, that you cast aspersions on me when at first I wasnt even being angry or accusatory, at all, is really a problem with You. If you are actually trying to be sincere that is, something I am far from convinced of. It is simply *the most common* troll tactic on the internet. If you cant win the argument, attack the person. If you cant censor them outright, demoralize them until they go away.

                    If you think this is fitting of the leader or admin of a community, maybe a lot of the negativity here is a rot that is coming from the head of the fish.

                    And of course you ignore the pile of innacurate and offensive mod points I have gotten. If it were one, and that is frequently the case, I would overlook it, but this week I have gotten about a dozen posts that should be disagree but which were troll or flamebait. Which is the other top tactic of mods that are running a site as a pschyological operation for the government, ignore the abuse that actual trolls throw on well-meaning people, whose ideas you would like to censor but cannot without revealing the true nature of the site.

                    And on top of that the "if you dont like it you can leave" or "my way or the highway" repeatedly encouraging me to just leave as a solution to a problem that should be yours, not mine. That is not what a community leader does, he should be trying to bring people together who contribute, not showing people the door at every opportunity.

                    Am I not just talking about the values of a linux based community of people who want fair tech? More eyeballs, less bugs. More openness, less secrets. More accountability, less black boxes.

                    I mean if you want to kick out the ideas of richard stallman and the core linux philosophies, you should let us know, and then I may have to decide to go somewhere else. But as long as my work gets seen here by some people who ostensibly have a sane attitude towards tech and systems, I will be here.

                    Thanks though for your work, any of it that you do in earnest, however much or however little that may be.

                    Oh one more thing, redditors are constantly clamoring for why there is no concerted movement against elite child abusers and epsteins victoria secret network, but sadly none of them are aware of my work and due to outright censorship there, I cant tell them about it,

                    https://archive.is/10qKh [archive.is]

                    So maybe if they knew about my domains someone would help me with them and their costs, or be more interested in my books. So even if it seems like everyone here wants me to shut up, anything you do to silence me is *an actual aid to people who harm children systematically on planet earth.* Same goes for this idiotic BLM solving racism with more racism balogna, the books that I wrote long before these events address these societal tensions directly, as well as the way they are steered by police agent provocateurs into utter idiocy, burning your country to the ground and hurting countless people for no discernible reason other than population control by those who clearly want to see your country in flames.

                    If you sleep well on that side of things, says a lot about you.

                    • (Score: 2) by janrinok on Thursday August 13 2020, @05:38PM

                      by janrinok (52) Subscriber Badge on Thursday August 13 2020, @05:38PM (#1036238) Journal

                      I went back and saw buzzards explanation, wow, that is all I wanted. It should ideally be placed in the original post about the event, not nested way down here.

                      There is a lesson there - read all the comments. The answer was posted 4 days ago and I pointed it out to you 2 days ago. They are not always immediately under where you might think because he was giving the explanation to somebody else who had asked in a polite manner, having let him get some sleep first.

                      Did you know ranting is an actual art form, and that in the course of history entire famous books have been published and enjoyed that uset his form?

                      Moderating down anybody who rants on this site is another art form. You have been educated.

                      that you cast aspersions on me when at first I wasnt even being angry or accusatory

                      "the database broke" just doesnt cut it. We are big boys who can actually understand log entries, why would you not post them?

                      We disagree - TMB was sleeping after being up all night fixing the problem. You should learn to be patient and avoid sarcasm in the future.

                      And of course you ignore the pile of innacurate and offensive mod points I have gotten.

                      If people don't like you then there is nothing I can do to help you. Try changing how you treat other people and you might be surprised how things improve for you.

                      And on top of that the "if you dont like it you can leave" or "my way or the highway" repeatedly encouraging me to just leave as a solution to a problem that should be yours, not mine.

                      You were the one who claimed that you did not like this site because of the way you felt you were being treated. If you do not like this site why would you stay? You suggested that we were trying to censor views with which we did not agree - you even sent me diagrams showing how censorship works. Martyb answered that accusation [soylentnews.org] most eloquently. You are welcome to remain in this community but you have to earn respect here - you cannot just demand it.

                      Thanks though for your work, any of it that you do in earnest, however much or however little that may be.

                      Your thanks are appreciated. How much have I contributed? You can judge that for yourself: from my User ID (52) intro: "Linux User: 380191. Former Editor-in-Chief on our site (2016-2018). Programming experience: Real-time avionic systems, Algol, CORAL66, C, C++, Python. For fun: Computing and technology. I also play guitar, banjo and tin whistle." I have, additionally, edited over 5000 stories on to the front page. TMB, martyb and I (and many others) have been here since before the site went live. I would imagine that anybody with a userid of less than 200 or so has been here from the very beginning, many working in the background to keep this site going. (No offence intended to those who have been here from the start with userids greater than 200!)

                      See you down the road...

                • (Score: 0) by Anonymous Coward on Thursday August 13 2020, @04:02AM

                  by Anonymous Coward on Thursday August 13 2020, @04:02AM (#1035982)

                  https://testlio.com/blog/the-ideal-bug-report/ [testlio.com]

                  You should also note that without STR or ENV data, you might as well not even report one because no one would be able to track it down.

            • (Score: 3, Insightful) by The Mighty Buzzard on Tuesday August 11 2020, @09:52PM

              by The Mighty Buzzard (18) Subscriber Badge <themightybuzzard@proton.me> on Tuesday August 11 2020, @09:52PM (#1035195) Homepage Journal

              the real reason I would not join the admin team is that I do not trust buzzard.

              Funny, that's the exact same reason I wouldn't be in favor of letting you join it.

              --
              My rights don't end where your fear begins.
          • (Score: 1) by jmichaelhudsondotnet on Tuesday August 11 2020, @06:41AM (1 child)

            by jmichaelhudsondotnet (8122) on Tuesday August 11 2020, @06:41AM (#1034780) Journal

            Also sorry for double post after a dramatic full stop, but in my last 25 notifications there are over 10 spurious spam/troll/flamebait mods.

            And if you recall I had my mod privileges revoked for a single misclick on the spam mod button, which was admitted to be an interface bug.

            And in the context, it appears to me in all of these instances someone wants my viewpoints slid and minimized in the thread view.

            I would not complain if it were simply disagree, which is what I think these people would say if they were for real.

            Using the spam/troll/flamebait mod as an offensive info weapon against those you disagree with should not be allowed, and combined with selective enforcement of this is a known tactic to drive away people you want to silence.

            Are you aware of this dynamic, which has been noted at just about every forum site? And that this is one of the last places I can even post without outright censorship? And that my views on these matters are basically centrism circa 1990?
            https://archive.is/SoybE [archive.is]

            Also relevant:
            https://archive.is/uZI5P [archive.is]

            Your response will be telling, I look forward to it.

            • (Score: 2) by The Mighty Buzzard on Tuesday August 11 2020, @09:55PM

              by The Mighty Buzzard (18) Subscriber Badge <themightybuzzard@proton.me> on Tuesday August 11 2020, @09:55PM (#1035198) Homepage Journal

              You mean the emails that you made no claim of it being a misclick until several email exchanges had taken place and we'd said that it would have been revoked had it been a misclick? Yeah, excuse us for being dubious.

              --
              My rights don't end where your fear begins.
        • (Score: 3, Insightful) by The Mighty Buzzard on Sunday August 09 2020, @06:03PM

          I'd personally give that an emphatic "no". I'd really rather not have a paranoid schizophrenic who refuses to so much as consider medication (or even just talking to a shrink) having root access to our servers and the master password list.

          --
          My rights don't end where your fear begins.
    • (Score: 1, Insightful) by Anonymous Coward on Sunday August 09 2020, @04:28PM

      by Anonymous Coward on Sunday August 09 2020, @04:28PM (#1033865)

      I fixed things for decades. It's best for all to just say "It broke"

    • (Score: 3, Insightful) by kazzie on Sunday August 09 2020, @04:59PM

      by kazzie (5309) Subscriber Badge on Sunday August 09 2020, @04:59PM (#1033880)

      The explanation for the lack of detail is in TFS:

      it was already four hours past my bedtime when I found out about the problem. My brain is no work good anymore.

    • (Score: 5, Insightful) by martyb on Sunday August 09 2020, @05:32PM (13 children)

      by martyb (76) Subscriber Badge on Sunday August 09 2020, @05:32PM (#1033895) Journal

      The site went down 2 hours after TMP was normally in bed. Not only did he isolate the problem and get things back up and running (while fighting off his body's demand for sleep), he even wrote and posted a story to tell everybody what happened.

      In his spare time.

      Having never been paid even one penny for all his efforts here.

      And you have the temerity to try to cast this as some kind of conspiracy

      to keep things from the community?

      I see you are new here (account created on 2019-05-27). Like TMB, I have been here from before the site even went live.

      In the early days (and weeks and months) of the site, crashes and outages were a regular occurrence. It is a testament to all the hard work that went into taking ancient, unsupported code dependent on back-level versions of things like apache, mod-perl, nginx (and many others) to the point that an outage is a relatively rare occurrence. Extremely well funded companies like Microsoft, AWS, Google, Twitter, etc. still have outages. So I'd say we're doing pretty darn well running on a shoestring budget. (It costs about $20 per day for everything to run this site for a day: servers, labor, taxes, incorporation fees, CPA, everything!)

      I've TMB put countless hours into this site. He is the one that implemented Unicode (international character sets, emoji, and the like) support just a few months after go-live. As the one who tested the implementation, I can attest that it was no small feat! Then when it was taking longer than he liked to render a story that had lots of comments on it, he completely refactored the code to make things faster. Yeah, that was a fun one to test. :/ And on those few occasions when the site goes sideways, and it exceeds my limited skills at rectification, it generally falls on his shoulders to get things going again.

      And I see that you have never, not even once, subscribed [soylentnews.org] to help support the site financially.

      This is the most strongly-worded comment I have ever posted to the site -- it takes a special level of tactlessness and hubris to accomplish that.

      So, maybe, try saying "Thank You" for what you have already freely received before even beginning to think about asking for more.

      --
      Wit is intellect, dancing.
      • (Score: 2, Insightful) by Anonymous Coward on Sunday August 09 2020, @05:59PM (6 children)

        by Anonymous Coward on Sunday August 09 2020, @05:59PM (#1033911)

        Oh, the irony. JMH uses SN to complain that nobody is financially supporting him [soylentnews.org] but he complains about a free service he receives and is apparently unwilling to pay it forward. It takes a very special type of person to complain about a service he receives for free, has not donated to, and yet complains that nobody is donating to his work.

        I've only interacted with you and TMB by email when I've wrote about site questions and issues, but my experience is that you've both been very patient and polite with me. I know I can be a pain in the ass to deal with, but I do appreciate the work you guys have done.

        • (Score: 3, Insightful) by The Mighty Buzzard on Sunday August 09 2020, @06:07PM (3 children)

          It takes a very special type of person to complain about a service he receives for free...

          Ya think? I considered it one of the most in line with the rest of humanity things he's ever said. Mind you, our community tends to be a cut above the rest of humanity in many aspects in my personal opinion.

          --
          My rights don't end where your fear begins.
          • (Score: 3, Insightful) by Reziac on Monday August 10 2020, @05:21AM (2 children)

            by Reziac (2489) on Monday August 10 2020, @05:21AM (#1034218) Homepage

            That's what happens when you lead by example, bein' yerselves a cut above.

            Thanks for keeping the lights on.

            --
            And there is no Alkibiades to come back and save us from ourselves.
            • (Score: 3, Insightful) by The Mighty Buzzard on Monday August 10 2020, @03:18PM (1 child)

              I figured it was caused by us self-selecting to come here from /. for the reasons we did.

              --
              My rights don't end where your fear begins.
              • (Score: 2) by Reziac on Monday August 10 2020, @03:44PM

                by Reziac (2489) on Monday August 10 2020, @03:44PM (#1034360) Homepage

                Heh... on the rare occasions I visit the Green Site, I have similar thoughts... starting with, " *what* are they drinking??"

                --
                And there is no Alkibiades to come back and save us from ourselves.
        • (Score: 1) by jmichaelhudsondotnet on Thursday August 13 2020, @01:48PM (1 child)

          by jmichaelhudsondotnet (8122) on Thursday August 13 2020, @01:48PM (#1036124) Journal

          This is a silly interpretation of my work, and events.

          It is not called "support" when an author is selling their work, or when an activist asks for members of a community to assist with a project.

          I am not asking for charity, I am saying, why the fuck are none of you doing anything about the stated values this community has being attacked on all fronts, to the extent they may disappear.

          Also, by contributing reams of content to this site, that is a form of payment in the form of labor. One of the primary failures of the current iteration of the internet is that our work filling it with interesting content is considered charity, while the people who simply own the infrastructure and had first mover advantage due to timely investment, accrue all of the financial rewards of our work.

          Every single rioter in chicago actually does deserve a piece of the internet fortunes of bezos and gates, and has a reason to be angry that all of their posts, which serve their data harvesting and marketing engines perfectly, do not generate income for those who did the labor and made the contribution.

          That you cannot see the injustice and theft taking place here, which directly affects especially people like myself who do not only shitpost but put entire reams of content online, at risk to ourselves even due to the political dicey nature of our times, says a lot about the extent of your ignorance and inability to see anything other than the tiny portion of the system that is visible to you.

          The rest of us who can deduce and infer and analyze larger systems and see them for what they are, are getting quite tired of having to explain things to people like you, as you insult us for being the ones demanding free stuff.

          You should rethink your life, this comment of yours betrays epic and willful ignorance and more than a little stupidity, and worse, you are calling the person who is trying to help you get your ostrich head out of the sand the asshole.

      • (Score: 2) by Reziac on Monday August 10 2020, @03:47PM

        by Reziac (2489) on Monday August 10 2020, @03:47PM (#1034362) Homepage

        +10, Underrated. All of you in the back room there, keepin' the wheel a-spinning.

        --
        And there is no Alkibiades to come back and save us from ourselves.
      • (Score: 1) by jmichaelhudsondotnet on Tuesday August 11 2020, @05:55AM (4 children)

        by jmichaelhudsondotnet (8122) on Tuesday August 11 2020, @05:55AM (#1034763) Journal

        1/2

        "predicted response: im lazy, tired, you are paranoid, it doesnt matter, or other cop outs, prove me wrong"

        actual response: "how dare you!" -clutches pearls- so the "other cop out" category FTW

        Color my surprised Marty, the entire last year you have repeatedly told me you do not have time to consider posting stories I submit and now you have time to clutch your pearls that I dare ask for a real incident report when the site went down inexplicably and 6 hours of peoples ideas disappeared.

        Would you like me to forward you those emails where you said everyone on the admin team is just so overloaded with work that you cant consider my contributions? Not that you dont think they are appropriate, but that you dont have time to click a button so my submissions slide to the bottom of the queue, and now you have time to pound your chest at length at how dare I ask for an explanation.

        Nor do you seem to be concerned that in the last 48 hours I was modded troll about 10 times for posts that are not in any waay troll posts. Do you know what words mean or are you locked in a chinese room? Have you seen the number of actual troll responses I have to put up with(and do with grace)? Am I allowed to call you "butthurt" as the buzzard said to me when I one time tried to protest in the IRC?

        Do you see the incongruity? Have you ever heard of the word "fair"? What if I called you and the buzzard butthurt in this situation, how would that feel? Have you ever heard of the concept "double standard"?

        Are you certain that you have the moral high ground here? I do not think you do. I do not see any reason to be offended that I simply asked for a real explanation, or to call me to the carpet for not having the financial wherewithall to contribution to the sites operation. Kindof flattered you looked up my signup date to buttress your arguments.

        If you must know I was in "trial" employment at that time and that job didnt work out. That was about when they started talking about *banning encryption* in the united states, which is an actual big fucking deal, in contrast to this issue which should be trivial. So I have been on unemployment, then the virus lockdown hit, and this has been a pretty fucking difficult time for me, so no I don't have 20 dollars when 90% of the people on here have good paying jobs, and despite me demonstrating that I am one of the most original posters on these topics, job offers are not forthcoming. Nor does anyone here seem to have one ounce of pity, empathy or concern of my reports that for stating my beliefs I have encountered nightmarish political repression. Part of which is the constant insults and aspersions against my sanity I encounter here, which you also seem to think is completely fair.

        All of that outright weaponized dickishness, and I am the asshole here, for asking for an incident report and hurting poor buzzards feelings when he sacrifices his precious retirement hours for the cause. I get troll brigaded mod, you get insightful brigaded plus 5. (though at least 3 other people upvoted my comment for insightful or interesting, so I am not alone here) Nor was my post in any way acrimonious, I checked. Sysadmins provide real reports when things go wrong, it should be fun, but you doth protest....so very much...

        Is the worlde turned upside downe? Fact of the matter is I have been doing actual work and I am one of the foremost experts on forum manipulation due to 10+ years at reddit, where my work is actually being erased, and none of my messages to other users were recieved, and every post I made got -10 downvotes within 10 minutes. I was censored at hackernews for daring to question mozilla, almost immediately. Twitter silos me, no one sees my posts. Other alt sites like saidit and raddle are the same, oddly with the same pearl clutching responses from the mods when I question their decisions. And outright gaslighting, when I respond to how insane the troll responses to me are, they say, just like you just did, that i am the asshole. And even /pol/ where the wildest nonsense and vile racism is allowed, erases my attempt to share my story and work within an hour. I came here for a potential alternative to censorship due to the obscurity of this site and that people here *claim* to have a strong belief in the fundamental principles of linux and writings of Richard Stallmman, among other things. Now I have to admit I am experiencing forms of censorship here and if you did care about this site so much as you claim to, having devoted 10 years of your life to it, you would take what I am saying very seriously. We will see.

        I am calling you into question on these values and if you do not see why then maybe you do not belong on the admin team of a site that claims to have them.

        • (Score: 1) by jmichaelhudsondotnet on Tuesday August 11 2020, @05:59AM

          by jmichaelhudsondotnet (8122) on Tuesday August 11 2020, @05:59AM (#1034764) Journal

          2/2

          You are being at best naive, at worst not arguing in good faith, so I find it difficult to take you seriously. This is the moment at which I begin to shift from being 60% certain this site is a fake honeypot to 90% certain at least 90% of you are here for some other reason than to have fair discussion. You seem to be unaware that sites going offline at odd times is a *major* tactic of censors, which allows them to control the viral spread of ideas they do not like. The same happened a year ago around the epstein death and my protests, and your response to my call for logs was very similar. The same happened *to the entire internet* in october 16 when a certain set of emails were leaked and vans were parked outside the ecuadorian embassy in London. I could go on and on.

          As a writer, someone who is trying to sell books, and who may not be able to have a normal job because when you tell the truth today it angers very, very powerful interests who do not play fair and wage 5th gen(or plausible deniable douchebaggery) warfare. Places where I work are ddos'd and receive phone harassment among other things, this sort of censorship is devastating. I have received threats of physical violence, at this site as well. (patella whack?) Far more of a life problem than having to answer mere questions about incident reports, when that is in the definition of your job as a sysadmin.

          Are you aware that last night when I was trying to respond to you I got the "cannot post to this page" error again, despite having 25 karma, and relogging? Is there some reason that is not a bug? Is there some resaon this error message is not descriptive? It may just be a bug but it is far from the only questionable thing going on here on the side of the admin team.

          This sadly fits a pattern of the "apples and oranges" form of censorship, described by someoe else in this excellent meme: https://archive.is/SoybE [archive.is] Is this the first time you have heard of this concept? Can you see how dangerous this is to the functioning of debate in human culture on earth?

          If you want to silence someone without direct censorship, you leave that person open to abuse, neglect and incompetence. You do not care that I am falsely modded, harassed. You neglect my posts and requests for fair modification. You ban me from modding for an entire month for a glitch that made me falsely mod someone spam, for some odd reason I am unable to post at random times without explanation. You insult me and say how dare you when I ask about it, repeatedly. q.e.d.

          As a final word, consider this exerpt from *my book* on the subject of security culture in civilian group, and how you are, one way or another, failing a major test of trustworthy community leadership:

          "Anti-security Culture – Infiltating agents will mock, subvert, reject any attempt at actual security and in every case argue to weaken security or act in such a way that security is undermined. In every case, agents will pretend that infiltration programs such as Cointelpro do not exist, and will rapidly change the subject or exit the situation. I have seen this on numerous, numerous occasions.

          Checklist Question – Do they respect and see the need for security culture, and contribute to the safety of the group, or do they undermine it intentionally or by ‘accident’?

          Checklist Question – Is the person aware and apologetic for theirIdentifying Police State Infiltration in Civilian Organizations disruption, or do they respond indignation and acrimony?"

          You fail both checklist questions, badly. So no I do not feel bad about asking for logs or pointing out the, at best, incompetence of downtime like this, or the potential avenue for censorship this could be. And I question your intentions as an admin of this site.

          Defend your actions or prove me right. At least I am glad now you have demonstated without doubt you have the time!

          Your response will tell me everything I need to know about how much you care about digital liberty and the trustworthiness of you and this site going forward. Is this just douchebaggery or is it a form of psych warfare? That is the only question that remains to be answered. All I asked for was a real incident report, but you had to be the first to make a personal attack on me. Clutch your pearls and spout "who do you think you are" all you want, but I have spent probably a larger amount of time contributing to this site than you have in the last 18 months(https://archive.is/FHpOa), and what I have to say is more interesting and critical to the dire issues at hand, by any measure. Feel free to word count our total comments and postings, maybe I am in error on this but I am not afraid to see the results, are you?

          I apologize for nothing. I think you should rethink your life if this is the best youve got.

          Full checklist, and link to free download of cited book:
          https://archive.is/u4d69 [archive.is] checklist for infiltration
          http://gen.lib.rus.ec/book/index.php?md5=5666F69751D2A84ECC3C74A2AC9F0C56 [lib.rus.ec] Mental Self Defense and Reverse Social Engineering against the Strategies Tactics and Psycho Bullshit of Modern Totalitarianism

          https://archive.is/2M0aT [archive.is] 7 step shuffle of controlled opposition
          https://archive.is/q5343 [archive.is] you do not fit in here
          https://archive.is/fYnfb [archive.is] new propaganda model
          This: https://0bin.net/paste/nkYnMBeeQPS3T4ZL#n1N2IqnDMPo-IU9ubOeSdi9yYuAxGjggFkCS++yCDA1 [0bin.net]

        • (Score: 3, Insightful) by janrinok on Tuesday August 11 2020, @08:00AM (2 children)

          by janrinok (52) Subscriber Badge on Tuesday August 11 2020, @08:00AM (#1034800) Journal

          you do not have time to consider posting stories I submit

          It's not just that martyb - or any other editor - does not have the time, it is that your submissions do not have the quality.

          ask for a real incident report

          Go start your own site and you can have loads of incident reports to read. Oh, you have - https://jmichaelhudson.net/. [jmichaelhudson.net]

          If you must know I was in "trial" employment at that time and that job didn't work out.

          Colour me unsurprised.

          All of that outright weaponized dickishness, and I am the asshole here

          Well, we can all agree on that statement; you are the asshole. When you make yourself available 24hrs a day for free you can come back and lecture the staff here. Until that time, I suggest that you shut up or you go find another community that more closely meets your needs. Isn't your own site doing as well as you had hoped? Still it must give you some logs to read, so ours aren't too important for you.

          though at least 3 other people upvoted my comment for insightful or interesting, so I am not alone here

          No, they didn't. You had 1 x interesting, 1 x insightful, 1 x overrated, and 2 x troll. Don't make 'facts' up that you cannot substantiate.

          Nor was my post in any way acrimonious

          No, but you assumed that the staff are here to serve you - we are not. We voluntarily serve the community but not necessarily individuals who think that they can just demand things. If you had bothered to look on IRC, or waited until the comments came from those who had been up all night sorting the problem out, you would have realised that nobody had had any time to produce a report of any kind. We explained that there had been a problem, and what measures had been taken to fix it. They were then sleeping after a difficult night's work. And as your 'support' to this site appears to be only in your self-centred comments you are not really in a position to ask for anything more.

          I am one of the foremost experts on forum manipulation due to 10+ years at reddit...

          Citation please - I am unable to find your Wiki entry. Do you mean 'foremost expert' in your street or apartment block?

          ... where my work is actually being erased, and none of my messages to other users were recieved, and every post I made got -10 downvotes within 10 minutes. I was censored at hackernews for daring to question mozilla, almost immediately. Twitter silos me, no one sees my posts. Other alt sites like saidit and raddle are the same, oddly with the same pearl clutching responses from the mods when I question their decisions.

          I'm seeing a pattern here - have you thought that it might not be everyone else that is wrong - perhaps it is you? The moderations are done by the community, are you also questioning their decisions? They disagree with you, hard luck.

          • (Score: 1, Touché) by Anonymous Coward on Tuesday August 11 2020, @08:33PM

            by Anonymous Coward on Tuesday August 11 2020, @08:33PM (#1035142)

            I bet he has zero incident reports. Given the horrible configuration of that website, I doubt he'd have the tools in place to notice any problems.

          • (Score: 2) by Reziac on Monday August 24 2020, @01:09PM

            by Reziac (2489) on Monday August 24 2020, @01:09PM (#1041115) Homepage

            Upon inspecting the indicated site, I note that the proprietor's life goal is to emulate Marx's career; which is to say, to never need to work.

            --
            And there is no Alkibiades to come back and save us from ourselves.
    • (Score: 4, Informative) by The Mighty Buzzard on Sunday August 09 2020, @05:43PM (18 children)

      It sure as hell does cut it when I'm still up at 1AM and I should have been in bed around 9PM.

      If you really want to know, one of our ndb nodes got rebooted and our init scripts for the ndb nodes specifically do not include the --initial argument in case they're the first node to come up. Which means it had a 5 Aug version of the db and could not connect to the ndb pool until someone starts it with the --initial argument. This is all fine and good, we can run off one node just as well as two for without even noticing and for as long as necessary. Right up until that one node crashes.

      When that happens, the 5 Aug version looks and sees that it's not fighting a different version and starts loading itself up. Since this takes quite a bit longer than googling up the error code that the 8 Aug node throws when you try to restart it with an earlier version of the db loading on the 5 Aug node, you end up finding the answer "You need to start it with --initial" before the 5 Aug node is done loading and you can see that it contains an out of date version. Which it turns out is about the worst possible answer you could find, because it tells the 8 Aug node "Screw our local data, pull a version from the other node and consider it current".

      So then you have to shut both nodes down, restore one from backup, get the ndb stuff running on it, and start the other node up with --initial. Which is what I did, with a system image backup Linode makes for us every night since it was the one with the least time lost.

      tl;dr Schedule your unanticipated crashes for when you're not nodding off as you troubleshoot && mysql-cluster is kind of a piece of shit if there are common circumstances where an out of date node can't automatically connect itself back to the pool and update from it.

      --
      My rights don't end where your fear begins.
      • (Score: 2, Funny) by RandomFactor on Sunday August 09 2020, @07:09PM

        by RandomFactor (3682) Subscriber Badge on Sunday August 09 2020, @07:09PM (#1033959) Journal

        1) Thanks TMB :-)

        2) I happen to appreciate the technicalities myself, but not five hours after bedtime, f..orget that. Still if/when you take the time to write up the meta stuff, we'll take the time to at least misread the summery :-)

        --
        В «Правде» нет известий, в «Известиях» нет правды
      • (Score: 1, Insightful) by Anonymous Coward on Sunday August 09 2020, @11:45PM (1 child)

        by Anonymous Coward on Sunday August 09 2020, @11:45PM (#1034077)

        Ignore them. Don't feel bad, the stresses are getting to all of us. I have printed checklists to keep me from fucking up, and boy did I fuck up recently. Fucked up enough that I offered to take the financial hit.

        It happens, and one event does not define a sysadmin. These are difficult times, and I extremely appreciate all of your hard work, and the explanation of what went wrong.

        -- Ed 791

        • (Score: 2) by The Mighty Buzzard on Monday August 10 2020, @04:41AM

          No worries, I ain't stressed about him. I was raised with good manners but also an outstanding command of English profanity. My time in the Army only honed that skill to a razor's edge. If tone of voice came through in text a lot less people would think it means I'm wound up when I say "fuck"; it's mostly just punctuation to me.

          --
          My rights don't end where your fear begins.
      • (Score: 2) by coolgopher on Monday August 10 2020, @04:34AM (2 children)

        by coolgopher (1157) on Monday August 10 2020, @04:34AM (#1034193)

        Thanks TMB, and good job on the recovery.

        • (Score: 2) by The Mighty Buzzard on Monday August 10 2020, @05:05AM (1 child)

          Eh, passable but I still ain't happy with it. That was a recoverable if sneaky crash. If I'd spent more time with clustered databases I might have caught it without data loss this time. I damned well will next time.

          --
          My rights don't end where your fear begins.
          • (Score: 2) by Reziac on Monday August 10 2020, @03:49PM

            by Reziac (2489) on Monday August 10 2020, @03:49PM (#1034364) Homepage

            Look on the bright side... thanks to your efforts, we never get to see the same fuckup twice. I realize this makes it hard on the slow learners, but such is life.

            --
            And there is no Alkibiades to come back and save us from ourselves.
      • (Score: 0) by Anonymous Coward on Monday August 10 2020, @04:40AM (9 children)

        by Anonymous Coward on Monday August 10 2020, @04:40AM (#1034196)

        Now I am confused. Where are the SQL/management/data nodes located? Are both data node machines also management nodes? Which is the arbitrator or is a third machine involved? If your layout is what I think it is, you may want to double check your deployment, as I don't think it is in line with the best practices.

        • (Score: 2) by The Mighty Buzzard on Monday August 10 2020, @04:44AM (8 children)

          It's probably not but we can't afford a whole bunch of servers just for the DB. We have two. Both are management and data nodes. If you babysit them any time anything unusual is going to happen, they're great. If you don't, you're in for headaches.

          --
          My rights don't end where your fear begins.
          • (Score: 1, Interesting) by Anonymous Coward on Monday August 10 2020, @05:37AM (7 children)

            by Anonymous Coward on Monday August 10 2020, @05:37AM (#1034223)

            I understand the concerns about adding or moving servers, but I'd strongly suggest adding another management node on one of your existing machines. The resources required are minimal and it would help quite a bit with your cluster availability.

            • (Score: 2) by The Mighty Buzzard on Monday August 10 2020, @03:23PM (6 children)

              Cheers, I'll see if I can find a suitable place to put one or two more. Like you said, if it's just the management daemons it doesn't take up much space or memory.

              --
              My rights don't end where your fear begins.
              • (Score: 0) by Anonymous Coward on Monday August 10 2020, @08:49PM (4 children)

                by Anonymous Coward on Monday August 10 2020, @08:49PM (#1034518)

                You really only need one more. That way you still have a quorum in case one fails. With that in place, the database nodes should be able to survive a rolling restart without "babysitting" them the whole time. However, the one located on its own machine should be the arbitrator. It might also help to set up unused slots in your cluster for all types of nodes to allow easier expansion in the future, as you can just assign new nodes to those slots in the future instead of rolling the whole thing.

                I hope my attempts to help don't feel like I'm piling on or intentionally demeaning.

                • (Score: 2) by The Mighty Buzzard on Tuesday August 11 2020, @02:53AM (3 children)

                  by The Mighty Buzzard (18) Subscriber Badge <themightybuzzard@proton.me> on Tuesday August 11 2020, @02:53AM (#1034696) Homepage Journal

                  Nah. One server assigned as arbitrator = single point of failure. Not having that is the entire reason we're clustered to begin with, so it's a non-starter.

                  --
                  My rights don't end where your fear begins.
                  • (Score: 0) by Anonymous Coward on Tuesday August 11 2020, @07:16AM (2 children)

                    by Anonymous Coward on Tuesday August 11 2020, @07:16AM (#1034791)

                    If you didn't disable arbitration, they elect one. You want a management node that isn't on the data nodes to be the preferred arbitrator with the highest rank, but you can set one of the others as fallback and even include your SQL nodes if you are paranoid. That way you require at least two failures, or as many as four, to bring the entire cluster down without losing data or degrading the cluster to the point of complete failure. But you state that is a SPOF. This, the fact your other data node stayed up, and your comments lead me to believe that you disabled arbitration, which isn't a good idea as you have no protection from a split brain and other problems in that case, or I wasn't clear enough.

                    • (Score: 2) by The Mighty Buzzard on Tuesday August 11 2020, @10:11PM (1 child)

                      by The Mighty Buzzard (18) Subscriber Badge <themightybuzzard@proton.me> on Tuesday August 11 2020, @10:11PM (#1035207) Homepage Journal

                      Point being, if there is only one non-data management node and it fails or is otherwise down for some reason, we're right back where we are now. I thought not being where we are right now was the entire idea.

                      --
                      My rights don't end where your fear begins.
                      • (Score: 0) by Anonymous Coward on Wednesday August 12 2020, @02:13AM

                        by Anonymous Coward on Wednesday August 12 2020, @02:13AM (#1035350)

                        No, if it fails, there is no arbitrator but you still have a quorum, which means they just elect a new one or continue without one. But, you also would have survived the explained incident automatically without the split brain that resulted had you had one. I could go on, but you are going to do whatever you want anyway, so I'm not going to bother anymore.

              • (Score: 0) by Anonymous Coward on Tuesday August 11 2020, @01:14AM

                by Anonymous Coward on Tuesday August 11 2020, @01:14AM (#1034657)

                I meant primary and not arbitrator. It should definitely be the arbitrator too. But it is recommended practice to have the primary management node not be a data node as well. It helps with solving problems with quorum, split brain, and other issues, while also allowing centralized service management from a single node.

      • (Score: 1) by jmichaelhudsondotnet on Thursday August 13 2020, @02:02PM (1 child)

        by jmichaelhudsondotnet (8122) on Thursday August 13 2020, @02:02PM (#1036133) Journal

        That is all I was asking for, if you just would have started with that and edit the original post to include this, I would be happy as a clam.

        Since I stopped reading your comments a few months ago, I didnt see this.

        It was nice to read something you wrote about the actual subject at hand instead of just a personal attack against me, you should try more of this and maybe in a year or so I will start reading your comments again.

        Sad, I would love to help with the admin of the site and I am probably in a timezone that is awake when youre sleeping.

        But until such a time as you can consistently discuss an issue and stop attacking my person, you can probably save your time telling me what is wrong with me, boomer.

    • (Score: 3, Funny) by Bot on Sunday August 09 2020, @08:25PM (2 children)

      by Bot (3902) on Sunday August 09 2020, @08:25PM (#1033990) Journal

      A better incident report:
      systemd

      (Note I don't imply it's better as more accurate, I mean it is more in harmony with the universe. That counts for something.

      --
      Account abandoned.
    • (Score: 2, Touché) by Anonymous Coward on Sunday August 09 2020, @09:05PM

      by Anonymous Coward on Sunday August 09 2020, @09:05PM (#1034007)

      We should all learn from how things like this happen to prevent it within our own zones of responsibility.

      Well, when your own personal "zone of responsibility" is defined as "site douche" - you really don't need to worry much.

(1) 2