Stories
Slash Boxes
Comments

SoylentNews is people

posted by martyb on Thursday January 10 2019, @06:07PM   Printer-friendly
from the we're-baaaaaack! dept.

[Updated to correct time of neon CPU's spiking. --martyb]

We experienced an unexpected outage of the site this morning (20190110 00:15-07:45 UTC). At shortly after midnight approximately 0415 (UTC), CPU usage on neon suddenly pegged at 400% and things went downhill from there. Am not sure at this point what happened between 0015 and 0415.

Root cause is being investigated, but for now it seems the site is back up and working. Please let us know if you have any issues.

Note: you may need to have your browser ignore its cache (e.g. refresh with Ctrl+F5) and bring down everything fresh.

FWIW, system came back up after we rebooted neon (using the Linode manager page), and then bounced varnishd on fluorine and hydrogen (/home/bob/bin/bounce on each.)

Many thanks go to SemperOSS and cosurgi for problem determination and steps to rectify and FatPhil for his cheerleading!

[Update: TMB] So, the deal was that some unknown time in the past the ndb database node on helium had gone down. This wasn't a problem since we run a clustered database but nobody noticing it was. Then last night something caused neon to lose its cheese. Since it hosts the other node of the db, we had no db for a while. Bytram(martyb) has sysadmin powers for when unpleasant substances of various types hit the fan and thankfully he knew enough to get the neon db node back up and bounce apache/varnish on the web frontends, so kudos to him and all the folks who were backseat driving at the time due to lack of admin perms on their parts.

My brain's currently fried from going from asleep to OMGWTFBBQ without so much as a cup of coffee and a cigarette first, so I'm not going to dig into the root causes until it unfries itself but as a stopgap we have four more staff with shiny, new admin access that I'll be emergency bootcamping in the very near future. There's also going to be some monitoring reimplemented very soon so we notice this kind nonsense before it blows up in our faces again. I'll either update and bump this story or post a new one if we manage to figure out what the root causes were but at the moment the logs aren't being particularly helpful.


Original Submission

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 4, Funny) by bzipitidoo on Thursday January 10 2019, @01:54PM (9 children)

    by bzipitidoo (4388) on Thursday January 10 2019, @01:54PM (#784491) Journal

    Russian hackers? If anyone would know, it's those who are colluding with them. How much money did they ask? What's SolyentNews worth?

    Starting Score:    1  point
    Moderation   +2  
       Funny=2, Total=2
    Extra 'Funny' Modifier   0  
    Karma-Bonus Modifier   +1  

    Total Score:   4  
  • (Score: 4, Funny) by ewk on Thursday January 10 2019, @02:40PM (7 children)

    by ewk (5923) on Thursday January 10 2019, @02:40PM (#784498)

    "What's SolyentNews worth?"

    Not sure, but SoylentNews is priceless :-)

    --
    I don't always react, but when I do, I do it on SoylentNews
    • (Score: 2) by Runaway1956 on Thursday January 10 2019, @03:36PM (3 children)

      by Runaway1956 (2926) Subscriber Badge on Thursday January 10 2019, @03:36PM (#784519) Journal

      I think it's worth a buck two-eighty.

      • (Score: 2) by bob_super on Thursday January 10 2019, @05:44PM (2 children)

        by bob_super (1357) on Thursday January 10 2019, @05:44PM (#784574)

        Lock Ness monster offers three-fifty

        All those meta posts recently, it's like website management is hard, or something. Haven't you considered yet that outsourcing it to some Indians would be better for the balance sheet and my stock ?

        • (Score: 2) by Runaway1956 on Thursday January 10 2019, @05:48PM (1 child)

          by Runaway1956 (2926) Subscriber Badge on Thursday January 10 2019, @05:48PM (#784576) Journal

          But - but - but - I thought Buzzard was Indian? Surely he's not faking it like certain congress critters?

          • (Score: 2) by maxwell demon on Thursday January 10 2019, @07:03PM

            by maxwell demon (1608) on Thursday January 10 2019, @07:03PM (#784615) Journal

            You mean, SoylentNews was outsourced to India? :-)

            --
            The Tao of math: The numbers you can count are not the real numbers.
    • (Score: 2) by DeathMonkey on Thursday January 10 2019, @05:46PM (2 children)

      by DeathMonkey (1380) on Thursday January 10 2019, @05:46PM (#784575) Journal

      Not sure, but SoylentNews is priceless :-)

      Wait, what, I thought it was people? Sheesh, how am I supposed to eat priceless!

  • (Score: 3, Insightful) by Gaaark on Thursday January 10 2019, @04:41PM

    by Gaaark (41) on Thursday January 10 2019, @04:41PM (#784538) Journal

    What's it worth?
    Less than $3000 for those who haven't subscribed?!

    SUBSCRIBE!

    --
    --- Please remind me if I haven't been civil to you: I'm channeling MDC. ---Gaaark 2.0 ---