Stories
Slash Boxes
Comments

SoylentNews is people

SoylentNews is powered by your submissions, so send in your scoop. Only 18 submissions in the queue.
Meta
posted by NCommander on Friday January 20 2017, @04:43PM   Printer-friendly
from the hot-upgrading-database-servers-ftw dept.

Earlier today, we ran an article detailing that Oracle released 270 critical security updates for many of its products, including MySQL cluster which we use here to provide high uptime and reliability for SoylentNews. Needless to say, it was time to upgrade both NDB backends, and the four MySQLd frontends. While the upgrade did not go completely smoothly due to the fact that MySQL strict mode got enabled, and broke the site briefly, our total downtime was less than five minutes or so. Right now, we had to do a full flush and purge of all caches, which means the site is running a bit larky until they can repopulate but I'm pleased to announce we're up to date and secure!

ndb_mgm> show
Cluster Configuration
---------------------
[ndbd(NDB)]	2 node(s)
id=2	@redacted (mysql-5.7.17 ndb-7.5.5, Nodegroup: 0)
id=3	@redacted (mysql-5.7.17 ndb-7.5.5, Nodegroup: 0, *)

[ndb_mgmd(MGM)]	2 node(s)
id=101	@redacted (mysql-5.7.17 ndb-7.5.5)
id=102	@redacted (mysql-5.7.17 ndb-7.5.5)

[mysqld(API)]	4 node(s)
id=11	@redacted (mysql-5.7.17 ndb-7.5.5)
id=12	@redacted (mysql-5.7.17 ndb-7.5.5)
id=13	@redacted (mysql-5.7.17 ndb-7.5.5)
id=14	@redacted (mysql-5.7.17 ndb-7.5.5)

If you notice any unusual breakages or slowdowns, please let me know in the comments. Otherwise, keep calm and carry on!

~ NCommander

Related Stories

Massive Oracle Critical Patch Update Fixes 270 Vulnerabilities 6 comments

Oracle has released the first Critical Patch Update scheduled for 2017, and it's massive. It fixes 270 vulnerabilities across multiple products, and over 100 of them are remotely exploitable by unauthenticated attackers.

The entire list of affected products and components is long, and Oracle advises users of all of them to implement the updates as soon as possible.

"The focus has shifted from Database and Java SE to critical business applications, as we predicted within the last 2 years," the ERPScan research team noted.

[...] The number of fixed issues is not the largest an Oracle CPU has ever delivered, but of the last five (since January 2016), four have passed the 240-mark.

Also: Oracle Patches 270 Vulnerabilities in January Update


Original Submission

This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 2) by ikanreed on Friday January 20 2017, @04:55PM

    by ikanreed (3164) Subscriber Badge on Friday January 20 2017, @04:55PM (#456597) Journal

    Who wants to waste time hacking a site an active userbase with a few hundred?

    Given(hopefully) that the database servers are adequately firewalled such that only apache can reach them, anyone wanting to use MySQL exploits would nominally have to specifically target slashcode to e'SELECT * FROM PERMISSIONS;--

    • (Score: 2) by NCommander on Friday January 20 2017, @04:59PM

      by NCommander (2) Subscriber Badge <michael@casadevall.pro> on Friday January 20 2017, @04:59PM (#456599) Homepage Journal

      In terms of posters, a few hundred is probably correct, but we get a lot more of view traffic. I haven't checked the statistics recently, but we get a lot of read traffic to the point the site starts lagging when we have web frontend down. Plus I can't be on the high ground on keeping up on security patches if I'm a hypocrite about it.

      --
      Still always moving
      • (Score: 2) by ikanreed on Friday January 20 2017, @05:10PM

        by ikanreed (3164) Subscriber Badge on Friday January 20 2017, @05:10PM (#456607) Journal

        I think you're right, I just wanted a particularly hypocritical post to add my fake sql injection joke to.

      • (Score: 2) by Webweasel on Friday January 20 2017, @11:38PM

        by Webweasel (567) on Friday January 20 2017, @11:38PM (#456775) Homepage Journal

        Hey! I like, post every couple of months and shit. sometimes.

        I moderate too... when I get time.

        STOP JUDGING ME!

        --
        Priyom.org Number stations, Russian Military radio. "You are a bad, bad man. Do you have any other virtues?"-Runaway1956
    • (Score: 2) by shipofgold on Friday January 20 2017, @05:05PM

      by shipofgold (4696) on Friday January 20 2017, @05:05PM (#456603)

      Someone who wants to take over the site and use it for more nefarious reasons?

      • (Score: 2) by AthanasiusKircher on Friday January 20 2017, @06:42PM

        by AthanasiusKircher (5291) on Friday January 20 2017, @06:42PM (#456639) Journal

        "Nefarious reasons"? What could be more nefarious than making news out of... PEOPLE?

        Puppies, maybe? Yeah -- that's usually worse. Today if they remade Soylent Green and really wanted a shocker, it would turn out that it was made from puppies.

        One of the most insightful scenes in film from the last year or two was in Look Who's Back [wikipedia.org], which involves the mysterious return of Adolf Hitler to modern Berlin. Shockingly, he seems to get away with saying just about anything from his old rhetoric -- ranting about immigrants and the glory of the German race, endorsing "labor camps" for undesirables, etc. It all just seems to make him more popular. But [SPOILER] -- the one thing the public cannot stand from him is violence to a small dog. I won't say more... you just have to see it.

        • (Score: 0) by Anonymous Coward on Saturday January 21 2017, @05:56AM

          by Anonymous Coward on Saturday January 21 2017, @05:56AM (#456869)

          * TVTropes Warning! *

          http://tvtropes.org/pmwiki/pmwiki.php/Main/KickTheDog [tvtropes.org]

          * TVTropes Warning! *

        • (Score: 2) by Phoenix666 on Saturday January 21 2017, @11:58AM

          by Phoenix666 (552) on Saturday January 21 2017, @11:58AM (#456942) Journal

          Most people familiar with the history know this, but it was a good reminder when the guy going nuts trying to stop him confronted him on the roof and Hitler pointed out, "All those people voted for me. They voted for me."

          --
          Washington DC delenda est.
    • (Score: 1) by nitehawk214 on Friday January 20 2017, @05:38PM

      by nitehawk214 (1304) on Friday January 20 2017, @05:38PM (#456621)

      I think you underestimate the motivations of script kiddies.

      --
      "Don't you ever miss the days when you used to be nostalgic?" -Loiosh
      • (Score: 2) by bob_super on Friday January 20 2017, @06:01PM

        by bob_super (1357) on Friday January 20 2017, @06:01PM (#456627)

        I'm strangely okay with the idea of script kiddies taking pride for pwning Soylentnews.

    • (Score: 0) by Anonymous Coward on Friday January 20 2017, @07:24PM

      by Anonymous Coward on Friday January 20 2017, @07:24PM (#456661)

      Who wants to waste time hacking a site an active userbase with a few hundred?

      No one is researching sites like SN. They just point their automated attack servers at domains, IPs + ports, etc. It's nothing personal ... it's just your number was called today.

      • (Score: 2) by ikanreed on Friday January 20 2017, @07:35PM

        by ikanreed (3164) Subscriber Badge on Friday January 20 2017, @07:35PM (#456665) Journal

        Yeah, but the MySQL servers, if properly setup, aren't on any public IP.

        • (Score: 2) by NCommander on Friday January 20 2017, @10:39PM

          by NCommander (2) Subscriber Badge <michael@casadevall.pro> on Friday January 20 2017, @10:39PM (#456750) Homepage Journal

          Ours aren't, but if someone manages to get a non-root shell on one of our machines, I don't want them to be able to break into the database. For logistical reasons, there are several machines on our network that can access the mysqld endpoints for backup and maintenance reasons. If someone scored access to the right box, they'd be in a position that they could talk to 3306 on one of the DB servers.

          --
          Still always moving
          • (Score: 2) by ikanreed on Friday January 20 2017, @11:01PM

            by ikanreed (3164) Subscriber Badge on Friday January 20 2017, @11:01PM (#456762) Journal

            Can't believe I didn't think of that kinda obvious case.

  • (Score: 3, Funny) by Thexalon on Friday January 20 2017, @05:04PM

    by Thexalon (636) on Friday January 20 2017, @05:04PM (#456601)

    I demand a full refund!

    (Just kidding. Keep up the good work, NCommander and crew)

    --
    The only thing that stops a bad guy with a compiler is a good guy with a compiler.
    • (Score: 1, Touché) by Anonymous Coward on Friday January 20 2017, @05:11PM

      by Anonymous Coward on Friday January 20 2017, @05:11PM (#456608)

      And I demand recognition!

    • (Score: 2) by The Mighty Buzzard on Friday January 20 2017, @10:29PM

      by The Mighty Buzzard (18) Subscriber Badge <themightybuzzard@proton.me> on Friday January 20 2017, @10:29PM (#456740) Homepage Journal

      Was all him. I was getting ready to go fishing and pj sounded like he was otherwise busy too.

      --
      My rights don't end where your fear begins.
  • (Score: 2) by GlennC on Friday January 20 2017, @05:16PM

    by GlennC (3656) on Friday January 20 2017, @05:16PM (#456611)

    It appears that you've done a good job. Thanks.

    --
    Sorry folks...the world is bigger and more varied than you want it to be. Deal with it.
  • (Score: 2) by Fnord666 on Friday January 20 2017, @05:23PM

    by Fnord666 (652) on Friday January 20 2017, @05:23PM (#456613) Homepage
    Well done sirs.
  • (Score: 2) by jdavidb on Friday January 20 2017, @05:41PM

    by jdavidb (5690) on Friday January 20 2017, @05:41PM (#456623) Homepage Journal
    Thanks!
    --
    ⓋⒶ☮✝🕊 Secession is the right of all sentient beings
  • (Score: 0) by Anonymous Coward on Friday January 20 2017, @06:06PM

    by Anonymous Coward on Friday January 20 2017, @06:06PM (#456628)

    That stories with a large number of comments takes a really long time for the page to load. It's not an ISP speed problem, I have 100Mb/s Cable.

    • (Score: 2, Interesting) by charon on Friday January 20 2017, @06:29PM

      by charon (5660) on Friday January 20 2017, @06:29PM (#456634) Journal
      This will be changing soon. There is an update pretty much ready to roll out that significantly improves page load time. And by significantly, I mean massively.
  • (Score: 2) by kazzie on Friday January 20 2017, @06:35PM

    by kazzie (5309) Subscriber Badge on Friday January 20 2017, @06:35PM (#456636)

    270 critical updates? That makes Microsoft's Patch Tuesday seem tame in comparison. Were they stockpiling these for a special occasion?

    • (Score: 3, Insightful) by urza9814 on Friday January 20 2017, @06:52PM

      by urza9814 (3954) on Friday January 20 2017, @06:52PM (#456646) Journal

      270 critical updates? That makes Microsoft's Patch Tuesday seem tame in comparison. Were they stockpiling these for a special occasion?

      No, they just had to leave those in until they finished building a more permanent backdoor for the NSA.

      • (Score: 0) by Anonymous Coward on Friday January 20 2017, @09:17PM

        by Anonymous Coward on Friday January 20 2017, @09:17PM (#456711)

        I don't think they need to program in back door access for the NSA. 270 updates / 3 months in a quarter = 90 updates a month. This means that with 45 separate pieces of software, there is one new critical update per product every two weeks, on average. People coding in back doors on purpose wouldn't be leaving that many weak ones for your competitors to take advantage of as well. This all just screams terrible code bases and no comprehensive testing.

    • (Score: 0) by Anonymous Coward on Saturday January 21 2017, @01:23AM

      by Anonymous Coward on Saturday January 21 2017, @01:23AM (#456807)

      A story still on the front page says

      The number of fixed issues is not the largest an Oracle CPU has ever delivered, but of the last five (since January 2016), four have passed the 240-mark.

      One might easily draw the conclusion that Oracle simply produces software of a low quality and doesn't do proper testing.

      .
      One more round of cheers for our all-volunteer staff.

      -- OriginalOwner_ [soylentnews.org]

  • (Score: 2) by arulatas on Friday January 20 2017, @07:12PM

    by arulatas (3600) on Friday January 20 2017, @07:12PM (#456656)

    I just wanted to say thank you for the great jobs you do in keeping the site running.

    --
    ----- 10 turns around
  • (Score: 0) by Anonymous Coward on Friday January 20 2017, @07:44PM

    by Anonymous Coward on Friday January 20 2017, @07:44PM (#456667)

    why not mariadb and Galera Cluster [galeracluster.com]?

    • (Score: 0) by Anonymous Coward on Friday January 20 2017, @09:21PM

      by Anonymous Coward on Friday January 20 2017, @09:21PM (#456712)

      Maybe because MariaDB will get those patches in 2 months after they manage to port them?

    • (Score: 2) by NCommander on Friday January 20 2017, @10:37PM

      by NCommander (2) Subscriber Badge <michael@casadevall.pro> on Friday January 20 2017, @10:37PM (#456744) Homepage Journal

      While I know that Maria is essentially compatible with MySQL (since it's a fork), if we're going to make serious efforts to change database engines, it will be to PostgreSQL with a cluster solution.

      One major advantage over cluster vs mariadb is that the entire thing sits in memory, and actually let us drastically simplify parts of the site architecture because we were available to rip out an entire layer of caching. Also, despite being fiddly as fuck to setup, it's been incredibly rock solid; we've had unexpected server downtime, and never once had the cluster fail to self-recover, and keep on trucking. The site hiccups we got during the DB upgrade mostly came from the fact that mysqld requires a fair bit of mindfuckery to run against NDB, and the upgrade broke some of the startup scripts.

      With some optimization and re-arrangement of the schema, we are in cases that we only ever span one table, or NDB's push-down JOINs can do their job. Since the vast majority of database activity is SELECT, we get absurdly good performance out of it on the whole. Galera isn't as battle tested as cluster.

      --
      Still always moving
    • (Score: 2) by The Mighty Buzzard on Friday January 20 2017, @10:37PM

      by The Mighty Buzzard (18) Subscriber Badge <themightybuzzard@proton.me> on Friday January 20 2017, @10:37PM (#456746) Homepage Journal

      Well, I'm going to go out on a limb and say because of the old adage "if it ain't broke, don't fix it". There was something specific that kept us from going mariadb originally, some feature the code used that mariadb did not have at that time. Now though, because it's an epic pain to swap dbs around without downtime and it would probably cause an hour or two of it as we found every last tiny difference between MySQL and its fork while trying to bring the site back up.

      --
      My rights don't end where your fear begins.
  • (Score: 2) by pkrasimirov on Friday January 20 2017, @08:05PM

    by pkrasimirov (3358) Subscriber Badge on Friday January 20 2017, @08:05PM (#456672)

    Kudos. Keep up the good work.

  • (Score: 1) by pTamok on Friday January 20 2017, @08:20PM

    by pTamok (3042) on Friday January 20 2017, @08:20PM (#456681)

    Your work in keeping the site going is very much appreciated.

  • (Score: 2) by weeds on Friday January 20 2017, @09:41PM

    by weeds (611) on Friday January 20 2017, @09:41PM (#456718) Journal

    Thank you NC and all of the Soylentnews.org staff. I visit several times a day and when I have time jump on IRC.

  • (Score: 1, Funny) by Anonymous Coward on Friday January 20 2017, @10:03PM

    by Anonymous Coward on Friday January 20 2017, @10:03PM (#456725)

    I need 6 nines of availability on my websites. This is shoddy and disgusting. Worst ever. You're FIRED. I'm going back to the other place.

  • (Score: 0) by Anonymous Coward on Saturday January 21 2017, @12:38AM

    by Anonymous Coward on Saturday January 21 2017, @12:38AM (#456794)

    > we had to do a full flush and purge of all caches

    That sounds like a euphemism for taking a big dump.

  • (Score: 0) by Anonymous Coward on Sunday January 22 2017, @03:10AM

    by Anonymous Coward on Sunday January 22 2017, @03:10AM (#457234)

    Overall, keep up the excellent technical and editorial work. SN is a great site.

    On the too-long list of things to do, you might look into the quality of service for Onion traffic. Quite often the site returns an error about that the page isn't redirecting properly. But much of the time it locks into either the Meta section or the Breaking section, after a very long delay. Please do take a look at that and at least triage it. It would be appreciated even if only by a smaller public.