from the hot-upgrading-database-servers-ftw dept.
Earlier today, we ran an article detailing that Oracle released 270 critical security updates for many of its products, including MySQL cluster which we use here to provide high uptime and reliability for SoylentNews. Needless to say, it was time to upgrade both NDB backends, and the four MySQLd frontends. While the upgrade did not go completely smoothly due to the fact that MySQL strict mode got enabled, and broke the site briefly, our total downtime was less than five minutes or so. Right now, we had to do a full flush and purge of all caches, which means the site is running a bit larky until they can repopulate but I'm pleased to announce we're up to date and secure!
ndb_mgm> show Cluster Configuration --------------------- [ndbd(NDB)] 2 node(s) id=2 @redacted (mysql-5.7.17 ndb-7.5.5, Nodegroup: 0) id=3 @redacted (mysql-5.7.17 ndb-7.5.5, Nodegroup: 0, *) [ndb_mgmd(MGM)] 2 node(s) id=101 @redacted (mysql-5.7.17 ndb-7.5.5) id=102 @redacted (mysql-5.7.17 ndb-7.5.5) [mysqld(API)] 4 node(s) id=11 @redacted (mysql-5.7.17 ndb-7.5.5) id=12 @redacted (mysql-5.7.17 ndb-7.5.5) id=13 @redacted (mysql-5.7.17 ndb-7.5.5) id=14 @redacted (mysql-5.7.17 ndb-7.5.5)
If you notice any unusual breakages or slowdowns, please let me know in the comments. Otherwise, keep calm and carry on!
~ NCommander
Related Stories
Oracle has released the first Critical Patch Update scheduled for 2017, and it's massive. It fixes 270 vulnerabilities across multiple products, and over 100 of them are remotely exploitable by unauthenticated attackers.
The entire list of affected products and components is long, and Oracle advises users of all of them to implement the updates as soon as possible.
"The focus has shifted from Database and Java SE to critical business applications, as we predicted within the last 2 years," the ERPScan research team noted.
[...] The number of fixed issues is not the largest an Oracle CPU has ever delivered, but of the last five (since January 2016), four have passed the 240-mark.
Also: Oracle Patches 270 Vulnerabilities in January Update
(Score: 2) by ikanreed on Friday January 20 2017, @04:55PM
Who wants to waste time hacking a site an active userbase with a few hundred?
Given(hopefully) that the database servers are adequately firewalled such that only apache can reach them, anyone wanting to use MySQL exploits would nominally have to specifically target slashcode to e'SELECT * FROM PERMISSIONS;--
(Score: 2) by NCommander on Friday January 20 2017, @04:59PM
In terms of posters, a few hundred is probably correct, but we get a lot more of view traffic. I haven't checked the statistics recently, but we get a lot of read traffic to the point the site starts lagging when we have web frontend down. Plus I can't be on the high ground on keeping up on security patches if I'm a hypocrite about it.
Still always moving
(Score: 2) by ikanreed on Friday January 20 2017, @05:10PM
I think you're right, I just wanted a particularly hypocritical post to add my fake sql injection joke to.
(Score: 2) by Webweasel on Friday January 20 2017, @11:38PM
Hey! I like, post every couple of months and shit. sometimes.
I moderate too... when I get time.
STOP JUDGING ME!
Priyom.org Number stations, Russian Military radio. "You are a bad, bad man. Do you have any other virtues?"-Runaway1956
(Score: 2) by shipofgold on Friday January 20 2017, @05:05PM
Someone who wants to take over the site and use it for more nefarious reasons?
(Score: 2) by AthanasiusKircher on Friday January 20 2017, @06:42PM
"Nefarious reasons"? What could be more nefarious than making news out of... PEOPLE?
Puppies, maybe? Yeah -- that's usually worse. Today if they remade Soylent Green and really wanted a shocker, it would turn out that it was made from puppies.
One of the most insightful scenes in film from the last year or two was in Look Who's Back [wikipedia.org], which involves the mysterious return of Adolf Hitler to modern Berlin. Shockingly, he seems to get away with saying just about anything from his old rhetoric -- ranting about immigrants and the glory of the German race, endorsing "labor camps" for undesirables, etc. It all just seems to make him more popular. But [SPOILER] -- the one thing the public cannot stand from him is violence to a small dog. I won't say more... you just have to see it.
(Score: 0) by Anonymous Coward on Saturday January 21 2017, @05:56AM
* TVTropes Warning! *
http://tvtropes.org/pmwiki/pmwiki.php/Main/KickTheDog [tvtropes.org]
* TVTropes Warning! *
(Score: 2) by Phoenix666 on Saturday January 21 2017, @11:58AM
Most people familiar with the history know this, but it was a good reminder when the guy going nuts trying to stop him confronted him on the roof and Hitler pointed out, "All those people voted for me. They voted for me."
Washington DC delenda est.
(Score: 1) by nitehawk214 on Friday January 20 2017, @05:38PM
I think you underestimate the motivations of script kiddies.
"Don't you ever miss the days when you used to be nostalgic?" -Loiosh
(Score: 2) by bob_super on Friday January 20 2017, @06:01PM
I'm strangely okay with the idea of script kiddies taking pride for pwning Soylentnews.
(Score: 0) by Anonymous Coward on Friday January 20 2017, @07:24PM
Who wants to waste time hacking a site an active userbase with a few hundred?
No one is researching sites like SN. They just point their automated attack servers at domains, IPs + ports, etc. It's nothing personal ... it's just your number was called today.
(Score: 2) by ikanreed on Friday January 20 2017, @07:35PM
Yeah, but the MySQL servers, if properly setup, aren't on any public IP.
(Score: 2) by NCommander on Friday January 20 2017, @10:39PM
Ours aren't, but if someone manages to get a non-root shell on one of our machines, I don't want them to be able to break into the database. For logistical reasons, there are several machines on our network that can access the mysqld endpoints for backup and maintenance reasons. If someone scored access to the right box, they'd be in a position that they could talk to 3306 on one of the DB servers.
Still always moving
(Score: 2) by ikanreed on Friday January 20 2017, @11:01PM
Can't believe I didn't think of that kinda obvious case.
(Score: 3, Funny) by Thexalon on Friday January 20 2017, @05:04PM
I demand a full refund!
(Just kidding. Keep up the good work, NCommander and crew)
The only thing that stops a bad guy with a compiler is a good guy with a compiler.
(Score: 1, Touché) by Anonymous Coward on Friday January 20 2017, @05:11PM
And I demand recognition!
(Score: 2) by The Mighty Buzzard on Friday January 20 2017, @10:29PM
Was all him. I was getting ready to go fishing and pj sounded like he was otherwise busy too.
My rights don't end where your fear begins.
(Score: 2) by GlennC on Friday January 20 2017, @05:16PM
It appears that you've done a good job. Thanks.
Sorry folks...the world is bigger and more varied than you want it to be. Deal with it.
(Score: 2) by Fnord666 on Friday January 20 2017, @05:23PM
(Score: 2) by jdavidb on Friday January 20 2017, @05:41PM
ⓋⒶ☮✝🕊 Secession is the right of all sentient beings
(Score: 0) by Anonymous Coward on Friday January 20 2017, @06:06PM
That stories with a large number of comments takes a really long time for the page to load. It's not an ISP speed problem, I have 100Mb/s Cable.
(Score: 2, Interesting) by charon on Friday January 20 2017, @06:29PM
(Score: 2) by kazzie on Friday January 20 2017, @06:35PM
270 critical updates? That makes Microsoft's Patch Tuesday seem tame in comparison. Were they stockpiling these for a special occasion?
(Score: 3, Insightful) by urza9814 on Friday January 20 2017, @06:52PM
No, they just had to leave those in until they finished building a more permanent backdoor for the NSA.
(Score: 0) by Anonymous Coward on Friday January 20 2017, @09:17PM
I don't think they need to program in back door access for the NSA. 270 updates / 3 months in a quarter = 90 updates a month. This means that with 45 separate pieces of software, there is one new critical update per product every two weeks, on average. People coding in back doors on purpose wouldn't be leaving that many weak ones for your competitors to take advantage of as well. This all just screams terrible code bases and no comprehensive testing.
(Score: 0) by Anonymous Coward on Saturday January 21 2017, @01:23AM
A story still on the front page says
One might easily draw the conclusion that Oracle simply produces software of a low quality and doesn't do proper testing.
.
One more round of cheers for our all-volunteer staff.
-- OriginalOwner_ [soylentnews.org]
(Score: 2) by arulatas on Friday January 20 2017, @07:12PM
I just wanted to say thank you for the great jobs you do in keeping the site running.
----- 10 turns around
(Score: 0) by Anonymous Coward on Friday January 20 2017, @07:44PM
why not mariadb and Galera Cluster [galeracluster.com]?
(Score: 0) by Anonymous Coward on Friday January 20 2017, @09:21PM
Maybe because MariaDB will get those patches in 2 months after they manage to port them?
(Score: 2) by NCommander on Friday January 20 2017, @10:37PM
While I know that Maria is essentially compatible with MySQL (since it's a fork), if we're going to make serious efforts to change database engines, it will be to PostgreSQL with a cluster solution.
One major advantage over cluster vs mariadb is that the entire thing sits in memory, and actually let us drastically simplify parts of the site architecture because we were available to rip out an entire layer of caching. Also, despite being fiddly as fuck to setup, it's been incredibly rock solid; we've had unexpected server downtime, and never once had the cluster fail to self-recover, and keep on trucking. The site hiccups we got during the DB upgrade mostly came from the fact that mysqld requires a fair bit of mindfuckery to run against NDB, and the upgrade broke some of the startup scripts.
With some optimization and re-arrangement of the schema, we are in cases that we only ever span one table, or NDB's push-down JOINs can do their job. Since the vast majority of database activity is SELECT, we get absurdly good performance out of it on the whole. Galera isn't as battle tested as cluster.
Still always moving
(Score: 2) by The Mighty Buzzard on Friday January 20 2017, @10:37PM
Well, I'm going to go out on a limb and say because of the old adage "if it ain't broke, don't fix it". There was something specific that kept us from going mariadb originally, some feature the code used that mariadb did not have at that time. Now though, because it's an epic pain to swap dbs around without downtime and it would probably cause an hour or two of it as we found every last tiny difference between MySQL and its fork while trying to bring the site back up.
My rights don't end where your fear begins.
(Score: 2) by pkrasimirov on Friday January 20 2017, @08:05PM
Kudos. Keep up the good work.
(Score: 1) by pTamok on Friday January 20 2017, @08:20PM
Your work in keeping the site going is very much appreciated.
(Score: 2) by weeds on Friday January 20 2017, @09:41PM
Thank you NC and all of the Soylentnews.org staff. I visit several times a day and when I have time jump on IRC.
Get money out of politics! [mayday.us]
(Score: 1, Funny) by Anonymous Coward on Friday January 20 2017, @10:03PM
I need 6 nines of availability on my websites. This is shoddy and disgusting. Worst ever. You're FIRED. I'm going back to the other place.
(Score: 3, Funny) by The Mighty Buzzard on Friday January 20 2017, @10:40PM
9
9
9
9
9
9
There you go. Let me know if you need any more. I got lots.
My rights don't end where your fear begins.
(Score: 3, Funny) by NCommander on Saturday January 21 2017, @01:42PM
Dude, you took that out of our bag of spare 6s and just mounted them upside-down!
Still always moving
(Score: 2) by The Mighty Buzzard on Saturday January 21 2017, @03:50PM
At least I'm not mounting them sideways just to see what happens like martyb...
My rights don't end where your fear begins.
(Score: 0) by Anonymous Coward on Saturday January 21 2017, @12:38AM
> we had to do a full flush and purge of all caches
That sounds like a euphemism for taking a big dump.
(Score: 0) by Anonymous Coward on Sunday January 22 2017, @03:10AM
Overall, keep up the excellent technical and editorial work. SN is a great site.
On the too-long list of things to do, you might look into the quality of service for Onion traffic. Quite often the site returns an error about that the page isn't redirecting properly. But much of the time it locks into either the Meta section or the Breaking section, after a very long delay. Please do take a look at that and at least triage it. It would be appreciated even if only by a smaller public.