SoylentNews is people
SoylentNews
from the hot-upgrading-database-servers-ftw dept.
Earlier today, we ran an article detailing that Oracle released 270 critical security updates for many of its products, including MySQL cluster which we use here to provide high uptime and reliability for SoylentNews. Needless to say, it was time to upgrade both NDB backends, and the four MySQLd frontends. While the upgrade did not go completely smoothly due to the fact that MySQL strict mode got enabled, and broke the site briefly, our total downtime was less than five minutes or so. Right now, we had to do a full flush and purge of all caches, which means the site is running a bit larky until they can repopulate but I'm pleased to announce we're up to date and secure!
ndb_mgm> show Cluster Configuration --------------------- [ndbd(NDB)] 2 node(s) id=2 @redacted (mysql-5.7.17 ndb-7.5.5, Nodegroup: 0) id=3 @redacted (mysql-5.7.17 ndb-7.5.5, Nodegroup: 0, *) [ndb_mgmd(MGM)] 2 node(s) id=101 @redacted (mysql-5.7.17 ndb-7.5.5) id=102 @redacted (mysql-5.7.17 ndb-7.5.5) [mysqld(API)] 4 node(s) id=11 @redacted (mysql-5.7.17 ndb-7.5.5) id=12 @redacted (mysql-5.7.17 ndb-7.5.5) id=13 @redacted (mysql-5.7.17 ndb-7.5.5) id=14 @redacted (mysql-5.7.17 ndb-7.5.5)
If you notice any unusual breakages or slowdowns, please let me know in the comments. Otherwise, keep calm and carry on!
~ NCommander
(Score: 0) by Anonymous Coward on Friday January 20 2017, @07:44PM
why not mariadb and Galera Cluster [galeracluster.com]?
(Score: 0) by Anonymous Coward on Friday January 20 2017, @09:21PM
Maybe because MariaDB will get those patches in 2 months after they manage to port them?
(Score: 2) by NCommander on Friday January 20 2017, @10:37PM
While I know that Maria is essentially compatible with MySQL (since it's a fork), if we're going to make serious efforts to change database engines, it will be to PostgreSQL with a cluster solution.
One major advantage over cluster vs mariadb is that the entire thing sits in memory, and actually let us drastically simplify parts of the site architecture because we were available to rip out an entire layer of caching. Also, despite being fiddly as fuck to setup, it's been incredibly rock solid; we've had unexpected server downtime, and never once had the cluster fail to self-recover, and keep on trucking. The site hiccups we got during the DB upgrade mostly came from the fact that mysqld requires a fair bit of mindfuckery to run against NDB, and the upgrade broke some of the startup scripts.
With some optimization and re-arrangement of the schema, we are in cases that we only ever span one table, or NDB's push-down JOINs can do their job. Since the vast majority of database activity is SELECT, we get absurdly good performance out of it on the whole. Galera isn't as battle tested as cluster.
Still always moving
(Score: 2) by The Mighty Buzzard on Friday January 20 2017, @10:37PM
Well, I'm going to go out on a limb and say because of the old adage "if it ain't broke, don't fix it". There was something specific that kept us from going mariadb originally, some feature the code used that mariadb did not have at that time. Now though, because it's an epic pain to swap dbs around without downtime and it would probably cause an hour or two of it as we found every last tiny difference between MySQL and its fork while trying to bring the site back up.
My rights don't end where your fear begins.