For a good portion of today, SoylentNews was generating 503 errors if you either logged in or tried to post a comment. While not 100% consistent, the underlying problem is that the design of MySQL cluster requires us to manually allocate space for indexes and data storage. Today, the index storage maxed out, and MySQL refused to insert new rows stating "Table 'name' is full".
We've doubled the size of the IndexMemory which should solve this issue in the short term. Longer term, we need to migrate some data to reside permanently on HDD storage. If anyone has experience with MySQL Cluster and can offer suggestions, we're all ears.
Here's our current memory usage on the cluster for those who are interested:
ndb_mgm> all report memoryusage Node 2: Data usage is 81%(53650 32K pages of total 65536) Node 2: Index usage is 46%(15407 8K pages of total 32800) Node 3: Data usage is 81%(53648 32K pages of total 65536) Node 3: Index usage is 46%(15407 8K pages of total 32800)
Sorry for any inconvenience
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
(Score: 3, Informative) by Anonymous Coward on Saturday August 08 2015, @12:18AM
next major version should a migration to PostgreSQL 9.5 (it should be out of beta when you are done)
(Score: 5, Insightful) by Phoenix666 on Saturday August 08 2015, @12:21AM
NCommander, Paulej, thanks for keeping things running. Nobody says it enough, but you guys are gold standard.
Every site goes down once in a while. C'est la vie, et l'Internet.
Washington DC delenda est.
(Score: 3, Informative) by NCommander on Saturday August 08 2015, @12:24AM
It's a long desired feature. I may put the time into making it happen much sooner than I expected. The odds are we'll migrate to PostgreSQL-XC which is a clustering variant; our experience is that we can't always be in control of restarts and such, and cluster technologies are much more resilient to a single node suddenly going down.
Still always moving
(Score: 3, Informative) by Post-Nihilist on Saturday August 08 2015, @12:37AM
have a look at https://github.com/postgres-x2/postgres-x2 [github.com] as PostgreSQL-XC seems dead. The last commit in PostgreSQL-XC is dated from 3 years ago
Be like us, be different, be a nihilist!!!
(Score: 2) by NCommander on Saturday August 08 2015, @01:11AM
Thanks. I'll keep this in mind.
Still always moving
(Score: 2) by KritonK on Monday August 10 2015, @08:02AM
According to the README [github.com] file, Postgres-X2 is Postgres-XC, renamed after it was moved to github. Internally, they still use the old name in various places, including the README file itself. (Check the "status" section.)
(Score: 2) by mendax on Saturday August 08 2015, @01:08AM
It's a good idea, but is the code base amenable to making that an relatively simple task? For example, in the Java world, as long as you're not doing anything terribly spooky or dependent upon a feature of a specific database, Hibernate makes switching DBMSes pretty straightforward (so long as you don't mind mediocre performance). However, I know next to nothing about Perl except that I tried learning it but had to keep forcing the vomit back down my throat.
It's really quite a simple choice: Life, Death, or Los Angeles.
(Score: 3, Informative) by NCommander on Saturday August 08 2015, @01:12AM
The database layer in rehash (the codebase that powers the site) is abstracted so changing database engines is possible, but its all handwritten SQL. aka, serious effort to change, but not impossible.
Still always moving
(Score: 2) by jdavidb on Saturday August 08 2015, @03:13AM
ⓋⒶ☮✝🕊 Secession is the right of all sentient beings
(Score: 0, Troll) by mendax on Saturday August 08 2015, @01:01AM
Oh, I think he fired himself some time ago! Never mind.
It's really quite a simple choice: Life, Death, or Los Angeles.
(Score: 0) by Anonymous Coward on Saturday August 08 2015, @01:30AM
Or choose a better RDB. I like DB2. Scales to relly LARGE machines and databases.
(Score: 2) by NCommander on Saturday August 08 2015, @02:01AM
I've heard good things about DB2, but even if we wanted to migrate to it, the licensing costs are prohibative. rehash is built on free-and-open source software, and there isn't a benefit in this case for using proprietary software.
Furthermore, the free version explicitly disallows a high availability configuration. Wikipedia exert:
Still always moving
(Score: 2) by kurenai.tsubasa on Saturday August 08 2015, @01:58AM
Thank you for all of your efforts! This site has become a favorite among the Chinese Amazons. They seek truth, as I hope all of us seek truth!
Long live Soylent! *cheers*
Long live Soylent! *cheers*
(Score: 0) by Anonymous Coward on Saturday August 08 2015, @03:39AM
Might I suggest Bobby Tables [xkcd.com], a renound SQL table expert.
(Score: 2) by hemocyanin on Saturday August 08 2015, @04:06AM
http://video.despair.com/art-of-demotivation/the-art-of-demotivation-storytelling/ [despair.com]
(Score: 0) by Anonymous Coward on Saturday August 08 2015, @04:25AM
sqlite.
(Score: 2) by mendax on Saturday August 08 2015, @04:45AM
Ah, it's not a bad idea, but it'll never work. You get top marks for effort. SQLite is really intended to be embedded in a program. I don't know for certain but I suspect it wasn't intended to work in a thread-safe manner and I bet it doesn't implement things like row locking. It supports transaction processing. That's a plus.
It's really quite a simple choice: Life, Death, or Los Angeles.
(Score: 0) by Anonymous Coward on Saturday August 08 2015, @05:17AM
Actually, SQLite works very well in threaded environments. It allows for concurrent access to data from multiple programs, though it does not support row-level locks - locks databases for updates/inserts.
Anyway, I don't know why not just use postgres and handle any HA with external scripts (like promoting slave to master, etc.)
https://wiki.postgresql.org/wiki/Replication,_Clustering,_and_Connection_Pooling [postgresql.org]
(Score: 2) by kbahey on Sunday August 09 2015, @03:13PM
MySQL Cluster/NDB is not as widely used as InnoDB.
InnoDB is ACID, and tried and tested much more than Cluster/NDB.
With InnoDB, you can get replication even with a Master-Master configuration.
Doing so, will get you off NDB, with little or no SQL source change. You will also be in the company of many big MySQL users.
This is far easier and less time consuming than a PostgreSQL migration. Worth a try for a while to see how things go.
2bits.com, Inc: Drupal, WordPress, and LAMP performance tuning [2bits.com].