Stories
Slash Boxes
Comments

SoylentNews is people

Normal Site Operations Restored

posted by NCommander on Saturday August 08 2015, @12:06AM   Printer-friendly
from the 503s-suck dept.

For a good portion of today, SoylentNews was generating 503 errors if you either logged in or tried to post a comment. While not 100% consistent, the underlying problem is that the design of MySQL cluster requires us to manually allocate space for indexes and data storage. Today, the index storage maxed out, and MySQL refused to insert new rows stating "Table 'name' is full".

We've doubled the size of the IndexMemory which should solve this issue in the short term. Longer term, we need to migrate some data to reside permanently on HDD storage. If anyone has experience with MySQL Cluster and can offer suggestions, we're all ears.

Here's our current memory usage on the cluster for those who are interested:

ndb_mgm> all report memoryusage
Node 2: Data usage is 81%(53650 32K pages of total 65536)
Node 2: Index usage is 46%(15407 8K pages of total 32800)
Node 3: Data usage is 81%(53648 32K pages of total 65536)
Node 3: Index usage is 46%(15407 8K pages of total 32800)

Sorry for any inconvenience

 
This discussion has been archived. No new comments can be posted.
Normal Site Operations Restored | Log In/Create an Account | Top | 19 comments | Search Discussion
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

  • (Score: 2) by NCommander on Saturday August 08 2015, @02:01AM

    by NCommander (2) Subscriber Badge <michael@casadevall.pro> on Saturday August 08 2015, @02:01AM (#219766) Homepage Journal

    I've heard good things about DB2, but even if we wanted to migrate to it, the licensing costs are prohibative. rehash is built on free-and-open source software, and there isn't a benefit in this case for using proprietary software.

    Furthermore, the free version explicitly disallows a high availability configuration. Wikipedia exert:

    DB2 Express-C is not permitted for use in high availability environments such as involving replication, active-passive, or shared disk clustering. In-place updates of the DB2 software (via patch sets called Fix Packs) are not available for DB2 Express-C; the existing DB2 Express-C engine must be removed and replaced with a newer release of DB2 Express-C. In either scenario, the DB2 database(s) remain intact on the server and are not touched by the software installation utility. IBM only offers the current version of Express-C on their website, but users may choose to install and run older releases of DB2 Express-C that they have previously downloaded from IBM.

    --
    Still always moving
    Starting Score:    1  point
    Karma-Bonus Modifier   +1  
    Total Score:   2