Stories
Slash Boxes
Comments

SoylentNews is people

posted by NCommander on Monday June 08 2015, @10:00AM   Printer-friendly
from the boat-anchor-detached dept.

As debugging efforts continue, I think most of the community should expect we're going to be running a daily story on our effort to return the site to normality. We did another bugfix rollout to production to continue cleaning up error logs, and Paul I continue to make modifications to improve site performance. As you may have already noticed, we've managed a serious coup in the battle for low page times by replacing Linode's NodeBalancer product with a self-rolled nginx-frontend proxy. The vast majority of page loads are now sub-second.

Here's the quick list of stuff we changed over the weekend:

Rehash 15.05.3 - Changelog

  • Optimized several slow queries in the frontend which were causing timeouts
  • Drastically improved rehash's reporting to error_log
  • Rolled out new frontend server (sodium) to handle load-balancing and SSL termination
  • SSL by-default re-enabled; Firefox now loads SoylentNews extremely quickly due to the new frontend supporting SSL keepalive
  • Disabled memcached on production temporarily for now (see past the break for details)

Although we've yet to formally locate and disable the cause of the 500s and HASH(*) entries you sometimes get on page load, I now have a working theory on what's going on.

During debugging, I would notice we'd almost universally get a bad load from the cache if varnish or the loadbalancer burped for a moment, As best I can tell running backwards through traces and various error logs, the underlying cause of the 500s and bad page loads is one of two causes: either we're getting bad data from memcached on a cache read, or a bad load into cache from the database. Of these two, I'm leaning closer to the former, since if we were loading bad data into memcached, we'd see consistent 500s once the cache was corrupted.

It's somewhat difficult to accept that memcached is responsible for our corruption issues; we've been using it since golive a year and a half ago. However, given a lack of other leads, I flipped the memcached config switch to off, and then loadtested the site to see how bad the performance drop would be. Much to my surprise, the combination of query optimizations, the faster apache 2 codebase, and (for the most part) increased responsiveness due to having a multimaster database seems to be able to cover the slack for the time being. As of writing, memcached has been disabled for several hours, and I've yet to see any of the telltale signs of corruption in error_log. I also need to note that its the middle of the night for the vast majority of our users, so this may just be the calm before the storm.

I dislike using production as something of a guenna pig, but given the very transitory nature of this bug, and our inability to reliably reproduce it on our dev systems, we're left somewhere between a rock and a hard place. I would like to thank the SoylentNews community for their understanding over the last week, and I hope normality may finally return to this site :)

~ NCommander

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 2) by kbahey on Monday June 08 2015, @09:28PM

    by kbahey (1147) on Monday June 08 2015, @09:28PM (#193823) Homepage

    Here is a guess: are you trying to large cache items? Perhaps you need to change the default from 1M to something more (using the -I [that is an uppercase "i"]).

    Starting Score:    1  point
    Karma-Bonus Modifier   +1  

    Total Score:   2  
  • (Score: 2) by NCommander on Monday June 08 2015, @10:01PM

    by NCommander (2) Subscriber Badge <michael@casadevall.pro> on Monday June 08 2015, @10:01PM (#193834) Homepage Journal

    We didn't change the configuration on memcached or any of the caching layer code on the upgrade. Generally what it does is a bunch of core functions such as getStory, getComments, or getUser tries to pull the data from memcache. If its not there, it runs a SELECT on the database to get it dynamically, and loads it into memcache for future loads. It's possible migrating from Perl 5.12 -> 5.20.1 broke the API though I have no idea how ...

    --
    Still always moving