SoylentNews

As an aside, I did experiment with EUIV's multiplayer functionality after posting the last topic. As I found, while it is functional, it is very likely to desync at high speeds, and thus causes large amounts of waiting periods, which I think would be uninteresting for folks to watch. Sorry about that. ~ NCommander

Why We Dumped LDAP

One of the golden rules of system administration is "if it ain't broke, don't fix it". Given that LDAP is generally considered critical infrastructure for sites that depend on it, its worth spending a few moments explaining why we replaced it. Our LDAP backend was powered by OpenLDAP, which is generally the de facto standard for LDAP servers on Linux. In our experience though, OpenLDAP is extremely difficult to configure due to storing its configuration information within the LDAP tree itself (under cn=config), and being incredibly difficult to examine its current state, as well as recovering from any misconfiguration. In practice, I found it necessary to dump the entire LDAP configuration, modify the raw LDIF files, and then reimport with slapcat, and then pray. Painful, but manageable since, in practice, the overall server configuration shouldn't change frequently.

Unfortunately, every aspect of OpenLDAP has proven to be painful to administer. In keeping with the idea that none of our critical infrastructure should have single points of failure, we established replica servers from our master, and configured client systems to look at the replicas in case the master server take a dive (or is restarting). While a noble idea, we found that frequently without warning or cause, replication would either get out of sync, or simply stop working all together with no useful error messages being logged by slapd. Furthermore, when failover worked, systems would start to lag as nss_ldap kept trying to query the master for 5-10 seconds before switching to the slave for each and every query. As a whole, the entire setup was incredibly brittle.

While many of these issues could be laid at OpenLDAP (vs. LDAP itself as a protocol), other issues compounded to make life miserable. While there are other LDAP implementations such as 389 Directory Server, the simple fact of the matter is that due to schema differences, no two LDAP instances are directly compatible with each other; one can't simply copy the data out of OpenLDAP and import it directly into 389. The issue is further compounded if one is using extended schemas (as we were to store SSH public keys). As such, when slapd started to hang without warning, and without clear indication as of why, the pain got to the point of looking for a replacement rather than keep going with what we were using.

As it turns out, there are relatively few alternatives to LDAP in general, and even fewer supported by most Linux distributions. Out of the box, most Linux distributions can support LDAP, NIS, and Hesiod. Although NIS is still well supported by most Linux distributions, it suffers from security issues, and many same issues with regards to replication and failover. As such, I pushed to replace LDAP with Hesiod, which was originally designed as part of Project Athena.

Project Athena

Hesiod was one of the many systems to originate out of Project Athena, a joint project launched between MIT, DEC, and IBM in the early 80s to create a system of distributed computing across a campus, eventually terminating in 1991. Designed to work across multiple operating systems, and architectures, the original implementation of Athena laid out the following goals:

• To develop computer-based learning tools that are usable in multiple educational environments
• Establish a base of knowledge for future decisions about educational computing.
• Create a computational environment supporting multiple hardware types
• Encourage the sharing of ideas, code, data, and experience across MIT

As such, work coming from Project Athena was released as free-and-open source software, and provided a major cornerstone in early desktop and networking environments that are commonly in use today such as X Windows, and Kerberos.

As of 2015, 34 years after Athena was started, its underlying technology is still at MIT today, in the form of DebAthena.

Overview of Hesiod

Moving away from the history, and onto the actual technology itself, as indicated above, Hesiod is based in DNS, and takes the form of TXT records (the TXT record type itself was designed for Hesiod, as was the HS class). A sample Hesiod record for a user account looks like this:

mcasadevall.passwd      IN TXT          "mcasadevall:*:2500:2500:Michael Casadevall:/home/mcasadevall:/bin/bash"
2500.uid                IN CNAME        mcasadevall.passwd
mcasadevall.grplist     IN TXT          "sysops:2501:dev_team:2503:prod_access:2504"

For those familiar with the format of /etc/passwd, the format is obvious enough. Out of the box, hesiod supports distributing users and groups, printcap records (for use with LPRng), mount tables, and service locatator records. With minor effort, we were also able to get it to support SSH public keys. Since Hesiod is based on DNS, data can be replicated via normal zone transfers, as well as updated via dynamic DNS updates. Since DNS is not normally enumerable in normal operation, CNAME records are required to allow lookups for ids to be successful.

New types of records can be created by simply adding a new TXT record. For instance, for each user, we encode their SSH public keys as a (username).ssh TXT record. The standard hesinfo can properly query and access these records, making it easy to script:

mcasadevalllithium~$ hesinfo mcasadevall ssh
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4T3rFl8HondKnGq3+OEAoXzhsZL3YyzRIMCFQeD6aLLHCoVGAwUs3cg7bqUVshGb3udz5Wl/C4ym1aF5Uk5xaZWr2ByKZG6ZPFQb2MZbOG+Lcd5A14gSS2+Hw6+LIoMM8u6CJvIjbTHVI2wbz/ClINDEcJC0bh+YpuaKWyt2iExHATq153ST3dih+sDDK8bq6bFMKM8sdJHl9soKGo7V7i6jIn8E84XmcdTq8Gm2gt6VhOIb/wtr1ix7nxzZ7qCxAQr//FhJ8yVsmHx7wRwkndS7muPfVlVd5jBYPN74AvNicGrQsaPtbkAIwlxOrL92BsS6xtb+sO2iJYHK/EJMoQ== mcasadevall@blacksteel

As such, Hesiod is easy to expand, and provides both command line applications, and the libhesiod API to both query and expand the information Hesiod is able to deliver, and can be deployed to any environment where a sysadmin can control DNS records. As of writing, a set of utilities to integrate and easily manage Hesiod on Amazon EC2's DNS Service (known as Route53) exist in the form of Hesiod53.

Drawbacks to Hesiod

Hesiod inherits several drawbacks due to being based upon DNS. Primarily, it can be affected by various cache poisoning attacks, or hijacking upstream DNS servers. These weaknesses can be mitigated by implementation of DNSCurse or client-side validation of DNSSEC records (standard DNSSEC does not autheticate the "last mile" for DNS queries). Like NIS, if password hashes are stored in Hesiod, they're world-readable, and vulnerable to offline analysis; for this reason, Hesiod should be deployed alongside Kerberos (and pam_krb5) for secure authentication of users and services. At SN, we've been using Kerberos since day 1 for server-to-server communication (and single-sign on for sysadmins), so this was trivial for us. Other organizations may have more difficulty.

Furthermore, under normal circumstances, DNS records can not be enumerated, and nss_hesiod will not provide any records if an application queries for a full list of users (for example, getent passwd on a shell will only return system local users). This may break some utilities who are dependent on getting a full list of users, though in over a month of testing on our development system (lithium), we weren't able to find any sort of breakage.

Finally, although this problem is not inherent to Hesiod, at least on Linux systems, attempts to query users not in /etc/passwd can hang early boot for several minutes. The same issue manifests itself with use of nss_ldap and SSSD. As of writing, we have not determined a satisfactory workaround for the problem, but as our core services are redundant and support automatic failover, a 5-10 minute restart time isn't a serious issue for us.

Finally, although most UNIX and UNIX-likes support Hesiod, there's no support for it on Windows or Mac OS X.

In Closing

Due to its ease of use, we're expectant that Hesiod will drastically reduce the pain of system administration, and removes a service that has proven to be both problematic, and overly complex. While I don't expect a major upswing in Hesiod usage, in practice, it works very well in cloud environments, and for those who find the use of LDAP painful, I highly recommend you experiment in evaluating it as long as one is mindful of it's limitations

I hope you all enjoyed this look at this rather obsecure, but interesting piece of history, and if people are interested, I can be tempted to write more articles of this nature.

~ NCommander

Avoiding Community Fragmentation

Every time the topic of expanding SoylentNews comes up, there's a fear that we may fragment what is already a very small community. While I understand where these concerns are coming from, I'm not sure that fear is justified. My thought here is that if you're reading SoylentNews for the articles and community, being able to stay on SN to read more niche topics would keep you here. For example, I'm a fairly avid Dwarf Fortress fan, but I can't discuss the game here; as such, I either post on /r/dwarffortress or on Bay12 about it. A Dwarf Fortress community on SoylentNews would allow me to discuss one of my favorite games here instead.

The intent is to keep the main page of SoylentNews as it is right now; a source of general news and information; the editors will remain in control of what is (or isn't) posting to SN front page. I don't want to move to a reddit (or firehose) style voting system for articles since I feel that would: lower the quality of content here, cause unpopular information to get buried, and wreck what IMHO has been a rather good system thus far. Individual nexuses may decide to use different criteria for their information, but said content would be limited to that nexus. Or in other words, this would be a purely additive change, not a revolutionary one.

Community Governance

Another thing we've promised since near the start is involving the community on major decisions and policies that this site would take. As such, it's been a relatively informal system with an article going up, feedback collected, and then acted on. I feel that if we're going to expand the site, a more formal framework of governance needs to be established — acting as a checks-and-balances system for the entire site. The fact of the matter is the original Slashcott, or reddit's current self-destruction process could have been averted had the community had a proper say in governance and other actions. For example, Wikimedia, the Ubuntu Project, Debian, and the Apache Foundation all have an elected set of users that act as community advocates.

Having community governance in effect is supposed to act as a circuit breaker — to prevent the staff from acting against the wishes of the community. We have a proven track record of being forefront in both listening to and acting on community concerns, but the entire system is dependent on the goodwill of the staff. People change as time goes on and it's possible that if we're here in ten years, none of the current staff and developers will still be here. Lest us forget that the green site was a haven for many of us until Dice took over. For SoylentNews to survive indefinitely, we need to have a system in place to make sure that the goodwill of the staff isn't the only thing keeping us from going into the abyss.

The problem is: where we do we define this line? Too much debate would cause everything on SN to grind to a halt; too little would prevent governance from being effective.

Furthermore, I'm concerned that a traditional-style community council would be ineffective. I struggle to remember any case where such a system really acted as true force of power in any organization I've been involved in. As such, I would like to think that we may want to mimic national conventions system from Article V of the US Constitution. Specifically, if X number of users (where X is a large percentage of the community), or nexus admins (acting as representatives of their community) across the site form and sign a petition, we could set up a convention to allow the community to overrule the staff and reform the site as necessary. Such a system would give the staff a relatively free hand in day-to-day operations, while acting as an effective circuit breaker to allow the collective force of the community to come to bear if it's ever necessary. This system can be directly incorporated into the bylaws, giving it legal power to enforce its demands, and not some wishy-washy system that can be ignored. I'd like to hear suggestions and feedback from the community on how best to proceed on establishing a system that allows the community to have significant power if it ever should need it.

Monetization and Revenue Sharing

I know this is going to be a touchy subject, so I want to clarify that nothing is set in stone. The simple fact of the matter though is that SoylentNews PBC requires money. As of right now, we can cover our server hosting costs with revenue coming in from subscriptions. This is a very good place to be, but I'd like to do better. I've made no secret that I'd love to get to the point that, long term, SN can do independent journalism, or at least have part or full time staff dedicated to improving the site. In the shorter term, I'd like to have the resources to form a parent not-for-profit to oversee the site, and the mission objectives as laid out by the manifesto, and even perhaps pursue 501(c)(3) status.

For those who don't remember, when I discussed incorporation originally, we hit a major snag that SN follows such an unusual business model, combined with the fact that most not-for-profit corporations deal with things like parks or fire departments. While it is certainly possible to form a not-for-profit that covers SN, it would require a lawyer to determine the specifics, in addition with the usual costs of forming a business. With that in mind, SoylentNews PBC (our legal overlord), simply doesn't have the resources to do that. Forming the non-profit and making sure checks-and-balances are directly incorporated into the bye-laws would make sure the site would never be at risk of a buyout, regardless of who is leading the site, and a cornerstone in fulfilling our promise to the community.

Furthermore, there's the moral aspect to consider. It's a simple fact that without the community, we wouldn't be here, or as successful as we have been. As such, if we end up monetizing community nexuses, at least part of those funds should go to those who volunteer their time and effort here, both the staff and the overseers of a nexus.

I've got a couple of ideas that I'd like to bounce off the community to see what the general feelings are. This is broken into two parts: monetization ideas, and revenue sharing.

Monetization Ideas

Require Payment To Create A Nexus

This was the most obvious idea I had when I started drafting this novel. Pay a bit, and create your own community. I'm not really a huge fan of this idea, because it means that someone has to pay to create a place to discuss things, and my gut is telling me that this would go against our mission statements, even if on paper it seems completely reasonable to me. There is perhaps a middle ground that we could limit nexus creation to subscribers. Overall, I'm very much on the fence for this idea though.

Offer A Premium Tier

As a second option, we could allow nexuses to get additional functionality, such as the ability to fully re-theme their section of the site, have nexuses-within-nexuses, provide subscriber features to all users within a nexus, or provide general file and image hosting. I'm largely open to ideas on both what we could offer, and how much it could cost.

Other Ideas

Beyond these two, I've also considered the possibility of allowing community nexuses to run their own advertising, or selling hosted independent rehash instances for a turnkey website. I'm not sure either of these are good ideas (though hosted instances may be useful as a side business), which is why I didn't write about them in length. I'm of course always open to good ideas from the community on the subject

Revenue Sharing

I've said it before, and I will say it again, but this site is nothing without its community. If we're successful in increasing our revenue, then part of that money should be given back to that community, either in the form of free subscriptions or nexus upgrades, or as cold hard cash. Any of the editors here can tell you that building a community of any sort is a massive job and very time consuming. The admins of community nexuses will have to face the same challenges and time commitments that the editorial team current does. As of right now, subscriptions cover our hosting and legal costs, but not much more. If I could, I would give every person who has volunteered their time and effort a paycheck, but that's simply not feasible. What we can do, however, is set a small portion of incoming revenue aside for re-investment.

Once community nexuses are live, subscribers (or those re-upping a subscription) will be able to set a nexuses that gets part of those funds (the default will be the nexuses the user is currently browsing), and the option to "leave a tip" so to speak. Most of that money will go into the general fund to pay for the site or to build a legal war chest, but the tip will be set aside, and placed in a fund for that community. That fund can be used by admins of a given nexus to give their users free subscriptions, buy the premium tier, or other site related functions. We can also create the possibility of "cashing out" so to speak, though that will require discussions with our CPA and lawyer. I realize that this is unlikely to ever generate a significant amount of money, but it may allow for a local gathering with free beer or something. Specifics (and legalities) have to be hashed out, but I would love to hear the communities thoughts on this.

In Closing

Its been a wild 1.5 years, and this site has grown far beyond my initial expectations. While I can't say what the future holds, I want to make sure we have the ability to cement our future in a more permanent fashion, and not fall victim to the same pitfalls that destroyed (or are destroying) sites like Slashdot, digg, or reddit. Its possible this is all a bad idea, and I'm depending on everyone to get your feedback and to readjust things.

On one final note, a few users keep asking us about warrant canaries. I've never done one of these before, but I'm hoping that this will help assure those who are concerned that we've been warranted or something:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

SoylentNews PBC has never received any requests for law enforcement, or
has diverged any user information as required by a court of law or similar.

Headlines of July 5th as of writing:

The Climate One Year On: Exit Carbon Tax, Enter Brown Coal

Contact Temporarily Lost With New Horizons

NVIDIA Shows a Realistic HairWorks 1.1 Demo with 500K Fluttering Hairs

President/CEO
Michael "NCommander" Casadevall

Signed with:

pub   4096R/D2247639 2011-05-12 [expires: 2016-06-09]
      Key fingerprint = 37F0 1189 3BAE 3611 C45B  8E15 733E 1A42 D224 7639
uid                  Michael Casadevall <mcasadevall@ubuntu.com>
uid                  Michael Casadevall <mcasadevall@debian.org>
uid                  Michael Casadevall <mcasadevall@kubuntu.org>
uid                  Michael Casadevall <mcasadevall@soylentnews.org>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBAgAGBQJVmfrkAAoJEHM+GkLSJHY5gKsP/3yfJjvvmgmZWXD9y2nlpqiG
xnKsVeuQcQRg/x4K42SRuDG7IvMdzM7DOqs8swD/UNhkms5nVt8YpjCBe8pmIwnz
L89O3IyqWVVApx5FxHP7Keve1vhg+Gdv9Ww7kFBqR8FvuUU6qEz/CsPzCZwBUdy9
vyvtNZ0dONgfxdKviMQFewBfzRXoqF8D7jT4yhzuKiDCgXMg/zXL6CAOI5i8qovJ
0CLx/7lWTDApmk0UsHyQg35e3IP4ETLl+wTvB8mEGdgWrctxDL8j4uhnhzuC3EvD
XFBkKZ7r3nHk4T6BVJe3AYCamT0d2itZWPEJJ1XTeA6Pjceifp8x2pbx47AaWCkE
PVmE+CVvS7Uxm6gM08uUlCrpN8lOZFrOh+zftNHpUTm9glPAOIvaJTgWGJjOGMBH
Faa1vr5H+jDmrIcyQFLWDACH95vaaDzb9mvrEJnm30ZzEwXYv8YfYErQL6MAouTQ
8GZxbQVEbt6ODM1aIQoTemHUWXGmMFdrlKURgssPgAAEMrS7COrPnjg1Rr8T60U6
05k9vKYggTGeOjzSOB5ls8XpffTOLheYjCMqbH3UfYkr6wgHiTox/GqzLsRtKPVH
7Mg1/WazWWjeCu/d0qO9GpurNK6knTSIxV/dTCqVF7l+RBdXfGyoYMEKbqZbuiB8
s2G/1OFnSE2GhvT2OQTu
=yZhY
-----END PGP SIGNATURE-----

If anyone wishes to build a chain of trust to my GPG key to verify this message, I'm based in western New York, and available for key signing parties. ~ NCommander

Necessity of The Upgrade

Rehash was by far the largest and most invasive upgrade to the site, requiring modifications to nearly every component. To understand what went wrong, a full understanding of the context of the rehash upgrade is important, so I apologize in advance if this is a bit wordy; much of this information was present in previous upgrade posts, and comments, but I want to put it in all in one canonical spot for ease of reference

For those unfamiliar with mod_perl, the original Slash codebase was coded against mod_perl 1, which in turn was tied to Apache 1.3 — which, by the time we launched this site, had been unsupported for years. Thus it was known from day one that if we were going to be serious on both continuing on the legacy Slash codebase and keeping SoylentNews itself going that this was something that was going to have to be done. Not if, but when.

As a stopgap, I wrote and applied AppArmor profiles to try and mitigate any potential damage, but we were in the web equivalent of Windows XP; forever vulnerable to zeroday exploits. With this stopgap in place, our first priorities were to improve site usability, sort out moderation (which of course is always a work in progress), and continue with the endless tweaks and changes we've made since go-live. During this time, multiple members of the dev team (myself included) tried to do a quick and dirty port, with no success. The API changes in mod_perl were extensive; beside obvious API calls, many of the data structures were changed, and even environment flags were changed. In short, every single line of code in the site that interacted with Apache had to be changed. In other words, every place the codebase interacted with $r (the mod_perl request object) had to be modified, and in other places, logic had to be reworked to handle changes in behavior such as this rather notable example in how form variables are handled. Finally, after an intensive weekend of hacking and pain, I managed to get index.pl to properly render with Apache2, but it became clear that due to very limited backwards compatibility, the port became all-or-nothing; there was no way to simply piecemeal the upgrade.

Furthermore, over time, we'd also noticed other aspects which were problematic. As coded, Slash used MySQL replication for performance, but had no support for multi-master or failover. This was compounded by the fact that if the database went down, even momentarily, the entire stack would completely hang; apache and slashd would have to be manually kill -9ed, and restarted for the site to be usable. This was further complicated in that in-practice, MySQL replication leaves a lot to be desired; there's no consistency to confirm that both the master and slave have 1:1 data. Unless the entire frontend was shutdown, and master->slave replication was verified, it is trivial to lose data due to an ill-timed shutdown or crash (and as time has shown, our host, Linode, sometimes has to restart our nodes to apply security updates to their infrastructure). In practice, this meant that failing over the master was a slow and error-prone process, and after failover, replication would have to be manually re-established in the reverse direction to bring the former-master up to date, then failed back by hand.

While MySQL 5.6 implemented GTID-based replication to remove some pain, it still failed to be a workable solution for us. Although it is possible to get multi-master replication in vanilla MySQL, it would require serious tweaks to how AUTO_INCREMENT work in the codebase, and violated what little ACID compliance MySQL has. As an aside, I found out that the other site uses this form of multi-master replication. For any discussion-based website, the database is the most important mission critical infrastructure. Thus, rehash's genesis had two explicate goals attached to it:

• Update The Underlying Software Stack
• Find A Solution To The MySQL Redundancy Problem

The MySQL redundancy problem proved problematic, and short of simply porting the site wholesale to another DB engine, the only solution I could find that would keep ACID compliance of our database was with MySQL cluster. In a cluster configuration, the database itself is stored by a backend daemon known as ndbd; instances of mysqld act as a frontend to NDB. In effect, the mysql daemon becomes a frontend to the underlying datastore, which in turn keeps everything consistent. Unfortunately, MySQL cluster is not 100% compatible with vanilla MySQL. FULLTEXT indexes aren't supported under cluster, and some types of queries involving JOINs have a considerably longer execution time if they cannot be parallelized.

As an experiment, I initialized a new cluster instance, and moved the development database to it, to see if the idea was even practical. Surprisingly, with the exception of the site search engine, at first glance, everything appeared to be more or less functional under cluster. As such, this provided us with the basis for the site upgrade.

One limitation of our dev environment is that we do not have the infrastructure to properly load test the changes before deployment. We knew that were going to be bugs and issues with the site upgrade, but we were also starting to get to the point that if we didn't deploy rehash, there was a good chance we won't do it at all. I subscribe to the notion of release early, release often. We believed that the site would be mostly functional post-upgrade, and that any issues encountered would be relatively minor. Unfortunately, we were wrong, and it took four site upgrades to get us back to normality which entailed: rewriting many queries, performing active debugging on production, and a lot of luck to get us there. All things considered, not a great situation to be in.

Because of this, I want to work out a fundamental plan to prevent a repeat of such a painful upgrade even if we make large scale changes to the site, and prevent the site from destabilizing even if we make additional large scale changes.

Improving Documentation

On a most basic level, good documentation goes a long way in keeping stuff both maintainable and usable. Unfortunately, a large part of the technical documentation on the site is over 14 years old. As such, I've made an effort to go through and try and update the PODs to be more in line, including, but not limited to, a revised README, updated INSTALL instructions, notes on some of the quirkier parts of the site and so forth. While I don't expect a huge uptake of people running rehash for their personal sites, being able to run a development instance of the site may hopefully increase the amount of involvement of drive-by contributions. As of right now, I've implements a "make build-environment" feature which automatically downloads and installs local copies of Apache, Perl, and mod_perl, plus all CPAN modules required for rehash. This both makes it easier for us to update the site, and get security fixes from upstream rolled in.

With luck, we can get to the point that someone can simply read the docs and have a full understanding of how rehash goes together, and as with understanding is always the first step towards succeeding at anything.

Database Upgrade Framework

One thing that came out of the aftermath of the rehash upgrade is that the underlying schema and configuration tables between production and dev have deviated from each other. The reason for this is fairly obvious; our method of doing database upgrades is crud at best. rehash has no automated method of updating the database; instead queries to be executed get written to sql/mysql/upgrades, and executed by hand during a site upgrade, and the initial schema is upgraded for new install. The practical end result of this is that the installation scripts, the dev database, and production all have a slightly different layout due to human error. Wherever possible, we should limit the amount of manual effort required to manage and administrator SoylentNews. If anyone knows of a good pre-existing framework we can use to do database upgrades, I'm all ears. Otherwise, I'll be looking at building one from scratch and intergrating it into our development cycle.

Unit Testing the Site

For anyone who has worked on a large project before, unit testing can be a developer's best friend. It lets you know that your API is doing what you want and acting as expected. Now, in normal circumstances, unit testing is difficult to impossible as much of the logic in many web applications is not exposed in a way that makes testing easy, requiring tools like Windmill to simulate page inputs and outputs. Based on previous projects I've done, I'd normally say this represents more effort than is warranted since you frequently have to update the tests even for minor UI changes. In our case, we have a more realistic option. A quirk of rehash's heritage is that approximately 95% of it exists in global perl modules that are either installed in the site_perl directory, or or in the plugins/ directory. As such, rehash strongly adheres to the Model-View-Controller design and methodology.

As such, we have a clear and (partially) documented API to code against which allows us to write simple tests, and confirm the output of the data structures instead of trying to parse HTML to know if something is good or bad. Such a test suite would have made porting the site to mod_perl 2 much simpler, and will come in useful if we ever change database engines or operating system platforms. As such, I've designated it a high priority to at least get the core libraries connected with unit tests to ensure consistent behavior in light of any updates we may make. This is going to be a considerable amount of effort, but I strongly suspect it will reduce our QA workload, and make our upgrades close to a non-event.

In Closing...

The rehash upgrade was a wakeup call for us that we need to improve our processes and methodology, as well as automate aspects of the upgrade process. Even though we're all volunteers, and operate on a best-effort basis, destabilizing the site for a week is not something I personally consider acceptable, and I accept full responsibility as I was the one who both pushed for it, and deployed the upgrade to production. As a community, you've been incredibly tolerant, but I have no desire to test your collective patience. As such, in practice, our next development cycle will be very low key as we work to build the systems outlined in this document, and further tighten up and polish rehash. To all, keep being awesome, and we'll do our best to meet your expectations.

~ NCommander

During debugging, I would notice we'd almost universally get a bad load from the cache if varnish or the loadbalancer burped for a moment, As best I can tell running backwards through traces and various error logs, the underlying cause of the 500s and bad page loads is one of two causes: either we're getting bad data from memcached on a cache read, or a bad load into cache from the database. Of these two, I'm leaning closer to the former, since if we were loading bad data into memcached, we'd see consistent 500s once the cache was corrupted.

It's somewhat difficult to accept that memcached is responsible for our corruption issues; we've been using it since golive a year and a half ago. However, given a lack of other leads, I flipped the memcached config switch to off, and then loadtested the site to see how bad the performance drop would be. Much to my surprise, the combination of query optimizations, the faster apache 2 codebase, and (for the most part) increased responsiveness due to having a multimaster database seems to be able to cover the slack for the time being. As of writing, memcached has been disabled for several hours, and I've yet to see any of the telltale signs of corruption in error_log. I also need to note that its the middle of the night for the vast majority of our users, so this may just be the calm before the storm.

I dislike using production as something of a guenna pig, but given the very transitory nature of this bug, and our inability to reliably reproduce it on our dev systems, we're left somewhere between a rock and a hard place. I would like to thank the SoylentNews community for their understanding over the last week, and I hope normality may finally return to this site :)

~ NCommander