SoylentNews

We're testing a new configuration between the site and the database. There may be unexpected issues with the site while we're testing. Keep calm and carry on.

So I'm pretty sure you're all aware, but I've gone through and done a massive amount of work on the backend and infrastructure in the name of sanity, proper user permissions and such, and documenting as much as I can.

As a note, a lot of this was brought on by the fact we have relatively credible threat against the site, so I wanted to go through and make sure everything was in good shape and hardened (there's a lot of good bits here). I might have gone overboard. Here's the cliff notes version of what was done.

  * Static Status Page

http://status.soylentnews.org

This is on boron in /var/www, we should probably move it to Oxygen in case the entire linode DC goes down, but its fine there for now

  * Through documentation on node access, SSH, etc.

Basically, the links here http://wiki.soylentnews.org/wiki/SystemAdministration are required reading for all staff who play with dev, or production.

There are still gaps, varnish, slash, and apache only have limited documentation which is outdated, but I'll try and get those written in the next few days

  * Node renaming

This one might seem silly, but its sometimes hard to know what we're refering to when we talk about webserver/etc and a specific node. While at the moment we have no redundancy, I changed the hostnames of everything. The original soylent-* names are aliased in the internal DNS. List is here:

http://wiki.soylentnews.org/wiki/SystemAdministration/TheHitchhikersGuideToTheli694-22Domain

  * Internal DNS

Major thanks to xlefay for getting this up and running. All nodes exist in an internal li694-22 TLD, and are both forward and reverse resolvable (needed to make kerberos work properly, and make life easier).

  * Dev server

Announced, but falls into stuff done this weekend :-).

  * Varnish

I drastically reworked the varnish configuration file for better performance. The server is considerably more responsive than it used to with apache hit considerably less. As a side effect, slash hitcounts will be skewed as ACs will not be counted.

Rate limiting to prevent DOS was implemented, and xlefay pounded the dev server with some impressive apachebench numbers to confirm we won't go down. The dev server is much more loaded than production due to sharing the database, so I'm optimistic it will take a serious effort to pound us into oblivion with just ab or similar tools from a few nodes.

  * Disabled static page generation

This has been a PITA and on the TODO for awhile. Dynamically generated pages are now used for articles and comments. Varnish caches for ACs on a 5 minute basis. Logged in users get access to the site directly

  * SSL on Production

Doesn't fully work, but I reworked the nginx termination, and the varnish configuration so it is possible to login and use SSL. slash redirects the login to http, but the cookie gets properly set now so if you login SSL then reload the SSL page, it works. Need someone to dig into slash and figure out why ConnectionIsSSL is returning false. Need a volunteer to setup nginx termination on dev to debug.

  * LDAP setup

God, this was a pain, but we have a full LDAP setup on helium now. Replication to boron is on the TODO list, so if helium goes down, SSH authethication goes down, which is a bad thing. People with linode accounts can access the console and log in as root directly

Documentation (with pictures!) here: http://wiki.soylentnews.org/wiki/SystemAdministration/LDAPManagementForDummies

  * Passwords logged and recorded

Went through, made sure every password is saved in a master PW file which is in helium in root's home directory. sysops should keep a local copy of this file as its needed to use lish to access boxes should LDAP be down. Other important passwords like mysql, LDAP, and kerberos are also in this file.

  * Centralized ACLs

All machines require that a user be in the correct POSIX group to access them. List of groups is available here. This ensures that also everyone who has access can have it

http://wiki.soylentnews.org/wiki/SystemAdministration/GroupPermissions

  * SSH Policy

This one probably going to cause me some flack, but you need to go through the staff box (boron) to access any more. I don't like having open SSH ports on any of our nodes because it feels like we have our balls in the wind and a misconfiguration can leave us vulnerable.

I'm not kidding on that last bit. On production for the last month, slash:slash has worked as a username and password to log into the slash account. Using LDAP doesn't solve this as we still have local accounts for things LIKE slash.

Everyone must use SSH public key to autheticate; keys are stored in LDAP and are pulled on the fly by OpenSSH (this required updating OpenSSH on all nodes with a backport).

I know that due to slashd seizing up at a bad time this caused people to get locked out as I haven't gotten SSH keys from most people. I've got 8 users now with keys in LDAP. Right now, I don't have all the sudo files fully massaged, so if you have access to the dev server, you also have full sudo on all nodes. This isn't really desirable as I believe in limiting permissions, but this is a case of preventing us from going SNAP. Looking for someone to work out the necessary sudo voodoo

Also need someone to write upstart files for apache 1.3 so it comes back on a restart (xlefay is doing this, but feel free to work with him)

  * New Node Bringups

lithium (dev server), carbon (IRC server), and oxygen (offsite backup) were brought up this weekend. Bringup documentation was written here: http://wiki.soylentnews.org/wiki/SystemAdministration/TheRiseAndFallOfNewNodeManagement

  * OpenVPN

Setup a OpenVPN server on boron with magic iptables setup to allow oxygen to access all nodes. There's a fair bit of magic going on here, and I don't have the setup documented yet, but its basically following the Ubuntu Serer documentation for OpenVPN, plus a few iptable rules (saved in /etc/iptables.rules) on boron. Should be pretty self-explainatory.

  * Kerberos

To handle users that can't use ProxyCommand, to make life easier for internode stuff, and to be sexy, kerberos was setup to allow single signon. As most people probably never have managed Kerberos, the quick start guide is here: http://wiki.soylentnews.org/wiki/SystemAdministration/KerberosAdministration

Kerberos replication is setup, but not running as I need to make sure everything is sane. KDC master is helium, slave is boron.

  * AppArmored Apache

This was the real reason for the scheduled downtime last week as we had to migrate to apparmor capable kernels. AppArmor is basically SELinux but less braindead, and I handwrote a config that essentially puts Slash in a straightjacket. This should prevent things like process exploitation or a bug in slash from getting any traction. The apparmor config is installed on both lithium and hydrogen and is in /etc/apparmor.d. If you take a look, Apache can't take a piss without explicate permission :-).

(note, this doesn't do much to help us with SQL injections but every bit helps. Nothing short of a full rewrite of MySQL.pm to use stored procedures will fix this. Any takesr? (or migrating us to pgSQL then doing this?)

There's more to do here, slashd should be apparmored as well, but thats more difficult, and as its not directly user accessible, I'm less concerned that with apache itself. Ideally, every userfacing component should be apparmored (nginx, varnish, and slashd), but the former two run under very restrictive user accounts, and slashd only works with data in the database that already passed through Apache, and for the most part is just simple maintenance scripts, so its not that easy to attack.

I need to write up and document apparmor like I did for other things, but its relatively idiot proof to write files, and it makes good logs in /var/log/syslog.

  * Preparations for offsite backup

We've got a dedicated server (oxygen) with a 500 GIB HDD from http://www.kimsufi.com/en/ for €10 a month in France (oxygen) This will be used for offsite backups. xlefay looking and will be implementing this for all nodes.

  * Ubuntu package repo

As we need to maintain at least one backport, and need other things packaged, I setup a Launchpad PPA to do package building and binary distribution to all nodes: https://launchpad.net/~li69422-staff/+archive/backports-for-precise

This repo is added on all nodes. As you need to know how to do Debian packaging to use it, build an example package or two, and then I'll add you to the team. Its pretty straight forward on how to do this.

  * Staff userdir

Any staff can generate a userdir on boron by creating a public_html and using staff.soylentnews.org/~username

I've updated my voting system to allow preferential voting (the previous method uses approval voting). It gives a score to each candidate based on its rank. E.g. if there are 10 candidates, and you give a candidate 1st preference, it gets 10 points, second preference gets 9 points, etc.

It's therefore functionally equivalent to "range voting".

<?php

/* How to use:
    Put all emails in an array with values in $emails["text"] and $emails["user_id"];
    Put the list of valid candidates in the function valid_candidate();
    Put the check for user-authorisation in function valid_user();
    Enjoy.
*/

$emails_array = array(
    array("user_id"=>234, "text"=>"
            option1 = 1
            option3 = 2
            optionwhatever = 3
        "),
    // Duplicate user, will be handled correctly.
    array("user_id"=>234, "text"=>"
        Oops forgot one I like:
        Optionfour = 2
        // Did I mention:
        option1=1
        "
        ),
    array("user_id"=>1234,
        "text"=>"
        // I hate option1
        Option1 = 6
        option2 = 1
        ")
    );

function valid_user($user_id){return true;} // put user filter in here if necessary
function valid_candidate($name){ return true;} // is the name one of the candidates?

$number_of_options = 6;

// Process all emails
foreach ($emails_array as $email)
{

  // Is it a valid registered user?
  if (valid_user($email["user_id"]))
  {

    // Process each line of the email
    $email_lines = explode("\n", trim($email["text"]));
    foreach ($email_lines as $this_line)
    {
      // Does it have an '=' sign and only one = sign
      $equals_sign = strpos($this_line, "=");
      if ($equals_sign !== false)
      {
        $cleaned_up_line_text = trim($this_line, ";.!\t\n\r\0");
        $parts_of_line = explode("=", $cleaned_up_line_text);
        if (count($parts_of_line) == 2)
        {
            // Is it a valid candidate and rank?
            // Candidate is in lower case.
            $candidate = strtolower(trim($parts_of_line[0]));
            $rank = intval(trim($parts_of_line[1]));
            if (valid_candidate($candidate) && ($rank > 0) && ($rank <= $number_of_options))
            {
                // Get the score for this candidate.
                // The score is the number of options - how far it is ranked below 1.
                // e.g. a rank of 1 would give it a score of $number_of_options.
                // a rank of 2 gives it a score of $number_of_options -1.
                // See "range voting".
                $score = $number_of_options - ($rank-1);

                // Make sure this vote for this user hasn't already been cast
                if (!isset($user_votes[$email["user_id"]]) ||
                    !isset($user_votes[$email["user_id"]][$candidate]))
                {
                    // Remember this user has voted for this name already.
                    $user_votes[$email["user_id"]][$candidate] = true;

                    // Count the vote towards the total
                    if (!isset($candidate_votes[$candidate]))
                        $candidate_votes[$candidate]=$score;
                    else $candidate_votes[$candidate]+= $score;
                }
            } // end of check for valid vote values.
        } // of check for correctly formatted vote
      } // of check for equals sign
    } // End of for loop for lines of email
  } // of check for valid user.
} // end of for loop for all emails.

print "votes:<br>";
print_r($user_votes);
print "<br><Br>";
print "candiate_votes<Br>";
print_r($candidate_votes);

?>

I came to Facebook quite late having stubbornly refused to create an account as I didn't see the point. I don't have many friends (by choice) and I simply don't care enough about other people's baby photos or daily diatribes about their current diet or fashion based purchases.

I tried Twitter when it was gaining momentum but decided quite quickly that it was broken as a social sharing model. However, it is great as a broadcast medium much in the same way that RSS is or was, but without the high technical barrier to entry. This is borne out by the massive follower numbers any celeb-de-jour can garner on there. The common people have mostly have settled into using twitter as read-only news feed, occasionally throwing out comments of their own in to maelstrom hoping that someone else will bite and reply back but secretly knowing that their tweets will be ignored just like the other 99% of non-celeb/corporate tweets are.

I never had a Myspace account as I'm about 20 years too old, even when it was popular. My understanding of that platform is that it too has been commandeered by the major music labels to provide yet another podium to pimp their artist's wares upon. The common user space over there is now rapidly becoming a wasteland of ill tended, dying on the vine profiles.

I use LinkedIn for work. By 'use' I mean I have a fairly full profile and I connect with people I work with or have worked with in the past and hope to again in the future. Beyond that, I don't think LinkedIn serves a purpose. It certainly didn't help me when I was looking for work, despite putting out pleas to my network. I could live without LinkedIn, and my career wouldn't be affected one way or another.

I used to be lucky enough to be one of select few bloggers on the BCS (British Computer Society) website. I had to submit a handful of written work to prove I could type more than a couple of sentences without collapsing into netspeak and emoticons. Unfortunately the BCS did a major website update a while back (when the logo went green in colour for those who keep track) and they not only demoted the blogs to the backend of the site but they also took away the self-publishing pages. This meant that any submission had to be emailed in and 'approved' for publication. I felt that this was no longer 'blogging' and as I didn't get paid add content to their website I decided to quietly stop writing for them.

Which brings me rambling to my point; Social media for me is broken. Once the clamouring masses get access to any social media platform, the celebrities and the corporate drones follow, ever eager to make another buck by selling their products to the witless idiots that follow them. This in turn makes pages hits go up, and the social media platform feature-set gets refreshed around this celebrity-centric surge, leaving those of us who do not care for mass idolatry struggling with a platform that no longer fits our needs or wishes.

Google+ was good; it was starting to develop into a social platform that encouraged the sharing of ideas, the dissemination of interesting information around obscure but fascinating topics, and the meeting of minds with common interests. This uniqueness is slowly fading away as the masses jump on board and demand their Jersey Shore and Britney Spears pages so they can slavishly follow them. Google are a company that wants to make profit like any other company, so they will provide the features that the masses want. This leaves those of us with high brow discussion tastes looking for another raft to nail our flag upon.

In the main though, I know that I am part of a minority, and I came to terms with this fact many years ago. What I want, is rarely what anyone else wants, leaving me with two options; Keep looking for the next-best-thing that may suit my needs, or just post endless streams of LOLCats and funny videos like everybody else.

[Cross-posted from http://www.jaruzel.com/]

1. Corrupted User Settings
   There is currently a bug with the way slash handles TimeZone information and user settings. Basically, it seems that user settings are somehow getting corrupted in the database leading to unexpected results. One of those unexpected results is that slash doesn't show you new stories in a consistent fashion. In order to fix it, visit your Homepage (link in the sidebar) -> Restore Defaults -> Set Time Zone -> Revisit the front page. We are aware of this issue and is in the dev pipeline. Please email dev@soylentntews.org if you're interested in helping out the development effort!
2. Static Page generation and caching.
   Anonymous users get statically generated pages for the main index and article pages. This is run by some cron-like process in the backend slash daemon. The particular job that regenerates the static pages and updates the comment counter on the main index along with some other stuff, is "freshenup". This runs on some interval, (I believe its configured to run every 5 minutes currently). Combine that with a Varnish caching server sitting in front of Slash's Apache instance, which will cache for 5 minutes and Anonymous users can see up to a 10 minute lag in some information on the front-page vs. Logged in users, who get dynamically generated content all the time. This is an artifact of our current configuration and can be re-evaluated in the future, but for now, given the site load and the hardware we're running on, it seems to be working well.
3. The "freshenup" task looses it's mind. (ie. SN is actually borked).
   There are times however, when the "freshenup" task goes wrong. At this point, static page generation and comment count updates stop occuring. A sysadmin needs to go bounce slashd at this point. This has occurred approximately 2 times in the month that site has been live. The symptoms are that when browsing as AC, stuff is more than 30 minutes out of date, story links are broken and comment counts are frozen in time (for all users), it could be that the "freshenup" process has hung, but a lag of up to 10 minutes is expected for AC. Please email dev@soylentnews.org if you think this is happening or hop into the #Soylent irc channel -- there are usually a bunch of staff and devs hanging out there.

Hopefully this will help to alleviate some confusion about when you should see differences and give everyone a little bit better idea of all how SN is put together.

We've recently had a credible threat made against the site w.r.t. to a security exploit on slash. While details are somewhat vague, what little information was posted, combined with my knowledge of slash, and reviewing old security posts on slashcode.com suggests we're looking at a potential SQL injection technique.

While slash does considerable data sanitization, and escapes information coming in (so you can't just "; drop table users;"), there are values that require special sanitization on top of the normal. This has been the source of other XSS and SQL exploits against Slash historically (look at the articles on the main index).

I took a brief look at Environment.pm (where the supposed exploit is supposed to live) and said sanitization, and didn't seen anything that immediately jumped out at me, but it is a *lot* of code to look through. A grep through the access logs suggests that no one has tried to execute raw SQL on the site, but its impossible to know for sure.

If you or someone you know is interested in trying help secure SoylentNews, grab the dev VM, the current git tree, and get auditing. If we're informed of any security exploits (which should be sent by mail, and not posted in the comments), we will patch it here, and informal all other slashcode sites that we aware of about the exploit before releasing information about it publicly on the main index.

Do note that IP addresses, and passwords are hashed in the database (although only with MD5; upgrading to stronger hashing is on the TODO list), so an information leak, while bad, is not catastrophic. We do take regular snapshots of the database and of the machines themselves as well, and we'll make sure to post immediately if we become aware of any specific exploit against the site.

Since my ban from IRC I've used a few proxies to check in on the place.

Nope, it's about to go downhill. That's going to spread to the site and that's going to be the end of the site, thanks to thin-skinned powers-that-be.

So I'm calling it now, first rat off this sinking ship.

I've expended enough time and effort trying to help this site get established, promoted, and fixed, and this is what I get. Nope. Sorry, Charlie.

to feed the trolls!

Occasionally, I enjoy the hell out of it.