SoylentNews

Wooo, that nearly killed me to finish it. It's going living at 20:00 UTC (4PM EST), which is the start of our peak hours. We'll be revising it based on community feedback and if other important points come up as time goes on.

Of course, words means only as much as the actions taken behind them, but I think we've been relatively consistent in meeting the goals I outlined. Here's a small sneak preview for those who read my journal til the whole thing goes live

Statement of Purpose

Our aim is to stand in stalwart opposition to these trends. We will be the best site for independent, not-for-profit journalism on the internet, where ideas and free discussion can take place without external needs overshadowing the community.

How to find the most useless page of SoylentNews:

From the dropdown selection button below the story (the one offering "Threaded", "Nested", etc. for comments) select "No Comments", then click "Change".

The option delivers what it promises: No comments. Also no story. Just the settings bar.

Note that this is not a critique (it's clearly just an artefact of how Slashcode works, and not harmful in any way). I just considered it amusing.

So ... I've recently gotten back into minecraft, and figured that perhaps there are other MC players here at SN, so I wanted to know if there was enough interest to setup a MC server in general. I'd probably use CraftBukkit, and I'm open to running mods if others are interesting. Leave a message below if you'd be interested.

Mods I'd like to run:
  * Traincraft
  * Railcraft
  * Mystcraft (useful for getting new ores without having to reset maps; age creation would be restricted to admins though; mystcraft is a server hog).

Leave your thoughts below.

Since we've got a fair number of complaints about us running too many site news articles, I'm going to condemn this to my journal, then link it next time we *do* post something about the site. For a large portion of today (4/16), SoylentNews users had issues with commenting, and moderation was completely hosed. This was due to a backend change; we shifted the site behind a loadbalancer in preparation of bringing up a new frontend and give us considerably more redundancy and latitude with working with the backend.

This change had been setup on dev for the last week with us testing it to see what (if anything) broken, and it was discussed and signed off by all of the staff. Last night, I flipped the nodebalancer to connect to production instead of dev, then changed the DNS A record for the site to point at the loadbalancer.

I stayed up for several hours at this point to ensure nothing odd was going on, and satisfied that the world would keep spinning, I went to bed. What I found though was I broke the formkeys system. Slash knows about the X-Forwarded-By header, a mechanism for when a site is behind a proxy on how to relay client IP information (this mechanism was already used by both varnish and nginx), however, for security reasons, we strip out the XFF header from inbound connections unless its on a specific whitelist. On both dev and production, we had whitelisted the nodebalancer to pass this header in properly.

Or so we thought. Linode's documentation doesn't mention, but the IP address listed in the admin interface is *not* the IP used to connect to the site; instead it uses a special internal IP address which isn't listed or documented anywhere. Our security precautions stripped out the X-Forwarded-By header, and made it appear that all inbound users were coming from the same IP. This wasn't noticed on dev as slash ignores the formkeys system for admins, and the few of us beating on it with non-admin accounts weren't able to do enough abuse to trigger the formkey limiters.

Our peak hours are generally evenings EDT, which means the low traffic at night wasn't enough to trip it either (or at least no one on IRC poked me about it, nor were there any bugs on it on our github page. However, once traffic started picking up, users began to clobber each other, commenting broke, and the site went to straight to hell. When I got up, debugging efforts were underway, but it took considerable time to understand the cause of the breakage; simply reverting LBing wasn't an easy fix since we'd still have to wait for DNS to propagate and we needed the load balancer anyway. After a eureka moment, we were able to locate the correct internal IPs, and whitelist them, which got the site partially functional again. (we have informed Linode about this, and they said our comments are on its way to the appropriate teams; hopefully no other site will ever have this same problem).

The last remaining item was SSL; we had originally opted out of terminating SSL on the loadbalancer, prefering to do it on the nginx instance, so Port 443 was set to TCP loadbalancing. This had the same effect as there is no way for us to see the inbound IP (I had assumed it would do something like NAT to make connections appear like they were coming from the same place). The fix was utlimately installing the SSL certificate on the load balancer, then modifying varnish to look for the X-Forwarded-Proto header to know if a connection was SSL or not. I'm not hugely happy about this as it means wiretapping would be possible between the load balancer and the node, but until we have a better system for handling SSL, there isn't a lot we can do about it.

As always, leave comments below, and I'll leave my two cents.

It has been a little while now that this fledgling community has been around - and it remains one of my favorite stories about communities. A splinter of a much larger community took it upon themselves to challenge the rest and make a move to a new home. Shedding the shackles that were being placed on them was a bold move, but one that has been fantastic.

The community here is great - but here is my question. Overall, we are amazingly tolerant of others, of the choices they make and of their beliefs. I would then be curious, if we are such a tolerant group, how do we address intolerance in our ranks? I recently came across what I can only say filled me with pity and sadness. I find it saddening that in this day and age - and especially in this group - there are still such hate filled people.

But this poses a question - how does a group that is tolerant deal with intolerance within it's ranks? Does our acceptance of others extend to accepting someone that has thoughts and beliefs which are far from the norm within this community, or is there a limit placed on how far from our own values a member of the community may be?

Now that I've had some time to clear my head, I want to expand on my original feelings. I'm pissed off about this, and my temper flared through on the original post. I'm leaving it as is because I'm not going to edit it to make myself look better, and because it sums up my feelings pretty succinctly. How would you feel if something you worked on under the promise of building the best site for a community was regularly and routinely causing corporate firewalls and IDS systems to go off like crazy?

You'd be pissed. Had we known about this behaviour in advance, it would have been disabled at golive or in a point release, and a minor note would have gone up about it. Instead, I found out because we were tripping a user's firewall causing the site to get autoblocked. I realize some people feel this is acceptable behaviour, but a website should *never* trigger IDS or appear malicious in any way. Given the current state of NSA/GCHQ wiretapping and such, it means that anything tripping these types of systems is going to be looked at suspiciously to say the least. I'm not inherently against such a feature (IRC networks check for proxying for instance), but its clearly detailed in the MOTD of basically every network that does it.

There wasn't a single thing in the FAQ that suggested it, and a Google search against the other site didn't pop something up that dedicated what was being done; just a small note that some proxies were being blocked. Had the stock FAQ file, or documentation, or anything detailed this behaviour, while I might still have thought it wrong, at least I wouldn't have gotten upset about it. I knew that there was proxy scanning code in slashcode, but all the vars in the database were set to off; as I discovered, they're ignored leading me to write a master off switch in the underlying scanning function.

Perhaps in total, this isn't a big deal, but it felt like a slap in the face. I know I have a temper, and I've been working to keep it under wraps (something easier said than done, but nothing worthwhile is ever easy). CmdrTaco himself commented on this on hackernews and I've written a reply to him about it. Slashdot did what they felt was necessary to stop spam on their site, and by 2008, slashcode only really existed for slashdot itself; other slash sites run on their own branches of older code. Right or wrong, such behaviour should be clearly documented, as its not something you expect, and can (and has) caused issues to users and concerns due to lack of communication. Transparency isn't easy, but I have found its the only way to have a truly healthy community. Perhaps you disagree. I'll respond to any comments or criticisms left below.

Just before, while moderating, I got this enormous error message:

Internal SAN check blew up checking moderation rights. Please let the powers that be know how and where you recieved this error Internal SAN check blew up checking moderation rights. Please let the powers that be know how and where you recieved this error Internal SAN check blew up checking moderation rights. Please let the powers that be know how and where you recieved this error Internal SAN check blew up checking moderation rights. Please let the powers that be know how and where you recieved this error Internal SAN check blew up checking moderation rights. Please let the powers that be know how and where you recieved this error Internal SAN check blew up checking moderation rights. Please let the powers that be know how and where you recieved this error Internal SAN check blew up checking moderation rights. Please let the powers that be know how and where you recieved this error Internal SAN check blew up checking moderation rights. Please let the powers that be know how and where you recieved this error

I have no idea where to properly report this, though. Thus I just put it here and hope that either the right person finds it, or someone who sees it can tell me where to report.

FWIW, apart from that message, everything seemed to be fine (including moderation, reduction of mod points, and no longer being able to moderate after using up my mod points).

We're testing a new configuration between the site and the database. There may be unexpected issues with the site while we're testing. Keep calm and carry on.

So I'm pretty sure you're all aware, but I've gone through and done a massive amount of work on the backend and infrastructure in the name of sanity, proper user permissions and such, and documenting as much as I can.

As a note, a lot of this was brought on by the fact we have relatively credible threat against the site, so I wanted to go through and make sure everything was in good shape and hardened (there's a lot of good bits here). I might have gone overboard. Here's the cliff notes version of what was done.

  * Static Status Page

http://status.soylentnews.org

This is on boron in /var/www, we should probably move it to Oxygen in case the entire linode DC goes down, but its fine there for now

  * Through documentation on node access, SSH, etc.

Basically, the links here http://wiki.soylentnews.org/wiki/SystemAdministration are required reading for all staff who play with dev, or production.

There are still gaps, varnish, slash, and apache only have limited documentation which is outdated, but I'll try and get those written in the next few days

  * Node renaming

This one might seem silly, but its sometimes hard to know what we're refering to when we talk about webserver/etc and a specific node. While at the moment we have no redundancy, I changed the hostnames of everything. The original soylent-* names are aliased in the internal DNS. List is here:

http://wiki.soylentnews.org/wiki/SystemAdministration/TheHitchhikersGuideToTheli694-22Domain

  * Internal DNS

Major thanks to xlefay for getting this up and running. All nodes exist in an internal li694-22 TLD, and are both forward and reverse resolvable (needed to make kerberos work properly, and make life easier).

  * Dev server

Announced, but falls into stuff done this weekend :-).

  * Varnish

I drastically reworked the varnish configuration file for better performance. The server is considerably more responsive than it used to with apache hit considerably less. As a side effect, slash hitcounts will be skewed as ACs will not be counted.

Rate limiting to prevent DOS was implemented, and xlefay pounded the dev server with some impressive apachebench numbers to confirm we won't go down. The dev server is much more loaded than production due to sharing the database, so I'm optimistic it will take a serious effort to pound us into oblivion with just ab or similar tools from a few nodes.

  * Disabled static page generation

This has been a PITA and on the TODO for awhile. Dynamically generated pages are now used for articles and comments. Varnish caches for ACs on a 5 minute basis. Logged in users get access to the site directly

  * SSL on Production

Doesn't fully work, but I reworked the nginx termination, and the varnish configuration so it is possible to login and use SSL. slash redirects the login to http, but the cookie gets properly set now so if you login SSL then reload the SSL page, it works. Need someone to dig into slash and figure out why ConnectionIsSSL is returning false. Need a volunteer to setup nginx termination on dev to debug.

  * LDAP setup

God, this was a pain, but we have a full LDAP setup on helium now. Replication to boron is on the TODO list, so if helium goes down, SSH authethication goes down, which is a bad thing. People with linode accounts can access the console and log in as root directly

Documentation (with pictures!) here: http://wiki.soylentnews.org/wiki/SystemAdministration/LDAPManagementForDummies

  * Passwords logged and recorded

Went through, made sure every password is saved in a master PW file which is in helium in root's home directory. sysops should keep a local copy of this file as its needed to use lish to access boxes should LDAP be down. Other important passwords like mysql, LDAP, and kerberos are also in this file.

  * Centralized ACLs

All machines require that a user be in the correct POSIX group to access them. List of groups is available here. This ensures that also everyone who has access can have it

http://wiki.soylentnews.org/wiki/SystemAdministration/GroupPermissions

  * SSH Policy

This one probably going to cause me some flack, but you need to go through the staff box (boron) to access any more. I don't like having open SSH ports on any of our nodes because it feels like we have our balls in the wind and a misconfiguration can leave us vulnerable.

I'm not kidding on that last bit. On production for the last month, slash:slash has worked as a username and password to log into the slash account. Using LDAP doesn't solve this as we still have local accounts for things LIKE slash.

Everyone must use SSH public key to autheticate; keys are stored in LDAP and are pulled on the fly by OpenSSH (this required updating OpenSSH on all nodes with a backport).

I know that due to slashd seizing up at a bad time this caused people to get locked out as I haven't gotten SSH keys from most people. I've got 8 users now with keys in LDAP. Right now, I don't have all the sudo files fully massaged, so if you have access to the dev server, you also have full sudo on all nodes. This isn't really desirable as I believe in limiting permissions, but this is a case of preventing us from going SNAP. Looking for someone to work out the necessary sudo voodoo

Also need someone to write upstart files for apache 1.3 so it comes back on a restart (xlefay is doing this, but feel free to work with him)

  * New Node Bringups

lithium (dev server), carbon (IRC server), and oxygen (offsite backup) were brought up this weekend. Bringup documentation was written here: http://wiki.soylentnews.org/wiki/SystemAdministration/TheRiseAndFallOfNewNodeManagement

  * OpenVPN

Setup a OpenVPN server on boron with magic iptables setup to allow oxygen to access all nodes. There's a fair bit of magic going on here, and I don't have the setup documented yet, but its basically following the Ubuntu Serer documentation for OpenVPN, plus a few iptable rules (saved in /etc/iptables.rules) on boron. Should be pretty self-explainatory.

  * Kerberos

To handle users that can't use ProxyCommand, to make life easier for internode stuff, and to be sexy, kerberos was setup to allow single signon. As most people probably never have managed Kerberos, the quick start guide is here: http://wiki.soylentnews.org/wiki/SystemAdministration/KerberosAdministration

Kerberos replication is setup, but not running as I need to make sure everything is sane. KDC master is helium, slave is boron.

  * AppArmored Apache

This was the real reason for the scheduled downtime last week as we had to migrate to apparmor capable kernels. AppArmor is basically SELinux but less braindead, and I handwrote a config that essentially puts Slash in a straightjacket. This should prevent things like process exploitation or a bug in slash from getting any traction. The apparmor config is installed on both lithium and hydrogen and is in /etc/apparmor.d. If you take a look, Apache can't take a piss without explicate permission :-).

(note, this doesn't do much to help us with SQL injections but every bit helps. Nothing short of a full rewrite of MySQL.pm to use stored procedures will fix this. Any takesr? (or migrating us to pgSQL then doing this?)

There's more to do here, slashd should be apparmored as well, but thats more difficult, and as its not directly user accessible, I'm less concerned that with apache itself. Ideally, every userfacing component should be apparmored (nginx, varnish, and slashd), but the former two run under very restrictive user accounts, and slashd only works with data in the database that already passed through Apache, and for the most part is just simple maintenance scripts, so its not that easy to attack.

I need to write up and document apparmor like I did for other things, but its relatively idiot proof to write files, and it makes good logs in /var/log/syslog.

  * Preparations for offsite backup

We've got a dedicated server (oxygen) with a 500 GIB HDD from http://www.kimsufi.com/en/ for €10 a month in France (oxygen) This will be used for offsite backups. xlefay looking and will be implementing this for all nodes.

  * Ubuntu package repo

As we need to maintain at least one backport, and need other things packaged, I setup a Launchpad PPA to do package building and binary distribution to all nodes: https://launchpad.net/~li69422-staff/+archive/backports-for-precise

This repo is added on all nodes. As you need to know how to do Debian packaging to use it, build an example package or two, and then I'll add you to the team. Its pretty straight forward on how to do this.

  * Staff userdir

Any staff can generate a userdir on boron by creating a public_html and using staff.soylentnews.org/~username

We've recently had a credible threat made against the site w.r.t. to a security exploit on slash. While details are somewhat vague, what little information was posted, combined with my knowledge of slash, and reviewing old security posts on slashcode.com suggests we're looking at a potential SQL injection technique.

While slash does considerable data sanitization, and escapes information coming in (so you can't just "; drop table users;"), there are values that require special sanitization on top of the normal. This has been the source of other XSS and SQL exploits against Slash historically (look at the articles on the main index).

I took a brief look at Environment.pm (where the supposed exploit is supposed to live) and said sanitization, and didn't seen anything that immediately jumped out at me, but it is a *lot* of code to look through. A grep through the access logs suggests that no one has tried to execute raw SQL on the site, but its impossible to know for sure.

If you or someone you know is interested in trying help secure SoylentNews, grab the dev VM, the current git tree, and get auditing. If we're informed of any security exploits (which should be sent by mail, and not posted in the comments), we will patch it here, and informal all other slashcode sites that we aware of about the exploit before releasing information about it publicly on the main index.

Do note that IP addresses, and passwords are hashed in the database (although only with MD5; upgrading to stronger hashing is on the TODO list), so an information leak, while bad, is not catastrophic. We do take regular snapshots of the database and of the machines themselves as well, and we'll make sure to post immediately if we become aware of any specific exploit against the site.