SoylentNews

i was inspired to work on this after i saw mention of piping irc to the wiki @ http://wiki.soylentnews.org/wiki/CommunitySupport#Projects

it's been tested some but is still a work in progress.
getting around the anti-spam/anti-bot features of wiki is something i'll have to consult a wizard on.

https://github.com/crutchy-/test/blob/master/bot.php

i'm not a professional programmer so it probably sucks.
any criticisms etc are welcome, and if i can be bothered i may even take them on board, or you can do a pull request if you feel like having a play.

this is my first open source code file :-)

So I'm pretty sure you're all aware, but I've gone through and done a massive amount of work on the backend and infrastructure in the name of sanity, proper user permissions and such, and documenting as much as I can.

As a note, a lot of this was brought on by the fact we have relatively credible threat against the site, so I wanted to go through and make sure everything was in good shape and hardened (there's a lot of good bits here). I might have gone overboard. Here's the cliff notes version of what was done.

  * Static Status Page

http://status.soylentnews.org

This is on boron in /var/www, we should probably move it to Oxygen in case the entire linode DC goes down, but its fine there for now

  * Through documentation on node access, SSH, etc.

Basically, the links here http://wiki.soylentnews.org/wiki/SystemAdministration are required reading for all staff who play with dev, or production.

There are still gaps, varnish, slash, and apache only have limited documentation which is outdated, but I'll try and get those written in the next few days

  * Node renaming

This one might seem silly, but its sometimes hard to know what we're refering to when we talk about webserver/etc and a specific node. While at the moment we have no redundancy, I changed the hostnames of everything. The original soylent-* names are aliased in the internal DNS. List is here:

http://wiki.soylentnews.org/wiki/SystemAdministration/TheHitchhikersGuideToTheli694-22Domain

  * Internal DNS

Major thanks to xlefay for getting this up and running. All nodes exist in an internal li694-22 TLD, and are both forward and reverse resolvable (needed to make kerberos work properly, and make life easier).

  * Dev server

Announced, but falls into stuff done this weekend :-).

  * Varnish

I drastically reworked the varnish configuration file for better performance. The server is considerably more responsive than it used to with apache hit considerably less. As a side effect, slash hitcounts will be skewed as ACs will not be counted.

Rate limiting to prevent DOS was implemented, and xlefay pounded the dev server with some impressive apachebench numbers to confirm we won't go down. The dev server is much more loaded than production due to sharing the database, so I'm optimistic it will take a serious effort to pound us into oblivion with just ab or similar tools from a few nodes.

  * Disabled static page generation

This has been a PITA and on the TODO for awhile. Dynamically generated pages are now used for articles and comments. Varnish caches for ACs on a 5 minute basis. Logged in users get access to the site directly

  * SSL on Production

Doesn't fully work, but I reworked the nginx termination, and the varnish configuration so it is possible to login and use SSL. slash redirects the login to http, but the cookie gets properly set now so if you login SSL then reload the SSL page, it works. Need someone to dig into slash and figure out why ConnectionIsSSL is returning false. Need a volunteer to setup nginx termination on dev to debug.

  * LDAP setup

God, this was a pain, but we have a full LDAP setup on helium now. Replication to boron is on the TODO list, so if helium goes down, SSH authethication goes down, which is a bad thing. People with linode accounts can access the console and log in as root directly

Documentation (with pictures!) here: http://wiki.soylentnews.org/wiki/SystemAdministration/LDAPManagementForDummies

  * Passwords logged and recorded

Went through, made sure every password is saved in a master PW file which is in helium in root's home directory. sysops should keep a local copy of this file as its needed to use lish to access boxes should LDAP be down. Other important passwords like mysql, LDAP, and kerberos are also in this file.

  * Centralized ACLs

All machines require that a user be in the correct POSIX group to access them. List of groups is available here. This ensures that also everyone who has access can have it

http://wiki.soylentnews.org/wiki/SystemAdministration/GroupPermissions

  * SSH Policy

This one probably going to cause me some flack, but you need to go through the staff box (boron) to access any more. I don't like having open SSH ports on any of our nodes because it feels like we have our balls in the wind and a misconfiguration can leave us vulnerable.

I'm not kidding on that last bit. On production for the last month, slash:slash has worked as a username and password to log into the slash account. Using LDAP doesn't solve this as we still have local accounts for things LIKE slash.

Everyone must use SSH public key to autheticate; keys are stored in LDAP and are pulled on the fly by OpenSSH (this required updating OpenSSH on all nodes with a backport).

I know that due to slashd seizing up at a bad time this caused people to get locked out as I haven't gotten SSH keys from most people. I've got 8 users now with keys in LDAP. Right now, I don't have all the sudo files fully massaged, so if you have access to the dev server, you also have full sudo on all nodes. This isn't really desirable as I believe in limiting permissions, but this is a case of preventing us from going SNAP. Looking for someone to work out the necessary sudo voodoo

Also need someone to write upstart files for apache 1.3 so it comes back on a restart (xlefay is doing this, but feel free to work with him)

  * New Node Bringups

lithium (dev server), carbon (IRC server), and oxygen (offsite backup) were brought up this weekend. Bringup documentation was written here: http://wiki.soylentnews.org/wiki/SystemAdministration/TheRiseAndFallOfNewNodeManagement

  * OpenVPN

Setup a OpenVPN server on boron with magic iptables setup to allow oxygen to access all nodes. There's a fair bit of magic going on here, and I don't have the setup documented yet, but its basically following the Ubuntu Serer documentation for OpenVPN, plus a few iptable rules (saved in /etc/iptables.rules) on boron. Should be pretty self-explainatory.

  * Kerberos

To handle users that can't use ProxyCommand, to make life easier for internode stuff, and to be sexy, kerberos was setup to allow single signon. As most people probably never have managed Kerberos, the quick start guide is here: http://wiki.soylentnews.org/wiki/SystemAdministration/KerberosAdministration

Kerberos replication is setup, but not running as I need to make sure everything is sane. KDC master is helium, slave is boron.

  * AppArmored Apache

This was the real reason for the scheduled downtime last week as we had to migrate to apparmor capable kernels. AppArmor is basically SELinux but less braindead, and I handwrote a config that essentially puts Slash in a straightjacket. This should prevent things like process exploitation or a bug in slash from getting any traction. The apparmor config is installed on both lithium and hydrogen and is in /etc/apparmor.d. If you take a look, Apache can't take a piss without explicate permission :-).

(note, this doesn't do much to help us with SQL injections but every bit helps. Nothing short of a full rewrite of MySQL.pm to use stored procedures will fix this. Any takesr? (or migrating us to pgSQL then doing this?)

There's more to do here, slashd should be apparmored as well, but thats more difficult, and as its not directly user accessible, I'm less concerned that with apache itself. Ideally, every userfacing component should be apparmored (nginx, varnish, and slashd), but the former two run under very restrictive user accounts, and slashd only works with data in the database that already passed through Apache, and for the most part is just simple maintenance scripts, so its not that easy to attack.

I need to write up and document apparmor like I did for other things, but its relatively idiot proof to write files, and it makes good logs in /var/log/syslog.

  * Preparations for offsite backup

We've got a dedicated server (oxygen) with a 500 GIB HDD from http://www.kimsufi.com/en/ for €10 a month in France (oxygen) This will be used for offsite backups. xlefay looking and will be implementing this for all nodes.

  * Ubuntu package repo

As we need to maintain at least one backport, and need other things packaged, I setup a Launchpad PPA to do package building and binary distribution to all nodes: https://launchpad.net/~li69422-staff/+archive/backports-for-precise

This repo is added on all nodes. As you need to know how to do Debian packaging to use it, build an example package or two, and then I'll add you to the team. Its pretty straight forward on how to do this.

  * Staff userdir

Any staff can generate a userdir on boron by creating a public_html and using staff.soylentnews.org/~username

some some notes & snippets from fun with the chat bots in IRC.
times are australian eastern daylight saving time.

[22:25] <@aqu4> crutchy: s/tim/blaat/
[22:27] <crutchy> $sr /i/u/s
[22:27] <@aqu4> s/u/i/
[22:27] <SedBot> <aqu4> /taalb/mit/s :yhctirc

[22:31] <NCommander> O_o;
[22:34] <crutchy> $sr /O_o/o_O/s :rednammoCN
[22:34] <@aqu4> NCommander: s/O_o/o_O/
[22:34] <SedBot> <aqu4> <NCommander> o_O;

[22:39] <crutchy> $sr /O_o/o_O/s :rednammoCN ## yas sb/
[22:39] <@aqu4> /bs say ## NCommander: s/O_o/o_O/

$sr ++nocab
/bs say ## $sr ++nocab
/bs say ## bacon++

yet to try (bender+aqu4+sedbot?):
xyz say first: bacon++
/bs say ## $sr /--/++/s :zxy

We've recently had a credible threat made against the site w.r.t. to a security exploit on slash. While details are somewhat vague, what little information was posted, combined with my knowledge of slash, and reviewing old security posts on slashcode.com suggests we're looking at a potential SQL injection technique.

While slash does considerable data sanitization, and escapes information coming in (so you can't just "; drop table users;"), there are values that require special sanitization on top of the normal. This has been the source of other XSS and SQL exploits against Slash historically (look at the articles on the main index).

I took a brief look at Environment.pm (where the supposed exploit is supposed to live) and said sanitization, and didn't seen anything that immediately jumped out at me, but it is a *lot* of code to look through. A grep through the access logs suggests that no one has tried to execute raw SQL on the site, but its impossible to know for sure.

If you or someone you know is interested in trying help secure SoylentNews, grab the dev VM, the current git tree, and get auditing. If we're informed of any security exploits (which should be sent by mail, and not posted in the comments), we will patch it here, and informal all other slashcode sites that we aware of about the exploit before releasing information about it publicly on the main index.

Do note that IP addresses, and passwords are hashed in the database (although only with MD5; upgrading to stronger hashing is on the TODO list), so an information leak, while bad, is not catastrophic. We do take regular snapshots of the database and of the machines themselves as well, and we'll make sure to post immediately if we become aware of any specific exploit against the site.

Currently trying to learn how to use git and github.
http://wiki.soylentnews.org/wiki/index.php/User:Crutchy#Git.2FGitHub

to feed the trolls!

Occasionally, I enjoy the hell out of it.

An idea that came to mind during talk about Soylent hosting on IRC today.
Thanks to Titanium, prospectacle, stderr, useless, swiss, FoobarBazbot and MrBluze for a lively discussion :-)

It started with:
[19:43] * crutchy wonders if a distributed service could be developed... divide the load, build in redundancy and if anyone's host goes down others will pick up the slack
Time is AEDT

The idea has been developed a little bit further since the IRC discussion.

General System
==============
- independent of DNS
- with a distributed model anyone can volunteer to host a node (no single person relied on to front hosting costs)
- system consists of a network of apache host nodes set up by volunteers willing to cover the costs of their host, and users connect to the web service using their web browser with the remote host selected by a launcher program

Host Node
=========
- not required to access web service (only for those who choose to offer to host)
- apache web server configured for web service (mysql, mod_perl, etc as required)
- must periodically execute a script (using crontab?) that requests nodelist from listed nodes and updates local nodelist as required (adds/removes based on some kind of agreement algorithm)
- must respond to nodelist requests from launchers, but can be isolated php/pl/etc script and need not be built into hosted service
- must contain scripts to synchronize data and site source code updates securely with other nodes as required (this will be the tricky bit)

Launcher
========
- user executes launcher to access web service
- no gui
- executable or source downloaded from trusted location (such as debian repository or github) along with nodelist containing one or more known host IP addresses
- purpose is to select a remote host node and open web browser pointed to selected remote host IP address
- before opening browser, a nodelist request is sent to every host in local nodelist, and local nodelist is updated in same way as server nodelist is updated (see above)
- possibly a simple settings file if required
- could be a bash script for Linux and a small Delphi or C program for Windows

On IRC there was concern expressed about security and verification of host nodes.
Since you're using a browser as a client (and all the security features that come with) and you're only receiving normal http responses (otherwise your browser would throw an error), there's only so much bad stuff a host node can do.
Worst case scenario might be that it redirects to goat.cx or some site with driveby downloads (which most browsers will block anyway).
If required a trusted network of host nodes could be formed using signed certificates (perhaps using OpenSSL).

Nodelists may not be that big since there isn't likely to be a huge number of hosts for the same service (such as SoylentNews) but if need be the list could be gzipped. As mentioned earlier, the tricky bit will be synchronizing website data and service application source code, but I don't think it is an insurmountable challenge.

edit: data could be distributed, but would need to be synchronized on all host nodes (the tricky bit mentioned above)

edit: thinking about data synchronizing... would either require modification of the service application to execute a script when data is changed (and script would do the work of sending data to other hosts) or a shell script with a loop that checks for changes to data file timestamps and if change is detected send data files to other hosts. for max efficiency it would be ideal just to post a single mysql insert/modify query whenever data changes, but that would require integration into the main application (slashcode in Soylent's case). you don't want to be sending entire database files around the place whenever there is a change. a good place to start might be to host the data on one or two high performance 'supernodes' until an improved synch system can be developed.

As many might know SoylentNews resides on http://li694-22.members.linode.com/¹ and because of this some people were talking and joking about using li694-22 as a new name. It's a cool name, I was tempted myself! Perhaps an even "weirder" inside joke than http//:/..org :)

No need to be tempted any more; a Mr. Watt (not me!) of Washington bought it and pointed it at SoylentNews¹ :)

¹ naturally your cookies are in different jars

Edit: just to practice safe surfing don't log in through redirection or move your cookies manually or anything like that. Not that I would think anything bad would happen in this case but one would never know until it was too late (maybe Mr. Watt suddenly develops an appetite for collecting low UID accounts).

Since I've moderated and can't be bothered to log out I'll write some thoughts here for my own interest. By no means is this meant to be any kind of complete answer or anything of the sort, just some idle thoughts/speculation.

0.a. It is an entirely unknown failure mode that is sudden and immediately cripples everything. Very unlikely.
0.b. It is an entirely unknown phenomenon that is sudden and immediately cripples everything. Extremely unlikely but not zero.
0.c. A confluence of simultaneous and lasting shoddy operation and systems malfunction in two culturally different countries (Malaysia and Viet Nam). This one is hard to judge; I wouldn't think so on behalf of Viet Nam but they hadn't yet taken airspace control/responsibility for the plane and might not have paid much if any attention to it. Malaysia is fully able to fuck anything up beyond rational belief (*cough* bigoted apartheid-style legislation on the use of a word *cough*) but even so Viet Nam should still have the radar records and be fully able to find anything if there in fact was a more normal disaster.

I guess the simplest ad hoc would be 0.b. with some kind of unusual simultaneous failure of radar range for whatever reason: the signal would then simply disappear giving no clues about anything. If this was caused by some freak meteorological event local to the aircraft it might explain the total lack of everything except debris which might be found later. It might not have to last all that long if the electronics in the plane are knocked out before any remaining related blips on now-functioning radars disappear among the noise. Still extremely unlikely. Inverted clear sky sprite plasma bolts (no such thing is known to exist) or time-space warp bubbles (sorry no link to the paper handy and no such thing is known to exist) or alians!!1 (etc.) or whatever, but who knows.

1. Whether or not some terrorist organization claims responsibility doesn't mean much. Some YKW (You Know Who) organizations claim just about everything or are created solely to claim credit for anything new (like happened for the attacks in Oslo before those claims were discredited) and all it takes for the opposite to happen are a few things:
1.a.1. Whoever did it has discovered and understood the meaning of tactics, and the incident while public in nature is also long term in nature (there are several possibilities here, I'm not comfortable with spelling it out). Somewhat likely.
1.a.2. Whoever responsible simply (and without any deeper thought) doesn't want to draw attention to something that is still ongoing. Fairly likely.

and

1.b.1. For whatever reason(s) the incident fails to trigger knee-jerk claims. Doing something to a flight from a YKW nation to China should naturally avoid most if not all such attention because China is kind of outside the horizon of most YKW despite the recent YKW attacks both in Beijing and western China. Not too unlikely.
1.b.2. Someone figured it was stupid and counterproductive to make bullshit claims and has the clout to stop those who still don't get it. Very unlikely but not impossible.

For a 1.a.2. that passes 1.b.1 it seems very likely that some YKW "Chinese" did this to simply kill as many Chinese as possible. Such YKW "Chinese" aren't known to be big on making public statements of responsibility, in fact they seldom say anything at all (probably it makes them very easy to catch and kill) so that fits.

Oil slicks don't mean much on their own but are often the first thing spotted. If nothing else is spotted (lots of debris floats for a fairly long time) then 0.b. increases.

Sometimes there isn't an answer.