SoylentNews

The last part of my fable for I.T. workers has been posted on my blog.

So I'm pretty sure you're all aware, but I've gone through and done a massive amount of work on the backend and infrastructure in the name of sanity, proper user permissions and such, and documenting as much as I can.

As a note, a lot of this was brought on by the fact we have relatively credible threat against the site, so I wanted to go through and make sure everything was in good shape and hardened (there's a lot of good bits here). I might have gone overboard. Here's the cliff notes version of what was done.

  * Static Status Page

http://status.soylentnews.org

This is on boron in /var/www, we should probably move it to Oxygen in case the entire linode DC goes down, but its fine there for now

  * Through documentation on node access, SSH, etc.

Basically, the links here http://wiki.soylentnews.org/wiki/SystemAdministration are required reading for all staff who play with dev, or production.

There are still gaps, varnish, slash, and apache only have limited documentation which is outdated, but I'll try and get those written in the next few days

  * Node renaming

This one might seem silly, but its sometimes hard to know what we're refering to when we talk about webserver/etc and a specific node. While at the moment we have no redundancy, I changed the hostnames of everything. The original soylent-* names are aliased in the internal DNS. List is here:

http://wiki.soylentnews.org/wiki/SystemAdministration/TheHitchhikersGuideToTheli694-22Domain

  * Internal DNS

Major thanks to xlefay for getting this up and running. All nodes exist in an internal li694-22 TLD, and are both forward and reverse resolvable (needed to make kerberos work properly, and make life easier).

  * Dev server

Announced, but falls into stuff done this weekend :-).

  * Varnish

I drastically reworked the varnish configuration file for better performance. The server is considerably more responsive than it used to with apache hit considerably less. As a side effect, slash hitcounts will be skewed as ACs will not be counted.

Rate limiting to prevent DOS was implemented, and xlefay pounded the dev server with some impressive apachebench numbers to confirm we won't go down. The dev server is much more loaded than production due to sharing the database, so I'm optimistic it will take a serious effort to pound us into oblivion with just ab or similar tools from a few nodes.

  * Disabled static page generation

This has been a PITA and on the TODO for awhile. Dynamically generated pages are now used for articles and comments. Varnish caches for ACs on a 5 minute basis. Logged in users get access to the site directly

  * SSL on Production

Doesn't fully work, but I reworked the nginx termination, and the varnish configuration so it is possible to login and use SSL. slash redirects the login to http, but the cookie gets properly set now so if you login SSL then reload the SSL page, it works. Need someone to dig into slash and figure out why ConnectionIsSSL is returning false. Need a volunteer to setup nginx termination on dev to debug.

  * LDAP setup

God, this was a pain, but we have a full LDAP setup on helium now. Replication to boron is on the TODO list, so if helium goes down, SSH authethication goes down, which is a bad thing. People with linode accounts can access the console and log in as root directly

Documentation (with pictures!) here: http://wiki.soylentnews.org/wiki/SystemAdministration/LDAPManagementForDummies

  * Passwords logged and recorded

Went through, made sure every password is saved in a master PW file which is in helium in root's home directory. sysops should keep a local copy of this file as its needed to use lish to access boxes should LDAP be down. Other important passwords like mysql, LDAP, and kerberos are also in this file.

  * Centralized ACLs

All machines require that a user be in the correct POSIX group to access them. List of groups is available here. This ensures that also everyone who has access can have it

http://wiki.soylentnews.org/wiki/SystemAdministration/GroupPermissions

  * SSH Policy

This one probably going to cause me some flack, but you need to go through the staff box (boron) to access any more. I don't like having open SSH ports on any of our nodes because it feels like we have our balls in the wind and a misconfiguration can leave us vulnerable.

I'm not kidding on that last bit. On production for the last month, slash:slash has worked as a username and password to log into the slash account. Using LDAP doesn't solve this as we still have local accounts for things LIKE slash.

Everyone must use SSH public key to autheticate; keys are stored in LDAP and are pulled on the fly by OpenSSH (this required updating OpenSSH on all nodes with a backport).

I know that due to slashd seizing up at a bad time this caused people to get locked out as I haven't gotten SSH keys from most people. I've got 8 users now with keys in LDAP. Right now, I don't have all the sudo files fully massaged, so if you have access to the dev server, you also have full sudo on all nodes. This isn't really desirable as I believe in limiting permissions, but this is a case of preventing us from going SNAP. Looking for someone to work out the necessary sudo voodoo

Also need someone to write upstart files for apache 1.3 so it comes back on a restart (xlefay is doing this, but feel free to work with him)

  * New Node Bringups

lithium (dev server), carbon (IRC server), and oxygen (offsite backup) were brought up this weekend. Bringup documentation was written here: http://wiki.soylentnews.org/wiki/SystemAdministration/TheRiseAndFallOfNewNodeManagement

  * OpenVPN

Setup a OpenVPN server on boron with magic iptables setup to allow oxygen to access all nodes. There's a fair bit of magic going on here, and I don't have the setup documented yet, but its basically following the Ubuntu Serer documentation for OpenVPN, plus a few iptable rules (saved in /etc/iptables.rules) on boron. Should be pretty self-explainatory.

  * Kerberos

To handle users that can't use ProxyCommand, to make life easier for internode stuff, and to be sexy, kerberos was setup to allow single signon. As most people probably never have managed Kerberos, the quick start guide is here: http://wiki.soylentnews.org/wiki/SystemAdministration/KerberosAdministration

Kerberos replication is setup, but not running as I need to make sure everything is sane. KDC master is helium, slave is boron.

  * AppArmored Apache

This was the real reason for the scheduled downtime last week as we had to migrate to apparmor capable kernels. AppArmor is basically SELinux but less braindead, and I handwrote a config that essentially puts Slash in a straightjacket. This should prevent things like process exploitation or a bug in slash from getting any traction. The apparmor config is installed on both lithium and hydrogen and is in /etc/apparmor.d. If you take a look, Apache can't take a piss without explicate permission :-).

(note, this doesn't do much to help us with SQL injections but every bit helps. Nothing short of a full rewrite of MySQL.pm to use stored procedures will fix this. Any takesr? (or migrating us to pgSQL then doing this?)

There's more to do here, slashd should be apparmored as well, but thats more difficult, and as its not directly user accessible, I'm less concerned that with apache itself. Ideally, every userfacing component should be apparmored (nginx, varnish, and slashd), but the former two run under very restrictive user accounts, and slashd only works with data in the database that already passed through Apache, and for the most part is just simple maintenance scripts, so its not that easy to attack.

I need to write up and document apparmor like I did for other things, but its relatively idiot proof to write files, and it makes good logs in /var/log/syslog.

  * Preparations for offsite backup

We've got a dedicated server (oxygen) with a 500 GIB HDD from http://www.kimsufi.com/en/ for €10 a month in France (oxygen) This will be used for offsite backups. xlefay looking and will be implementing this for all nodes.

  * Ubuntu package repo

As we need to maintain at least one backport, and need other things packaged, I setup a Launchpad PPA to do package building and binary distribution to all nodes: https://launchpad.net/~li69422-staff/+archive/backports-for-precise

This repo is added on all nodes. As you need to know how to do Debian packaging to use it, build an example package or two, and then I'll add you to the team. Its pretty straight forward on how to do this.

  * Staff userdir

Any staff can generate a userdir on boron by creating a public_html and using staff.soylentnews.org/~username

If you're the owner of a copy of Nobots, you now own a rare book. Fewer than two dozen were printed. If you don't yet have a copy, the price is a little higher.

When I originally published I was brand-new to all of this. I guess I still am. Until now the only place it was for sale was Lulu; I hadn't properly registered its ISBN and the bar code on the cover was wrong (Lulu put it there).

When I was readying The Paxil DiariesI got better at navigating Lulu's interface and figured out how to add one of my ISBNs and get it for sale at Amazon, B&N, etc., and get it listed on Google Book Search. I fixed the front cover, too. It now looks like it does on my web site.

Those fewer than two dozen copies will be worth quite a bit in a few years. I worked with a fellow named (iirc) Dave Luttrell a couple of decades ago when computers were expensive. His sister won the lottery and fulfilled his dream of writing a book about his time in the Vietnam jungles. She bought him a computer for him to write it on, and a small local publishing house published it.

There was only a single printing, I don't know how big the print run was, but the local library had a copy. Interesting book, could have been better edited.

Years after I'd last seen Dave, Amy was telling me about her late uncle who had written a book about Vietnam and I realized that Dave was Amy's uncle. She was wishing she had a copy of his book and tried to find one.

The Elf Shelf, a used bookstore here, had a waterlogged copy for $250. So hang on to those books!

No sooner than I'd ordered a galley proof of The Paxil Diaries when I found a huge blunder -- a lot of chapter numbers were wrong and there were no page numbers. That's now fixed, and barring any further stupidity on my part you should be able to get a copy in a few weeks at the latest -- they shipped the galley proof three days ago.

The third part of the I.T. fable, "A bedtime story for A.I.s. in training", is now available at my blog,

Chapter One
Previously

Alarm
        The alarm went off when we were watching a movie; a real one this time, a modern holo rather than the ancient two dimensional ones we'd been watching. So of course I thought "damned whores."
        "Sorry, hon, we have a fire in the commons. I'll be back when I can." Damned whores.
        When the yellow light flashes over most doors, they can only be opened from the outside. When it flashes red outside it won't let you in, when it flashes red on the inside you'd better get the hell out of there.
        There were a few exceptions, like my quarters. It would only keep me in if there was a vacuum or a fire outside the door. It only flashed yellow as a warning.
        I went to the commons and another alarm went off. What the hell? This one was in the passenger section, apartment 12. Nobody should be in there. Whores? More electric problems?
        The commons was closer and I had to make sure the cargo had evacuated.
        There were no whores and no fire. My tablet reported it was a scheduled drill. That explained number twelve, sometimes they simulated more than one fire.
        It went off again. "Cargo section, #6." I laughed, the computer was posing a conundrum for me. And the cargo. If your quarters caught fire you were supposed to go to the commons but what if it were on fire, too?
        Number six... that was the Thai girls, wasn't it?
        There was screaming from the other side of the door. "Computer, open the door" I ordered.
        "Unable to comply. Danger to ship, passengers, other cargo, and crew."
        "Report."
        "Fire in cargo hold six. Fire suppression technologies deployed."
        The damned thing talks like it's went to college.
        "Let those girls out, damn it!!"
        "Unable to com..."
        "GOD DAMN IT!!"
        And then another damned alarm went off. Son of a bitch! "Computer, source of new alarm."
        "Meteor shower ahead." The door opened and the girles stumbled out, along with the fat blonde, coughing. Smoke billowed from the door before it closed.
        "Meet me in the commons, I have an emergency." I ran to the pilot room on my sore legs.
        This time, like most times, meteors meant slow down. I reduced gravity to 10%. This time I wasn't going to face the whores until it was over, we were already behind schedule.
        After the rocks all passed in front of us I sped back up and adjusted course to make up for the damned rocks.
        I checked the passenger quarters and sure enough it was a drill. What morons program this shit, anyway? Having emergency drills when there's a real emergency? That's dangerous. Stupid dangerous. Those bozos might have went to college but they were morons. God damned idiots!
        What? Yeah, yeah, just shut up and let me talk, I want to get this over. Anyway, the three girls were still sitting on the medic outside their apartment sucking oxygen. The door light was red but no longer flashing.
        "So what happened?" I asked them.
        "Don't know," the blonde said. I can never remember her name. Anyway, she says "we were just talking when that damned noisy maid burst into flames and the room locked us in! We were scared shitless!"
        It happened sometimes, but they usually smoked for a while before they started burning, and then only when they were old and worn out. I hoped the ship had a robot that made robots.
        The light went out, the door opened, the Thai women went in and the blonde went home. So did I.
        Destiny had fallen asleep, so I got a beer and put the movie back to where I'd left off.

Continues...

Part 2 of "A bedtime story for A.I.s In Training" is now available at my writing blog Falling thru Reality. It's a five part fable concerning the ways an A.I. can fail to learn essential lessons as it endeavors to become an I.T. professional.

I've just posted part one of a five part "I.T. fable" entitled "A bedtime story for A.I.s. in training."

It is a rather tongue in cheek warning to those of you who "do" technology for a living... especially those working in toxic work environments.

You can find it on my blog Falling Thru Reality

Last night I cloned the git repository. Today I was perusing some static pages on the site and I noticed there's the term Slashdot in a few places, most notably on moderation page. But in the version I pulled down these have been changed to SN.

How long does it take to get changes merged down to the site's main trunk?

The iGuard IP250E requires cookies for authentication. Modifying Part 3's script is pretty simple. Below are just the additional code chunks. Whole script is available here.

use HTML::Form;
use HTTP::Cookies;

Add these to your includes.

my $ua = LWP::UserAgent->new;
$ua->cookie_jar(HTTP::Cookies->new(file => "/tmp/iguard-ip250e.txt", autosave => 1));
$ua->agent("Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.13) Gecko/2009080317 Fedora/3.0.13-1.fc10 Firefox/3.0.13");

Here we tell our useragent object where to store any cookies.

my $uri = 'http://192.168.1.243/image.cgi';
my $res = $ua->request(HTTP::Request->new(GET => $uri));
my $status = $res->status_line;
printf LOG "unable to process initial cookie request, status code is: %s", $status
  unless $status eq '200 OK' || $status eq '302 Found';

my ($form) = HTML::Form->parse($res);
print LOG "form: ", Dumper($form) if DEBUG;

$form->value('$login_un' => 'username');
$form->value('$login_pw' => 'password');

On the iGuard IP250E, there's a simple login form returned if you request image.cgi We parse the form and assign the username/password params.

$res = $ua->request($form->click);
$status = $res->status_line;
printf LOG "unable to submit username/password, status code is: %s", $status
  unless $status eq '200 OK' || $status eq '302 Found';

$uri = 'http://192.168.1.243/showimg_pda.cgi?cam=1';

my $im = Image::Magick->new();

Submit the form by issuing click. The URI showimg_pda.cgi?cam=1 is the way to return just the image from the iGuard. The rest of the code is the same.

Chapter One
Previously

Pressure
        When I woke up, all my muscles were on fire. We would have had to turn the ship around today, and in fact that's what was scheduled, except for the meteors and the drama that followed.
        Destiny was sleeping peacefully. I got up, thankful that we weren't at Earth gravity but wishing we had turned around for deceleration then, because they have it plotted so that you start the journey close to the planet you're leaving's gravity, and reach your destination close to that planet's gravity. We were at half Earth gravity now and it would gradually be lowering to Mars' gravity.
        The girls didn't like half Earth gravity, they were going to hate Mars. I guess these girls were being well paid or something, they sure were paying me good. Except that from what I'd learned about these women they probably just promised free drops. Drops were the addicts' only motivation, only goal, only thing that mattered to them.
        God but my muscles were all on fire. I sat down on the couch and had the robot make a cup of shitty coffee, my legs hurt. I had it bring me water and Naproxin and drank the lousy coffee. Yech. Why can't they program those damned things to make drinkable coffee? I should have went to college and learned programming.
        I only drank half of the nasty brew and hauled myself painfully to the shower. A hot shower would do wonders for my aching muscles.
        The hot water felt as good as the coffee had tasted bad. I took a really long one. It helped ease the pain, and the pill had started working some, too.
        I took one sip of the remaining cold, nasty coffee and started a pot. Damned stupid robots.
        I was just pouring a cup when Destiny came in. "John!" she said. "You look like hell!"
        "I feel like hell. All that damned climbing yesterday nearly killed me. And I still have to check the instruments and inspect the boat."
        "You did inspection yesterday. I thought inspections were weekly?"
        "Yeah, normally, but yesterday wasn't the least bit normal. I have to inspect that busted generator since it would have cooled enough by now, and the other one, too, since it's working harder now that there's only one."
        "Poor baby!"
        "Well, at least I don't have to inspect cargo today. Want to watch a movie later?"
        "Sure. Isn't it almost time to check your instrumentation?"
        "Yeah, it is." I kissed her. "See you in a while."
        I went towards the pilot room, which was really just outside my quarters. Yesterday I'd been wishing for a bicycle, today I was wishing for a cane.
        All the readouts were normal except one – air pressure in the port generator was twenty kilopascal low. That wasn't a good sign at all, I was going to need a suit and tether in case a bulkhead blew while I was in there.
        I noted the log and stopped by our cabin... heh, "our cabin," how about that? Anyway I stopped to fill a bug mug and summon a medic.
        Medics are robots that look kind of like narrow tables with padded tops and appendages to measure bodily functions and administer medicine. Planetside they called them "gurneys" but everything is named different on a boat. Like port and starboard.
        I sat on the medic and ordered it to the port generator and got another robot on the fone to fetch the suit from the starboard hold where Destiny had gone out the airlock.
        After I'd suited up and tethered, the difference in pressure made it hard to get the hatch open. I tried a crowbar and couldn't even make it hiss. So I lowered the pressure where I was and the door popped open by itself. I took a floater with me to hunt for the leak.
        A floater is just a small balloon filled with helium with a small counterweight to make it gravity neutral. It goes where the air goes.
        I found where the air was escaping and patched it. Why can't they program robots to do that? Stupid robots, they could act as maids and medical doctors and all sorts of other functions but the damned things can't patch a hole or make a decent cup of coffee. At least they're cheap.
        The pressure was slowly rising so I sat on the medic and waited until it matched the rest of the ship so I could get out of the room. I hadn't needed the suit, but left it on just to keep my ears from popping.
        The gauge said pressure was normal so I tried the hatch. It opened easy, so I took off the suit and gave it to a robot and rode the medic back to my rooms.
        I was dying of thirst, even after downing that big glass of water when I took the naproxin. I said something to Destiny about it when I got back, taking another pill and drinking more water.
        She laughed. "You're dehydrated, dummy. You told me yesterday you thought you were going to drown in your suit from sweating. You probably need electrolytes, too."
        "And I'm hungry, I just didn't feel like eating when I got up. You hungry?"
        "I could eat. Robot eggs okay or do you want me to cook?"
        "No, robots cook okay as long as it doesn't involve coffee. How do you want your eggs?"
        "Ham and cheese omelette is okay, maybe with some hash browns."
        "Okay. Robot, a ham and cheese omelet, a Denver omelette, two hash browns and toast. No coffee!"
        Them damn robots suck at coffee, and they can't patch a hole at all. I'm glad they can cook.

Continues