SoylentNews

After some of the comment exchanges this morning on topics such as Pollard and SCOTUS deciding money == speech, I think I'm going to swear off commenting on "political talk".

It does nothing to make the needed legal changes and does nothing to expand social discourse. Instead it becomes clear there is willful misunderstanding and a unwillingness to compromise by those who support the politics of selfishness and moral corruption. I don't have a problem with people if they want to live in hell and do despicable acts... that's their choice... but when they want to drag everybody else along that's a whole 'nother story.

It only makes me sad that people can be so gullible and believe in things which are not in their own best interest... but as I have learned from my mistakes I can only hope they learn before it is too late.

So go ahead and party boys while the country's morality burns to the ground, and the U.S. becomes just another corrupt rapacious Evil Empire... and our standing as a beacon to the rest of the world becomes the light of the on-coming train of our destruction... we all know how that story line ends.

No I'm not talking "religious morals"-- I'm talking humanist values and obligations to be kind to your family and community. To build and give rather than to destroy and take. I'm sorry but I do not find Libertarian or Tea Party philosophy has any moral value nor any grounds to stand on. It is selfish, destructive, short sighted, and anarchistic pack of twaddle which is being sold to the gullible as a real viable thing... except it ain't...

But I can't teach the willfully ignorant... and the only way they will learn is by letting themselves become an example of what not to do when trying to live a good life. Alas... some smart people can be really, really stupid.

So I'm swearing off political commentary... it serves me no good purpose and I'm sure those with whom I have disagreed will be grateful.

You're welcome... ;)

Creating a "movie" from a bunch of stills is pretty simple when you have ffmpeg handy. As before, code below is in chunks and the full script can be found here.

Since we're storing all a day's images in a particular directory, all we need to do is glob them up and pipe a concatenate to ffmpeg.

#!/usr/bin/perl -w

use strict;
use File::Glob ':glob';
use Date::Calc qw{ Today Add_Delta_Days };
use Data::Dumper;

use constant DEBUG => 1;
use constant BASE_DIR => '/home/fliptop/tv-ip551wi/';

We'll use File::Glob to gather all the image filenames into a list. We'll also need the Add_Delta_Days method to figure out yesterday's date so we know which files to glob (this code gets run sometime after midnight and operates on the previous day's pictures).

Please note that this script is designed to work on images from just one camera. It could easily be modified to handle multiple directories for more than one camera by setting the BASE_DIR constant to an array of directories then looping over them with the ffmpeg code below.

open LOG, '>>/tmp/ffmpeg_tv-ip551wi.log' or die "can't open log file: $!";

my @Yesterday = Add_Delta_Days(Today, -1);

# does the dir exist?
my ($year, $month, $day) = @Yesterday;
my $dir = sprintf "%s%s/%02d/%02d", BASE_DIR, @Yesterday;
my $movie_dir = sprintf "%s%s/%02d", BASE_DIR, $year, $month;

unless (-e $dir) {
  printf LOG "base directory %s does not exist!\n", $dir if DEBUG;
  die;
}

As always, open a log file for debugging purposes. After we figure out yesterday's date, make sure it exists and die if it doesn't.

my $files = sprintf "%s/*.jpg", $dir;
my @list = bsd_glob($files);

printf LOG "found %s images in directory %s\n", scalar(@list), $dir if DEBUG;
my $mpg = sprintf "%s/%s%02d%02d.mpg", $movie_dir, @Yesterday;

If the directory exists, we use the bsd_glob method to put all the filenames into the @list array. We'll give the final movie a name that corresponds to yesterday's date as well.

print LOG "creating mpg from images...\n" if DEBUG;
my $resp = qx{ /bin/cat $files |
               /usr/bin/ffmpeg -f image2pipe -sameq -vcodec mjpeg -i - -y $mpg };
print LOG "finished creating mpg\n" if DEBUG;

if ($resp) {
  printf LOG "unable to create mpeg: $resp\n" if DEBUG;
}

close LOG;

exit();

Here we use the cat command and pipe the output to ffmpeg, which will create a mjpeg movie. If you take one picture every 10 seconds over a 24-hour period, the final movie will be about 5 minutes long. See the ffmpeg man page for more information.

Coming next, invoking it all with cron.

I've been a little busy this week, too busy to spend much time soylenting. I've only written about three more paragraphs of Mars, Ho!; I've been working on Nobots and The Paxil Diaries. The Paxil Diaries was waiting on my porch when I got home from Patty's Tuesday evening, and boy was it a mess. I've mostly been working on it. It's funny how much easier it is for me to notice mistakes on paper I miss on screen.

I finished editing it again last night and am waiting for another copy, which they haven't shipped yet. When it comes I'll go over it again, upload the revisions and buy another copy. It may be green outside before you can get a copy after all.

Nobots needed more sales outlets, so I worked on that, too. You should be able to get it at bookstores in a few weeks. If you bought a copy last year, you may own a rare book. If my name is on the bottom right of the front cover instead of right under the title, you have one of fewer than two dozen copies. It should be worth something in a decade or so.

I may work on the Mars book today, but then again I might just take the day off, take the computer to Felber's and watch Cosmos on Hulu since channel 55 was off the air last night; their web site said there was equipment failure. And drink beer in the beer garden and listen to music and enjoy the 65 degrees they're forecasting.

Or maybe sweep the floor... nah.

We're testing a new configuration between the site and the database. There may be unexpected issues with the site while we're testing. Keep calm and carry on.

Now we're a few weeks in, and most stories when I come here are re-treads of things that I read on Slashdot a few days earlier. There's no point commenting on them, because I've already commented on the ones I'm interested in on Slashdot. Everyone else seems to feel the same way, because I rarely see a story with more than 10 comments. For a site that is meant to be all about the comments, that's an abject failure.

How could this have been solved? Well, as I proposed around launch time, the editors could have made a point of commenting on each story to prime the pump. When a story scrolls off the bottom of the front page with fewer comments than there are editors, then it's a failure. It means that either editors are posting stories that they're not interested in (in which case, why are they posting them?) or that they don't actually visit the site (in which case why are they editors?).

The only stories currently on the front page with more than 12 comments are 'people opting for dumb phones instead of smartphones' (which I'm just about to read - sounds like a typical rehash of the 'I have no self control so I'm going to use crappy technology to limit my exposure to stuff' story) and 'SCOTUS Signals Support for Corporate Religion?'. Where's the tech news? Are there any people here interested in discussing tech stories?

It feels like the staff gave up after the public temper tantrum between two of them and the community followed.

A Soylent News story posted last Saturday got me to thinking about the potential for a Markov chain based music "toy" application. Understand that at this moment I don't quite have the skill to write this app but it is an interesting thought experiment... so I'm writing it down here.

Yes I realize that you can do a lot of things with Max [$400!] but that is outside my budget [$0]... OTOH CSound is free but is a primarily sound design software suite rather than a composition application... tho' I have found some interesting things that I might investigate...

Most of the Pi based music I have seen / heard is based upon using the numbers 0 through 9 in the Pi sequence to correspond to a single note in a key. An example is the pi10k flash app.

The Markov Sliced Pi idea is to take a file with n number of digits of Pi and run a n-gram analysis where n=2. The result gives you the probability of two digit sequences of 00 to 99.

With this one can then do a number of interesting experimental musical things:

1) Using these digits as midi note numbers, and then picking a note at random, a note sequence of arbitrary length can be generated in a note rage of C0 to D#8 [greater than the range of a piano]. This will probably sound chaotic.

2) Add a "key Filter" which will drop notes that are not in the "key" of the starting note. For example if the starting not is C3= midi not number 36 then any note that does not fall in the Key of C [C D E F G A B] would be dropped from musical playback.

3) Add a "Shift filter" to shift discarded notes to be played in the relative minor or other key related shift [i.e. the idea is to force the output stream to sound "musical". While this might sound like twisting the input stream one might use pi or tau as a means to shift the note

3a) Modify the shift filter to work on notes which fall off of the circle of fifths for the key the song is played. this would prevent "non-harmonic notes"

4) Add additional midi note channels and assign octave ranges to specific instruments. Assuming that you had 4 instrument channels the first billion digits of Pi played at 1 second per note, the composition should last at least 3.96 years at 24 x 7 x 365.25. Generally normal performances are not this long. but once one had the software script written it would be the ultimate in elevator music as it would be an ever changing tune

5) Add a filter which forces the note stream into an 8 or 16 note Sequencer of notes. This causes "patterned musicality". This is done by taking the next 8 or 16 notes of a instrument and playing them until the next 8 or 16 notes are captured. The process could then be repeated until you run out of notes.

6) Add a Chord construction algorithm / filter / sequencer that builds chords much in the same way that the filter / sequencer operates. Take 3 notes, construct a chord, play it until the buffer files again. Variation between single notes and the chord may be based upon how far apart in the note sequence they are.

7) Run the Markov filter again and this time use n=4. this will give two note [or chord] sequences

7a) Run the Markov filter again and this time use n=6. this will give three note [or chord] sequences

8) use 7) and 7a) to build a Markov chain based Chord patterns using the circle of fifths.

Of course this is all a written out thought experiment on some of the things which might be done by crossing Markov Chains and Pi... since I am the lazy guy I am [and get easily distracted] it remains to be seen if I will get around to doing them.

I'd planned on traveling to Cincinnati last Monday to visit my daughter and came down with the flu. I called Patty and told her it would be the next Monday; she works full time and is a full time student at Cincinnati State, and Monday is the only day she has off.

I looked her address up on Google Maps. It looked pretty easy to find. "Don't trust Google," Patty said. "They're doing road construction and it will try to send you down a road that's closed. Take the Hoppit exit, turn right and I'll meet you at the Shell station.

My nose was still producing copious amounts of snot, I was still coughing up lots of mucus but felt a hell of a lot better than I had last week. I woke up about 5:30 Monday morning, did my morning routine functions, especially coffee, one function of which was checking my phone. Three missed calls and a voicemail from Patty. I called, knowing she wouldn't answer because she's never awake that early and left a message that I was on my way and to call when she woke up.

I have a big laptop bag and a small laptop; the bag had cost me $5 and came with a broken laptop. I put spare clothing, charging accessories in it and loaded it, my battery jumper, and Patty's cat's ashes in the car.

I had a half tank of gas and figured it would get me to Indiana, where fuel would surely be cheaper. After all, it's a red state and Republicans hate taxes, right? No such luck, I was down to an eighth of a tank by the time I reached Bloomington.

It's a little frustrating that Cincinnati is southeast of Springfield, but you have to go northeast to get there unless you want to drive over three hundred miles of two lane road with 30 to 45 MPH speed limits and lots of stop signs and so forth. It would take forever that way.

Gas was a nickle cheaper than Springfield; $3.55. I put twenty bucks in, figuring I'd fill up in Indiana and started on my way again. I had my phone plugged into the car stereo for times there was no music and I'd heard all the CDs, which I'd neglected to change before I left. There was a rest area so I stopped to urinate and change CDs. I checked the phone; Patty had called. I called back, and again she warned me about Google.

Apparently people from Illinois aren't welcome in Indiana, as the usual "Welcome to [state]" sign was nowhere in evidence. The only way I knew I'd crossed state lines was that the pavement got a lot worse. I-74 had apparently been badly neglected for years in Indiana, except for a stretch by Indianapolis. Gasoline was more expensive than at home.

The sun was shining, the pavement was dry, and there was little traffic. "Welcome to Ohio!" the big sign proudly proclaimed in bright graphics as the pavement improved. I reached Cincinnati and the traffic was terrible. I-74 East split into I-75 north and south; I guessed south but wasn't sure. I pulled over to the shoulder and called Patty to make sure I wasn't going the wrong way. I wasn't.

The next exit was the Hoppit exit. I met Patty at the gas station. "You shaved!" she said.

"Yeah, my upper lip hasn't seen the sun since before you were born." Patty had never seen me completely shaven; most of her life I've had a beard, or at least a mustache when my chin hair went gray.

"I don't like it," she said, frowning."

"Neither do I. I'm growing it back this fall." I noticed the gas cap door on her car was open as she pulled out and was about to honk to let her know when she pulled over and shut it.

We got to her apartment and we hugged and I shook her fiance's hand an gave Patty the metal box and envelopes. I hadn't opened one of them, which had come from Coble Animal Hospital. I'd thought it contained Princess' ashes but they called a week later to inform me I could pick her up.

"Ooh, this is a pretty box," she said. "What's in it?"

I still can't believe I spent over three hundred dollars for a dead cat, part for the vet to tell me she was dying and part to have her cremated, since the ground was frozen and I couldn't bury her. I discovered that animals and humans are cremated in the same crematorium, which is why it's so expensive. If Little One dies in the winter I'm storing her in a deep freeze until the ground thaws.

Patty opened the unopened envelope and started crying. It was a plastic placard that read "PRINCESS" and had her paw prints in it. No, I guess I didn't spend $300 on a dead cat, I spent it on my daughter. "Put this with Calie under the tree," she instructed. "When you move, take it and Calie's grave marker with you."

Colby had planned on making Reuben sandwiches for lunch but the corned beef was still frozen. "Let's go to Chick Filet," he said. "OK," I replied,"but then Patty needs a phone." Her iPhone had been broken for months, its screen cracked. And she'd liked my phone and especially liked my low phone bill.

We had chicken sandwiches and went to Best Buy. The price of the phone was half what I'd paid for mine. She was trying to decide between it and a more expensive one with a front facing camera but decided she liked the idea of it being waterproof and resistant to shock.

"Lets buy a TV while we're here" she said to Colby. After they talked for a while she said "well, I'm buying a TV. I have the money." They have an old twenty two inch tube TV that doesn't work and a little nineteen inch widescreen.

But she didn't like the prices so we went to H.H. Gregg, whose prices were no better than Best Buy's. Best Buy's crack Geek Squad couldn't activate Patty's new phone so we took it home and did it ourselves.

I'd bought Gravity, which had come from Amazon amazingly the day before it was supposedly released for sale. It was a "combo pack" with a DVD, Blu-Ray and download. I'd brought the Blu-Ray for Patty, and we watched it using her Playstation and little TV set.

None of us had seen the previous night's Cosmos so she fired up Hulu plus on the Playstation. After watching it and an episode of Doctor Who I decided that I wanted Hulu Plus.

The next morning she gave me a big bowl of corned beef, cabbage, carrots, and potatoes, and two T shirts. One was almost a joke; a St. Patrick's Day Reds shirt. The other was hawking some video game, a nerdy shirt I'll wear proudly.

She wanted to see how badly Google would have set me astray so I gave her my phone. She was amazed. "They got it perfect, that's how I told you to go." I loaded up the car, we said our goodbyes and I set off on the long journey home.

The trip home was as unpleasant as the trip there had been pleasant. First, I missed my turn to get on I-74. Five miles later I got on I-75, saw I was headed to Dayton and took the next exit. I stopped at a gas station, got gas, and consulted the map.

It would be nice of these things came with manuals. I think it ironic that everything used to have a detailed manual when technology was primitive enough you didn't need one, and now that interfaces have only icons and no way to discern WTF they mean, they don't. Let's see, looks like I go that way...

The radio was playing commercials so I switched it to the phone to listen to KSHE. The disk jockey started giving directions! "Go west on" whatever street the gas station was on "point seven miles and turn right." It wasn't KSHE, it was Google Maps. It easily got me back on I-74 north and it wouldn't shut up so I switched back to the radio.

Traffic was horrible; a semi that read "TARGET" zoomed past me doing at least twenty miles above the speed limit and almost made me miss my exit. Looks like it isn't just their IT that could use more training.

A little green sign with white lettering said "Welcome to Indiana". It started snowing. Twenty miles later visibility was poor, and twenty minutes after that the pavement was covered.

It was a miserable trip. The snow stopped around Indianapolis and the traffic was almost as bad as Cincinnati. Halfway to Illinois the wind started blowing. A couple of semis almost got blown off the highway.

Gas in Bloomington was $3.49.

When I got home there was a box on my doorstep; The Paxil Diaries had arrived. I'd screwed it up terribly. So you still can't have a copy yet...

The last part of my fable for I.T. workers has been posted on my blog.

So I'm pretty sure you're all aware, but I've gone through and done a massive amount of work on the backend and infrastructure in the name of sanity, proper user permissions and such, and documenting as much as I can.

As a note, a lot of this was brought on by the fact we have relatively credible threat against the site, so I wanted to go through and make sure everything was in good shape and hardened (there's a lot of good bits here). I might have gone overboard. Here's the cliff notes version of what was done.

  * Static Status Page

http://status.soylentnews.org

This is on boron in /var/www, we should probably move it to Oxygen in case the entire linode DC goes down, but its fine there for now

  * Through documentation on node access, SSH, etc.

Basically, the links here http://wiki.soylentnews.org/wiki/SystemAdministration are required reading for all staff who play with dev, or production.

There are still gaps, varnish, slash, and apache only have limited documentation which is outdated, but I'll try and get those written in the next few days

  * Node renaming

This one might seem silly, but its sometimes hard to know what we're refering to when we talk about webserver/etc and a specific node. While at the moment we have no redundancy, I changed the hostnames of everything. The original soylent-* names are aliased in the internal DNS. List is here:

http://wiki.soylentnews.org/wiki/SystemAdministration/TheHitchhikersGuideToTheli694-22Domain

  * Internal DNS

Major thanks to xlefay for getting this up and running. All nodes exist in an internal li694-22 TLD, and are both forward and reverse resolvable (needed to make kerberos work properly, and make life easier).

  * Dev server

Announced, but falls into stuff done this weekend :-).

  * Varnish

I drastically reworked the varnish configuration file for better performance. The server is considerably more responsive than it used to with apache hit considerably less. As a side effect, slash hitcounts will be skewed as ACs will not be counted.

Rate limiting to prevent DOS was implemented, and xlefay pounded the dev server with some impressive apachebench numbers to confirm we won't go down. The dev server is much more loaded than production due to sharing the database, so I'm optimistic it will take a serious effort to pound us into oblivion with just ab or similar tools from a few nodes.

  * Disabled static page generation

This has been a PITA and on the TODO for awhile. Dynamically generated pages are now used for articles and comments. Varnish caches for ACs on a 5 minute basis. Logged in users get access to the site directly

  * SSL on Production

Doesn't fully work, but I reworked the nginx termination, and the varnish configuration so it is possible to login and use SSL. slash redirects the login to http, but the cookie gets properly set now so if you login SSL then reload the SSL page, it works. Need someone to dig into slash and figure out why ConnectionIsSSL is returning false. Need a volunteer to setup nginx termination on dev to debug.

  * LDAP setup

God, this was a pain, but we have a full LDAP setup on helium now. Replication to boron is on the TODO list, so if helium goes down, SSH authethication goes down, which is a bad thing. People with linode accounts can access the console and log in as root directly

Documentation (with pictures!) here: http://wiki.soylentnews.org/wiki/SystemAdministration/LDAPManagementForDummies

  * Passwords logged and recorded

Went through, made sure every password is saved in a master PW file which is in helium in root's home directory. sysops should keep a local copy of this file as its needed to use lish to access boxes should LDAP be down. Other important passwords like mysql, LDAP, and kerberos are also in this file.

  * Centralized ACLs

All machines require that a user be in the correct POSIX group to access them. List of groups is available here. This ensures that also everyone who has access can have it

http://wiki.soylentnews.org/wiki/SystemAdministration/GroupPermissions

  * SSH Policy

This one probably going to cause me some flack, but you need to go through the staff box (boron) to access any more. I don't like having open SSH ports on any of our nodes because it feels like we have our balls in the wind and a misconfiguration can leave us vulnerable.

I'm not kidding on that last bit. On production for the last month, slash:slash has worked as a username and password to log into the slash account. Using LDAP doesn't solve this as we still have local accounts for things LIKE slash.

Everyone must use SSH public key to autheticate; keys are stored in LDAP and are pulled on the fly by OpenSSH (this required updating OpenSSH on all nodes with a backport).

I know that due to slashd seizing up at a bad time this caused people to get locked out as I haven't gotten SSH keys from most people. I've got 8 users now with keys in LDAP. Right now, I don't have all the sudo files fully massaged, so if you have access to the dev server, you also have full sudo on all nodes. This isn't really desirable as I believe in limiting permissions, but this is a case of preventing us from going SNAP. Looking for someone to work out the necessary sudo voodoo

Also need someone to write upstart files for apache 1.3 so it comes back on a restart (xlefay is doing this, but feel free to work with him)

  * New Node Bringups

lithium (dev server), carbon (IRC server), and oxygen (offsite backup) were brought up this weekend. Bringup documentation was written here: http://wiki.soylentnews.org/wiki/SystemAdministration/TheRiseAndFallOfNewNodeManagement

  * OpenVPN

Setup a OpenVPN server on boron with magic iptables setup to allow oxygen to access all nodes. There's a fair bit of magic going on here, and I don't have the setup documented yet, but its basically following the Ubuntu Serer documentation for OpenVPN, plus a few iptable rules (saved in /etc/iptables.rules) on boron. Should be pretty self-explainatory.

  * Kerberos

To handle users that can't use ProxyCommand, to make life easier for internode stuff, and to be sexy, kerberos was setup to allow single signon. As most people probably never have managed Kerberos, the quick start guide is here: http://wiki.soylentnews.org/wiki/SystemAdministration/KerberosAdministration

Kerberos replication is setup, but not running as I need to make sure everything is sane. KDC master is helium, slave is boron.

  * AppArmored Apache

This was the real reason for the scheduled downtime last week as we had to migrate to apparmor capable kernels. AppArmor is basically SELinux but less braindead, and I handwrote a config that essentially puts Slash in a straightjacket. This should prevent things like process exploitation or a bug in slash from getting any traction. The apparmor config is installed on both lithium and hydrogen and is in /etc/apparmor.d. If you take a look, Apache can't take a piss without explicate permission :-).

(note, this doesn't do much to help us with SQL injections but every bit helps. Nothing short of a full rewrite of MySQL.pm to use stored procedures will fix this. Any takesr? (or migrating us to pgSQL then doing this?)

There's more to do here, slashd should be apparmored as well, but thats more difficult, and as its not directly user accessible, I'm less concerned that with apache itself. Ideally, every userfacing component should be apparmored (nginx, varnish, and slashd), but the former two run under very restrictive user accounts, and slashd only works with data in the database that already passed through Apache, and for the most part is just simple maintenance scripts, so its not that easy to attack.

I need to write up and document apparmor like I did for other things, but its relatively idiot proof to write files, and it makes good logs in /var/log/syslog.

  * Preparations for offsite backup

We've got a dedicated server (oxygen) with a 500 GIB HDD from http://www.kimsufi.com/en/ for €10 a month in France (oxygen) This will be used for offsite backups. xlefay looking and will be implementing this for all nodes.

  * Ubuntu package repo

As we need to maintain at least one backport, and need other things packaged, I setup a Launchpad PPA to do package building and binary distribution to all nodes: https://launchpad.net/~li69422-staff/+archive/backports-for-precise

This repo is added on all nodes. As you need to know how to do Debian packaging to use it, build an example package or two, and then I'll add you to the team. Its pretty straight forward on how to do this.

  * Staff userdir

Any staff can generate a userdir on boron by creating a public_html and using staff.soylentnews.org/~username

If you're the owner of a copy of Nobots, you now own a rare book. Fewer than two dozen were printed. If you don't yet have a copy, the price is a little higher.

When I originally published I was brand-new to all of this. I guess I still am. Until now the only place it was for sale was Lulu; I hadn't properly registered its ISBN and the bar code on the cover was wrong (Lulu put it there).

When I was readying The Paxil DiariesI got better at navigating Lulu's interface and figured out how to add one of my ISBNs and get it for sale at Amazon, B&N, etc., and get it listed on Google Book Search. I fixed the front cover, too. It now looks like it does on my web site.

Those fewer than two dozen copies will be worth quite a bit in a few years. I worked with a fellow named (iirc) Dave Luttrell a couple of decades ago when computers were expensive. His sister won the lottery and fulfilled his dream of writing a book about his time in the Vietnam jungles. She bought him a computer for him to write it on, and a small local publishing house published it.

There was only a single printing, I don't know how big the print run was, but the local library had a copy. Interesting book, could have been better edited.

Years after I'd last seen Dave, Amy was telling me about her late uncle who had written a book about Vietnam and I realized that Dave was Amy's uncle. She was wishing she had a copy of his book and tried to find one.

The Elf Shelf, a used bookstore here, had a waterlogged copy for $250. So hang on to those books!

No sooner than I'd ordered a galley proof of The Paxil Diaries when I found a huge blunder -- a lot of chapter numbers were wrong and there were no page numbers. That's now fixed, and barring any further stupidity on my part you should be able to get a copy in a few weeks at the latest -- they shipped the galley proof three days ago.