SoylentNews

Since we've got a fair number of complaints about us running too many site news articles, I'm going to condemn this to my journal, then link it next time we *do* post something about the site. For a large portion of today (4/16), SoylentNews users had issues with commenting, and moderation was completely hosed. This was due to a backend change; we shifted the site behind a loadbalancer in preparation of bringing up a new frontend and give us considerably more redundancy and latitude with working with the backend.

This change had been setup on dev for the last week with us testing it to see what (if anything) broken, and it was discussed and signed off by all of the staff. Last night, I flipped the nodebalancer to connect to production instead of dev, then changed the DNS A record for the site to point at the loadbalancer.

I stayed up for several hours at this point to ensure nothing odd was going on, and satisfied that the world would keep spinning, I went to bed. What I found though was I broke the formkeys system. Slash knows about the X-Forwarded-By header, a mechanism for when a site is behind a proxy on how to relay client IP information (this mechanism was already used by both varnish and nginx), however, for security reasons, we strip out the XFF header from inbound connections unless its on a specific whitelist. On both dev and production, we had whitelisted the nodebalancer to pass this header in properly.

Or so we thought. Linode's documentation doesn't mention, but the IP address listed in the admin interface is *not* the IP used to connect to the site; instead it uses a special internal IP address which isn't listed or documented anywhere. Our security precautions stripped out the X-Forwarded-By header, and made it appear that all inbound users were coming from the same IP. This wasn't noticed on dev as slash ignores the formkeys system for admins, and the few of us beating on it with non-admin accounts weren't able to do enough abuse to trigger the formkey limiters.

Our peak hours are generally evenings EDT, which means the low traffic at night wasn't enough to trip it either (or at least no one on IRC poked me about it, nor were there any bugs on it on our github page. However, once traffic started picking up, users began to clobber each other, commenting broke, and the site went to straight to hell. When I got up, debugging efforts were underway, but it took considerable time to understand the cause of the breakage; simply reverting LBing wasn't an easy fix since we'd still have to wait for DNS to propagate and we needed the load balancer anyway. After a eureka moment, we were able to locate the correct internal IPs, and whitelist them, which got the site partially functional again. (we have informed Linode about this, and they said our comments are on its way to the appropriate teams; hopefully no other site will ever have this same problem).

The last remaining item was SSL; we had originally opted out of terminating SSL on the loadbalancer, prefering to do it on the nginx instance, so Port 443 was set to TCP loadbalancing. This had the same effect as there is no way for us to see the inbound IP (I had assumed it would do something like NAT to make connections appear like they were coming from the same place). The fix was utlimately installing the SSL certificate on the load balancer, then modifying varnish to look for the X-Forwarded-Proto header to know if a connection was SSL or not. I'm not hugely happy about this as it means wiretapping would be possible between the load balancer and the node, but until we have a better system for handling SSL, there isn't a lot we can do about it.

As always, leave comments below, and I'll leave my two cents.

I've hardly logged on to the internet at all this past week, too busy correcting a mistake software houses frequently do: Trying to rush a project out the door. The fact is, I'm tired of The Paxil Diaries, but I don't want to ship a flawed piece of crap.

The first copy had a messed up cover; my printer's "cover generation wizard" has an interface almost as bad as GIMP. I fixed it and ordered a corrected copy, and a day later as I was converting the .odt to .html I discovered that some of the chapter numbers were wrong and there were no page numbers. I fixed it, resubmitted it and thought "This time it'll be right."

Number three showed up bright and early Thursday morning. I started going over it with a fine toothed comb. Almost halfway through and I started to think I'd be able to release it. The weather got really nice so I decided to read it in Felber's beer garden.

I discovered I was far better at proofreading when I've had a few beers than sober. When I'm sober what the words are saying distracts me from the words themselves, and I read too fast and miss errors.

It was full of errors, many of them whoppers. I marked them drinking, and finished correcting this morning while sober and sent for copy #4. It may be available in a couple of weeks depending on if I find more errors when it comes. I'll upload the book's HTML and PDF versions as soon as I decide I can release it.

Meanwhile, I can get back to Mars, Ho! this week.

Now that I've had some time to clear my head, I want to expand on my original feelings. I'm pissed off about this, and my temper flared through on the original post. I'm leaving it as is because I'm not going to edit it to make myself look better, and because it sums up my feelings pretty succinctly. How would you feel if something you worked on under the promise of building the best site for a community was regularly and routinely causing corporate firewalls and IDS systems to go off like crazy?

You'd be pissed. Had we known about this behaviour in advance, it would have been disabled at golive or in a point release, and a minor note would have gone up about it. Instead, I found out because we were tripping a user's firewall causing the site to get autoblocked. I realize some people feel this is acceptable behaviour, but a website should *never* trigger IDS or appear malicious in any way. Given the current state of NSA/GCHQ wiretapping and such, it means that anything tripping these types of systems is going to be looked at suspiciously to say the least. I'm not inherently against such a feature (IRC networks check for proxying for instance), but its clearly detailed in the MOTD of basically every network that does it.

There wasn't a single thing in the FAQ that suggested it, and a Google search against the other site didn't pop something up that dedicated what was being done; just a small note that some proxies were being blocked. Had the stock FAQ file, or documentation, or anything detailed this behaviour, while I might still have thought it wrong, at least I wouldn't have gotten upset about it. I knew that there was proxy scanning code in slashcode, but all the vars in the database were set to off; as I discovered, they're ignored leading me to write a master off switch in the underlying scanning function.

Perhaps in total, this isn't a big deal, but it felt like a slap in the face. I know I have a temper, and I've been working to keep it under wraps (something easier said than done, but nothing worthwhile is ever easy). CmdrTaco himself commented on this on hackernews and I've written a reply to him about it. Slashdot did what they felt was necessary to stop spam on their site, and by 2008, slashcode only really existed for slashdot itself; other slash sites run on their own branches of older code. Right or wrong, such behaviour should be clearly documented, as its not something you expect, and can (and has) caused issues to users and concerns due to lack of communication. Transparency isn't easy, but I have found its the only way to have a truly healthy community. Perhaps you disagree. I'll respond to any comments or criticisms left below.

Chapter One
Previously

Pirates
        Nothing happened in the last week that I didn't log in the ship's log. At least not what you want to hear, I get it. You don't need to know every time I take a shit or what I had for breakfast, right? Anyway, the whores pretty much behaved themselves. Like the log says, robots were trying to fix the busted generator but I knew they couldn't. They do what they're programmed to do no matter how impossible.
        Anyway, after a week there were some more little rocks in our way, but these were mapped; we could just go around them. The computers would do the actual steering but I have to sit in the pilot seat in case the four of them disagree about something and I have to make a decision. I've never seen that happen, though.
        While we were driving around the rocks, Wild Bill called over the MASER link. "John, Bill here. I'm about a light minute ahead of you and I'm standing still again, but this time it's on purpose. There's pirates ahead, and I can't outrun them on batteries. If your systems are all in good shape, run like hell. If you're having problems you should stop."
        Shit. I could out run them on one generator but what if the other one went out? Hell, I could just detour around them. Too bad Bill didn't have that advantage, batteries just didn't hold enough energy.
        I answered him back. "Pirates? This far out? Are you sure they're pirates?"
        It would be a couple of minutes before I heard back. I put the course correction into the computers' input console while I waited, then addressed the folks on board. "Passengers and cargo, attention. Prepare for unexpected gravity changes. That is all."
        Bill answered. "It's a fleet and they're not listed in the computer. Hell if I know what they're doing out here."
        Damn. Bill was a damned good friend who had helped me out of jams more than once. And he was hauling tons of different metals, a valuable cargo inside a valuable ship. His short circuit could have been sabotage; pirates have been known to infiltrate the company before. The company wouldn't too much mind pirates killing Bill but they'd hate to lose the ship and cargo, so maybe I wouldn't get in too much trouble for what I planned. I picked up the phone and addressed the ship's P.A. System. You can probably get a lot more detail from the computers, but anyway I got on the P.A. "Attention, ladies, this is the captain," I said. "Strap down, we're going to have some crazy gravity in a few minutes. That is all."
        I strapped myself into the pilot's chair myself. I turned the boat around and decelerated, shut down half the engines, made one look like it was sputtering, and informed Bill to get ready. Then I went toward the pirates while the computers figured out the trajectory for what I'd planned. I'm glad I have those computers, I could never do the math myself.
        They saw me, and I pretended I'd just noticed them and changed course. I wasn't kidding when I told the women gravity was going to be weird.
        They took chase. I went just slow enough to keep them the right distance and get where I was headed when I was headed there. From the radar it looked like they were steering those things by hand. Good, that raised my chances. Actually there wasn't any danger to me since I could outrun 'em easy and they can't shoot at me or anything that might damage the boat and cargo, which is what their goal is. But it raised my chances of saving Bill's ship.
        You know how the pirate fleets work, with a lead ship carrying an EMP. They don't know we designed these ships with pirates in mind and their EMP wouldn't stop us. And I didn't want them to know so I sent them a nice little present, fired from the rail.
        I hear the pirates still use gunpowder.
        The bastard's ship exploded and we were almost there –
        When I reached the right spot we took off like a bat out of hell. Ten seconds later the poor pirates got caught in the rain, as we say. They probably all died. I sure hope so, murderous bastards after my friend!
        I set the course back to Mars and addressed the ladies. "You can unstrap now."
        Time for inspection, since I'd pushed her hard on one generator.
        Like it says in the log, it was fine but a little warm. The engines were in good shape, too, but I shut down the one I made stutter for twenty four hours, just like the book says.
        This called for a beer. Hell, this called for champagne but I didn't have any. I started back to my quarters for a beer.

To be continued.

As the title says. An injury two days ago apparently requires surgery. I will be on hiatus for as long as I need to be.

Mattie_p

EDIT: surgery is complete, she is in recovery right now. She and I are both pretty tired right now. I'm home to care for my son and my wife took over at the hospital. I'll try to keep everyone posted but there is a lot I don't know at this point.

I've been a little busy this week, too busy to spend much time soylenting. I've only written about three more paragraphs of Mars, Ho!; I've been working on Nobots and The Paxil Diaries. The Paxil Diaries was waiting on my porch when I got home from Patty's Tuesday evening, and boy was it a mess. I've mostly been working on it. It's funny how much easier it is for me to notice mistakes on paper I miss on screen.

I finished editing it again last night and am waiting for another copy, which they haven't shipped yet. When it comes I'll go over it again, upload the revisions and buy another copy. It may be green outside before you can get a copy after all.

Nobots needed more sales outlets, so I worked on that, too. You should be able to get it at bookstores in a few weeks. If you bought a copy last year, you may own a rare book. If my name is on the bottom right of the front cover instead of right under the title, you have one of fewer than two dozen copies. It should be worth something in a decade or so.

I may work on the Mars book today, but then again I might just take the day off, take the computer to Felber's and watch Cosmos on Hulu since channel 55 was off the air last night; their web site said there was equipment failure. And drink beer in the beer garden and listen to music and enjoy the 65 degrees they're forecasting.

Or maybe sweep the floor... nah.

We're testing a new configuration between the site and the database. There may be unexpected issues with the site while we're testing. Keep calm and carry on.

I'm seeing a lot of unwarranted (IMO) down-moderation lately, mostly on posts that express a minority opinion or that question a majority opinion (judging minority/majority based on discussion context). As a result, there are many posts ranked -1 or 0 that probably deserve higher scores.

I'd like to remind moderators that you're supposed to "Concentrate more on promoting than demoting". In the meantime, I'm going to spend all my mod points on posts that I think have been modded down unfairly. I encourage others to consider spending some mod points in this fashion as well.

Now we're a few weeks in, and most stories when I come here are re-treads of things that I read on Slashdot a few days earlier. There's no point commenting on them, because I've already commented on the ones I'm interested in on Slashdot. Everyone else seems to feel the same way, because I rarely see a story with more than 10 comments. For a site that is meant to be all about the comments, that's an abject failure.

How could this have been solved? Well, as I proposed around launch time, the editors could have made a point of commenting on each story to prime the pump. When a story scrolls off the bottom of the front page with fewer comments than there are editors, then it's a failure. It means that either editors are posting stories that they're not interested in (in which case, why are they posting them?) or that they don't actually visit the site (in which case why are they editors?).

The only stories currently on the front page with more than 12 comments are 'people opting for dumb phones instead of smartphones' (which I'm just about to read - sounds like a typical rehash of the 'I have no self control so I'm going to use crappy technology to limit my exposure to stuff' story) and 'SCOTUS Signals Support for Corporate Religion?'. Where's the tech news? Are there any people here interested in discussing tech stories?

It feels like the staff gave up after the public temper tantrum between two of them and the community followed.

I'd planned on traveling to Cincinnati last Monday to visit my daughter and came down with the flu. I called Patty and told her it would be the next Monday; she works full time and is a full time student at Cincinnati State, and Monday is the only day she has off.

I looked her address up on Google Maps. It looked pretty easy to find. "Don't trust Google," Patty said. "They're doing road construction and it will try to send you down a road that's closed. Take the Hoppit exit, turn right and I'll meet you at the Shell station.

My nose was still producing copious amounts of snot, I was still coughing up lots of mucus but felt a hell of a lot better than I had last week. I woke up about 5:30 Monday morning, did my morning routine functions, especially coffee, one function of which was checking my phone. Three missed calls and a voicemail from Patty. I called, knowing she wouldn't answer because she's never awake that early and left a message that I was on my way and to call when she woke up.

I have a big laptop bag and a small laptop; the bag had cost me $5 and came with a broken laptop. I put spare clothing, charging accessories in it and loaded it, my battery jumper, and Patty's cat's ashes in the car.

I had a half tank of gas and figured it would get me to Indiana, where fuel would surely be cheaper. After all, it's a red state and Republicans hate taxes, right? No such luck, I was down to an eighth of a tank by the time I reached Bloomington.

It's a little frustrating that Cincinnati is southeast of Springfield, but you have to go northeast to get there unless you want to drive over three hundred miles of two lane road with 30 to 45 MPH speed limits and lots of stop signs and so forth. It would take forever that way.

Gas was a nickle cheaper than Springfield; $3.55. I put twenty bucks in, figuring I'd fill up in Indiana and started on my way again. I had my phone plugged into the car stereo for times there was no music and I'd heard all the CDs, which I'd neglected to change before I left. There was a rest area so I stopped to urinate and change CDs. I checked the phone; Patty had called. I called back, and again she warned me about Google.

Apparently people from Illinois aren't welcome in Indiana, as the usual "Welcome to [state]" sign was nowhere in evidence. The only way I knew I'd crossed state lines was that the pavement got a lot worse. I-74 had apparently been badly neglected for years in Indiana, except for a stretch by Indianapolis. Gasoline was more expensive than at home.

The sun was shining, the pavement was dry, and there was little traffic. "Welcome to Ohio!" the big sign proudly proclaimed in bright graphics as the pavement improved. I reached Cincinnati and the traffic was terrible. I-74 East split into I-75 north and south; I guessed south but wasn't sure. I pulled over to the shoulder and called Patty to make sure I wasn't going the wrong way. I wasn't.

The next exit was the Hoppit exit. I met Patty at the gas station. "You shaved!" she said.

"Yeah, my upper lip hasn't seen the sun since before you were born." Patty had never seen me completely shaven; most of her life I've had a beard, or at least a mustache when my chin hair went gray.

"I don't like it," she said, frowning."

"Neither do I. I'm growing it back this fall." I noticed the gas cap door on her car was open as she pulled out and was about to honk to let her know when she pulled over and shut it.

We got to her apartment and we hugged and I shook her fiance's hand an gave Patty the metal box and envelopes. I hadn't opened one of them, which had come from Coble Animal Hospital. I'd thought it contained Princess' ashes but they called a week later to inform me I could pick her up.

"Ooh, this is a pretty box," she said. "What's in it?"

I still can't believe I spent over three hundred dollars for a dead cat, part for the vet to tell me she was dying and part to have her cremated, since the ground was frozen and I couldn't bury her. I discovered that animals and humans are cremated in the same crematorium, which is why it's so expensive. If Little One dies in the winter I'm storing her in a deep freeze until the ground thaws.

Patty opened the unopened envelope and started crying. It was a plastic placard that read "PRINCESS" and had her paw prints in it. No, I guess I didn't spend $300 on a dead cat, I spent it on my daughter. "Put this with Calie under the tree," she instructed. "When you move, take it and Calie's grave marker with you."

Colby had planned on making Reuben sandwiches for lunch but the corned beef was still frozen. "Let's go to Chick Filet," he said. "OK," I replied,"but then Patty needs a phone." Her iPhone had been broken for months, its screen cracked. And she'd liked my phone and especially liked my low phone bill.

We had chicken sandwiches and went to Best Buy. The price of the phone was half what I'd paid for mine. She was trying to decide between it and a more expensive one with a front facing camera but decided she liked the idea of it being waterproof and resistant to shock.

"Lets buy a TV while we're here" she said to Colby. After they talked for a while she said "well, I'm buying a TV. I have the money." They have an old twenty two inch tube TV that doesn't work and a little nineteen inch widescreen.

But she didn't like the prices so we went to H.H. Gregg, whose prices were no better than Best Buy's. Best Buy's crack Geek Squad couldn't activate Patty's new phone so we took it home and did it ourselves.

I'd bought Gravity, which had come from Amazon amazingly the day before it was supposedly released for sale. It was a "combo pack" with a DVD, Blu-Ray and download. I'd brought the Blu-Ray for Patty, and we watched it using her Playstation and little TV set.

None of us had seen the previous night's Cosmos so she fired up Hulu plus on the Playstation. After watching it and an episode of Doctor Who I decided that I wanted Hulu Plus.

The next morning she gave me a big bowl of corned beef, cabbage, carrots, and potatoes, and two T shirts. One was almost a joke; a St. Patrick's Day Reds shirt. The other was hawking some video game, a nerdy shirt I'll wear proudly.

She wanted to see how badly Google would have set me astray so I gave her my phone. She was amazed. "They got it perfect, that's how I told you to go." I loaded up the car, we said our goodbyes and I set off on the long journey home.

The trip home was as unpleasant as the trip there had been pleasant. First, I missed my turn to get on I-74. Five miles later I got on I-75, saw I was headed to Dayton and took the next exit. I stopped at a gas station, got gas, and consulted the map.

It would be nice of these things came with manuals. I think it ironic that everything used to have a detailed manual when technology was primitive enough you didn't need one, and now that interfaces have only icons and no way to discern WTF they mean, they don't. Let's see, looks like I go that way...

The radio was playing commercials so I switched it to the phone to listen to KSHE. The disk jockey started giving directions! "Go west on" whatever street the gas station was on "point seven miles and turn right." It wasn't KSHE, it was Google Maps. It easily got me back on I-74 north and it wouldn't shut up so I switched back to the radio.

Traffic was horrible; a semi that read "TARGET" zoomed past me doing at least twenty miles above the speed limit and almost made me miss my exit. Looks like it isn't just their IT that could use more training.

A little green sign with white lettering said "Welcome to Indiana". It started snowing. Twenty miles later visibility was poor, and twenty minutes after that the pavement was covered.

It was a miserable trip. The snow stopped around Indianapolis and the traffic was almost as bad as Cincinnati. Halfway to Illinois the wind started blowing. A couple of semis almost got blown off the highway.

Gas in Bloomington was $3.49.

When I got home there was a box on my doorstep; The Paxil Diaries had arrived. I'd screwed it up terribly. So you still can't have a copy yet...