Stories
Slash Boxes
Comments

SoylentNews is people

posted by NCommander on Wednesday April 09 2014, @07:26PM   Printer-friendly
from the seething-with-anger dept.
I've pushed an emergency fix to production to close bug #142 on the tracker. For those unaware, Slashcode portscans every user when they login or post a comment. While we knew that there was some code involved in checking for open proxies, I thought it had been disabled, and the default settings in the database all default to off. The fact of the matter though is the backend was ignoring all disable checks in the database and scanning every IP to see if they were a proxy on ports 80, 3123, 8000, and 8080.

I'm f****** seething; this is unacceptable for any site, and this behaviour isn't documented anywhere; we've been portscanning since day one and were completely unaware of it. My guess is almost everyone here was unaware of this "feature" as well. Our submitter reports slashdot did this as well. There is no notification or link in the FAQ that this is done, unless you were checking your firewall rules religiously, this would have been completely unnoticed.

I'm seething and furious at the moment. How on earth is this acceptable behaviour? I understand proxy scanning; most IRC networks do it, but they notify you that they are doing so. Furthermore, a basic web application should not be probing their end users; I'm absolutely flabbergasted that this exists, as were most of the staff when it was brought to our attention. On behalf of the site, I want to offer a formal apology for this clusterf***.

Addendum: Since writing this, I've written a follow up on why this got me so upset in my journal. I've got journal replies set to on, and will respond to anyone both here and there.Here's the revelent bit of code from Slash/DB/MySQL/MySQL.pm (yes, it lives in the DB API, no I don't know why)
sub checkForOpenProxy {
my($self, $ip) = @_;
# If we weren't passed an IP address, default to whatever
# the current IP address is.
if (!$ip && $ENV{GATEWAY_INTERFACE}) {
my $r = Apache->request;
$ip = $r->connection->remote_ip if $r;
}

# If we don't have an IP address, it can't be an open proxy.
return 0 if !$ip;
# Known secure IPs also don't count as open proxies.
my $constants = getCurrentStatic();
my $gSkin = getCurrentSkin();

my $secure_ip_regex = $constants->{admin_secure_ip_regex};
return 0 if $secure_ip_regex && $ip =~ /$secure_ip_regex/;

# If the IP address is already one we have listed, use the
# existing listing.
my $port = $self->getKnownOpenProxy($ip);
if (defined $port) {
#print STDERR scalar(localtime) . " cfop no need to check ip '$ip', port is '$port'\n";
return $port;
}
#print STDERR scalar(localtime) . " cfop ip '$ip' not known, checking\n";

# No known answer; probe the IP address and get an answer.
my $ports = $constants->{comments_portscan_ports} || '80 8080 8000 3128';
my @ports = grep /^\d+$/, split / /, $ports;
return 0 if !@ports;
my $timeout = $constants->{comments_portscan_timeout} || 5;
my $connect_timeout = int($timeout/scalar(@ports)+0.2);
my $ok_url = "$gSkin->{absolutedir}/ok.txt";

my $pua = Slash::Custom::ParUserAgent->new();
$pua->redirect(1);
$pua->max_redirect(3);
$pua->max_hosts(scalar(@ports));
$pua->max_req(scalar(@ports));
$pua->timeout($connect_timeout);

#use LWP::Debug;
#use Data::Dumper;
#LWP::Debug::level("+trace"); LWP::Debug::level("+debug");

my $start_time = Time::HiRes::time;

local $_proxy_port = undef;
sub _cfop_callback {
my($data, $response, $protocol) = @_;
#print STDERR scalar(localtime) . " _cfop_callback protocol '$protocol' port '$_proxy_port' succ '" . ($response->is_success()) . "' data '$data' content '" . ($response->is_success() ? $response->content() : "(fail)") . "'\n";
if ($response->is_success() && $data eq "ok\n") {
# We got a success, so the IP is a proxy.
# We should know the proxy's port at this
# point; if not, that's remarkable, so
# print an error.
my $orig_req = $response->request();
$_proxy_port = $orig_req->{_slash_proxytest_port};
if (!$_proxy_port) {
print STDERR scalar(localtime) . " _cfop_callback got data but no port, protocol '$protocol' port '$_proxy_port' succ '" . ($response->is_success()) . "' data '$data' content '" . $response->content() . "'\n";
}
$_proxy_port ||= 1;
# We can quit listening on any of the
# other ports that may have connected,
# returning immediately from the wait().
# So we want to return C_ENDALL. Except
# C_ENDALL doesn't seem to _work_, it
# crashes in _remove_current_connection.
# Argh. So we use C_LASTCON.
return LWP::Parallel::UserAgent::C_LASTCON;
}
#print STDERR scalar(localtime) . " _cfop_callback protocol '$protocol' succ '0'\n";
}

#print STDERR scalar(localtime) . " cfop beginning registering\n";
for my $port (@ports) {
# We switch to a new proxy every time thru.
$pua->proxy('http', "http://$ip:$port/");
my $req = HTTP::Request->new(GET => $ok_url);
$req->{_slash_proxytest_port} = $port;
#print STDERR scalar(localtime) . " cfop registering for proxy '$pua->{proxy}{http}'\n";
$pua->register($req, \&_cfop_callback);
}
#print STDERR scalar(localtime) . "pua: " . Dumper($pua);
my $elapsed = Time::HiRes::time - $start_time;
my $wait_timeout = int($timeout - $elapsed + 0.5);
$wait_timeout = 1 if $wait_timeout wait($wait_timeout);
#print STDERR scalar(localtime) . " cfop done with wait, returning " . (defined $_proxy_port ? 'undef' : "'$port'") . "\n";
$_proxy_port = 0 if !$_proxy_port;
$elapsed = Time::HiRes::time - $start_time;

# Store this value so we don't keep probing the IP.
$self->setKnownOpenProxy($ip, $_proxy_port, $elapsed);

return $_proxy_port;
}


Leave your comments below, I want to know how others feel about this "feature".

Update: We've confirmed that slashdot.jp and Barrapunto predate this feature being added to the codebase; according to the git log, it was added on commit 177e2213 at 2008-04-16 19:07:46 +0000.
 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 2) by combatserver on Thursday April 10 2014, @12:24AM

    by combatserver (38) on Thursday April 10 2014, @12:24AM (#29166)

    You do realize that this site exists as a result of the "Fuck Beta!" movement, right? I think the response to this discovery was proportional, especially since Slashdot is still doing the same port-scanning. Just one more reason to "Fuck Beta!".
    .

    I think NCommander is royally pissed because other people are (among other reasons, I am sure). Just another cut in the Death-by-a-thousand-cuts I've brought up more than once.

    --
    I hope I can change this later...
    Starting Score:    1  point
    Karma-Bonus Modifier   +1  

    Total Score:   2  
  • (Score: 0) by Anonymous Coward on Thursday April 10 2014, @10:03AM

    by Anonymous Coward on Thursday April 10 2014, @10:03AM (#29350)

    You do realize that this site exists as a result of the "Fuck Beta!" movement, right? I think the response to this discovery was proportional, especially since Slashdot is still doing the same port-scanning. Just one more reason to "Fuck Beta!".
    .

    I think NCommander is royally pissed because other people are (among other reasons, I am sure).

    I can 100% see where you are coming from, but I think the other side of the coin is that- in this day and age, the fault clearly doesn't lie with the author of the code. But rather with the person who _chose_ to run the _open source_ code _without auditing it sufficiently to notice this in either (a) the source code or (b) inspection of the network traffic to the server they were administering.

    Now _that said_, I can totally see myself being in NC's position, doing and feeling the exact same things for the exact same reasons.

    The fact of the matter is that even today, nearly a year post Snowden, the _entire internet security community_ knows that it screwed up pretty badly for a decade, and has a $#!+ ton of work to do. And that the first half of that work has to be done simultaneous with the knowledge that the 2nd half of that work will remain undone for the duration.

    What has happened is basically this (virtual pontiff hat just donned)- for 10 years, the internet security community got away with being pretty fucking lazy and cheap. For 10 years, the NSA and GCHQ and others facilitated and encouraged (with traditionally criminal tactics under sanction of modern governments) a laughable level of lazyness and cutting of corners in the internet security field. They fucking loved it. The big publicly traded companies fucking loved it- hell, good security ain't cheap, and that does cut into the bottom line.

    But then Snowden came and called out the naked emperor. And this is where we are. Lots of shit to clean up. Lots of dark corners to shine lights in and find more shit to clean up.

    Step 1) (with a service like soylent news): take a sample client/server user session, either live or testbed, and run tcpdump, and justify the existence of every packet you see.

    I really don't mean to sound arrogant. This stuff is not simple. It takes time, effort. Vigilance is exhausting. But getting angry at the slashcode authors? Seems useless and at least somewhat misdirected. Yet completely understandable (as an initial WTF?!? reaction)

    • (Score: 2) by NCommander on Thursday April 10 2014, @05:27PM

      by NCommander (2) Subscriber Badge <michael@casadevall.pro> on Thursday April 10 2014, @05:27PM (#29588) Homepage Journal

      Slashcode is 250k LOC, how do you quickly audit something of that size? I agree you should audit mission-critical codebases, but SN was put together in less than a week by our crack team of flying monkeys. The situation is further complicated that we're tied to an antique Apache + antique Perl because the Apache 2.x ports never were released open source and the API completely changed. We applied apparmoring to it as a bandaid, and we've modified slash and apache to run non-root which should hopefully reduce our attack profile.

      The situation was compounded that the code lies; unless you went through the 10k-ish MySQL.pm line by line, you might not have noticed the port scanning, or saw variables that prevent port scanning in some places, but not all places.

      The OpenBSD guys got this right; every bit of code they support gets audited and Theo really understands security on a level most people don't. That being said, OBSD seems to lack quite a few features that would prevent me from using it in production, i.e., something equivelent to AppArmor or SELinux for one. I'd love to see a Linux distro come out with that same policy of code auditing though.

      --
      Still always moving
      • (Score: 0) by Anonymous Coward on Thursday April 10 2014, @08:11PM

        by Anonymous Coward on Thursday April 10 2014, @08:11PM (#29682)

        Slashcode is 250k LOC, how do you quickly audit something of that size? ...

        I'll give a straight answer to your rhetorical question- Either (a) you don't, or (b) you throw several millions of dollars worth of paid eyeballs at it. I'm guessing nobody involved in SN has several millions of dollars to spare for that (nor the belief that the money wouldn't be better spent on other things), so the resulting choice is (a), but with the caveat- you accept the consequences. This thread being one of them.

        I feel like I could almost bet my life on the assertion that if you did spend those millions of dollars, you'd find a dozen other things just as horrifying. About my only criticism of SN I was trying to express was that some of the anger expressed was basically useless if not totally understandable. My advice is merely one of shifting the attitude. I myself take to heart the words of either Snowden, or someone fairly famous talking about the Snowden revelations who said- You basically have in this day and age to assume that several superpowers have owned any internet connected system you are working with. Even non-internet connected systems can have radar reflecting bugs embedded invisibly in their USB ports or anywhere else. We have a decade at least of basic security work before we can even delude ourselves into thinking we have any real cybersecurity anywhere. And that's an optimistic timeframe. I doubt I'll live to see the day I feel like I can write a private journal on a computer that doesn't get checked by big brother for - if not crimethink- terrorist profile calculations (not to mention advertising optimization systems).

        I mean fuck, seriously, the washingtonpost reporting on heartbleed referred to the situation with the words "infected systems". If even the washington fucking post can't be troubled to get the terminology of cybersecurity correct- distinguishing between such things as "buggy/vulnerable/outdated" and "infected" then we seriously have a very long way to go still. By choosing to use the word 'infected' WP is (unless they have unreported info they are withholding) furthering the same kind of disinformation campaigns that the NSA successfully used to keep people from doing security right. People need to understand that cybersecurity involves plenty of malicious actors, as well as unintentional mistakes. The fault of the heartbleed bug wasn't the committer of the bug, but the dozens of billion dollar companies that depended on the security of that code, but couldn't be troubled to throw a few engineers at the task of reviewing that code and finding the bug earlier. And the worst thing is that the NSA fucking loves corporate behavior like that.

        My point NC isn't that you should have done more, it's that you should have gotten less upset.