Earlier tonight, I modified our varnish rules to redirect all traffic to https://soylentnews.org if they came in as plain HTTP. Unfortunately, due to dropping SSLv3 support to prevent POODLE attacks, IE6 clients will no longer be able to reach SoylentNews. If this seriously inconveniences a large number of users, we may go through the trouble of whitelisting IE6 to drop down to HTTP only.
In addition, I applied an experimental update to production to try and clear as many errors as possible from the Apache error logs, in an attempt to continue isolating any remaining bugs and slowdowns. I also ripped out more dead code related to FireHose, Achievements, and Tags. As such, site performance appears to roughly be back to where it should be, and I have yet to see any 500 errors post-upgrade (though I concede that said update has only been up for about 2 hours at this point).
Tor traffic is set to bypass HTTPS due to the fact there is no way to prevent a self-signed certificate warning, and by design, tor both encrypts and authenticates hosts when connecting to them. A few lingering issues with the tor proxy were fixed with most recent code push, and the onion site should be back to functioning normally
P.S. I'm aware that the site is generating warnings due to the fact we use a SHA-1 based certificate. We will be changing out the certificate as soon as reasonably possible.
(Score: 0) by Anonymous Coward on Sunday June 07 2015, @01:17AM
Don't your web server logs indicate how many requests come from IE 6?
(Score: 4, Informative) by NCommander on Sunday June 07 2015, @01:23AM
As part of our policy on retaining as little information as possible on our users, the short answer is we don't record it in the access logs. We just record IP address, the URL accessed, return code, and bytes sent. We ran piWik for awhile to get some real stats, but disabled it after a few months.
Still always moving
(Score: 1, Funny) by Anonymous Coward on Sunday June 07 2015, @01:31AM
Peewick! PEEWICK! PEEEEEEEEWICK! PEEEEEEEEEEEEEEEEEEEEEEWICK! PEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEWICK!
(Score: 2) by cmn32480 on Sunday June 07 2015, @01:48AM
Perhaps, given the Rehash update, we ought to run piWik again to see what the traffic looks like using the last run as a baseline.
And frankly, if people are still running IE6, they deserve to be cut off.
"It's a dog eat dog world, and I'm wearing Milkbone underwear" - Norm Peterson
(Score: 2) by NCommander on Sunday June 07 2015, @01:25AM
A second note is CloudFlare disabled SSLv3 awhile ago, which in turn would render large parts of the internet inaccessible to IE6.
Still always moving
(Score: 3, Interesting) by TheRaven on Sunday June 07 2015, @09:00AM
sudo mod me up
(Score: 5, Insightful) by Anonymous Coward on Sunday June 07 2015, @01:35AM
(Score: 1) by Soupbowl on Sunday June 07 2015, @01:40AM
Yeah I agree.
(Score: 2) by Techwolf on Sunday June 07 2015, @02:11AM
What about all the
OK
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, admin@soylentnews.org and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.
That I have MANY other been getting and bitching about in other stories?
(Score: 2) by frojack on Sunday June 07 2015, @02:15AM
Yeah that is kind of annoying, and I doubt any reports by us would ever be able to be tied back to the logged event.
These things are always useless..
If I simply hit back button and resubmit, it invariably goes through.
No, you are mistaken. I've always had this sig.
(Score: 2) by NCommander on Sunday June 07 2015, @02:16AM
Have you gotten any within the last hour? The issue is intermediate and isn't leaving much of an error log to work from. I'm trying to remove noise from the logs to better isolate the error.
Still always moving
(Score: 2) by gman003 on Sunday June 07 2015, @04:40AM
I got two trying to load comments on another article, and one trying to reply to your comment. It's not as many as I was seeing but there's still errors there.
(Score: 2) by NCommander on Sunday June 07 2015, @05:05AM
I was able to catch at least one 500 in the error_log file (posted elsewhere in this thread). I'm slightly at a loss whats causing it, but will continue to debug.
Still always moving
(Score: 2) by goodie on Sunday June 07 2015, @08:05AM
Just got one trying to load this thread's comments. hit back and click again, worked.
(Score: 1) by PReDiToR on Sunday June 07 2015, @12:34PM
It's indistinguishable from magic [catb.org].
Do not meddle in the affairs of geeks for they are subtle and quick to anger.
(Score: 2) by Techwolf on Sunday June 07 2015, @02:25AM
This error cause the screw on a couple posts earlier today. I did a quick post about f-droid. Then had time later to look up the website name, did a reply to that with the name of the site, hit post and got the error, hit the back button and tired again. It went through. HOWEVER, I found myself logged out and that post was done as AC.
(Score: 2) by NCommander on Sunday June 07 2015, @02:40AM
Yeah, I've been having the same sort of issues. This patch fixes everything I could find, but the error log was spamming a lot of warnings and it was hard to isolate what was noise and what wasn't so I took a blowtorch and removed as much as I could. I'll get another changeset together to remove more warnings to find the actual error
Still always moving
(Score: 0) by Anonymous Coward on Sunday June 07 2015, @03:19AM
It didn't happen before. So something changed to introduce it, and this happened recently.
(Score: 2) by NCommander on Sunday June 07 2015, @03:24AM
Site upgrade to rehash is what broke it. These are issues that didn't show up in the development environment before deployment.
Still always moving
(Score: 0) by Anonymous Coward on Sunday June 07 2015, @03:41AM
What the fuck is rehash? If the site worked without rehash, but doesn't work with rehash, and whatever rehash is it doesn't offer any obvious benefit, then rehash should be removed.
(Score: 1, Interesting) by Anonymous Coward on Sunday June 07 2015, @04:30AM
Maybe you missed this Upgrade Plans: Slashcode 15.03/Rehash 15.04 [soylentnews.org] which was published to the site on March 13th. In part, it stated:
The code that runs the site is based on *old* versions of Apache and perl; they are so old that they are no longer supported (i.e. no longer receive security or bug fixes.) As I understand it, this leaves the site open to security vulnerabilities, among other things. Of course, there were incompatible changes made to those so things that used to work fine, no longer do so. Besides those changes, the DB engine, as I understand it, needed an upgrade, too.
Car analogy: the car looked okay, but the engine and the frame were all rusted out. We've done a complete replacement of all that underneath stuff, and things need to get ironed out and tuned so that it all works together again. But, now, we have low-profile radial tires for better grip, a turbo on the engine for more power, and a completely new suspension system for better handling in the turns and a better ride. Everything didn't quite line up properly, so we're doing some adjustments to get it all working together properly.
That is from my limited perspective and understanding; I'm sure Ncommander or one of the other devs will correct me if I'm too wide of the mark on that.
Remember how the site used to regularly *crash* for the first few months are go-live? The site had become so stable that these current annoyances are causing such a stir suggests to me just how fine a job our devs have done. I, for one, commend them for all their hard work which they generously donated in their free time, and without pay.
(Score: 0) by Anonymous Coward on Sunday June 07 2015, @07:28AM
You can't say that! You just can't!
ACs only post AC because they don't want the blowback! Named accounts are the only way you can tell who someone really is! Nobody would ever use a pseudonym, that would just be dishonest...
(Score: 2, Interesting) by Anonymous Coward on Sunday June 07 2015, @04:29AM
Could the errors be related to the mod_perl 2 conversion?
There are know issues where it over aggressively caches old code and modules that have changed and will not de-cache then without a hard restart, even when tools used to hint or prompt it to refresh it's cache are used. This causes it to incorrectly try to call into the no longer current code resulting in server errors. The caching can be inconsistent across different apache child processes, which further ads to the mayhem.
(Score: -1, Offtopic) by Anonymous Coward on Sunday June 07 2015, @02:15AM
I can feel my BITS being MANGLED and it's so EROTIC. I want to shove a FIREHOSE into my ANUS and receive an ENEMA of current EVENTS.
(Score: 2) by maxwell demon on Sunday June 07 2015, @05:08AM
That post is rightfully at -1, but wrongfully marked as spam. Please, don't abuse the spam moderation.
Maybe the spam marking should be untied from the moderation system, so people don't get tempted to misuse the spam moderation.
The Tao of math: The numbers you can count are not the real numbers.
(Score: 0) by Anonymous Coward on Sunday June 07 2015, @05:19AM
> Maybe the spam marking should be untied from the moderation system, so people don't get tempted to misuse the spam moderation.
If it becomes a regular problem, then we can cross that bridge.
(Score: -1, Troll) by Anonymous Coward on Sunday June 07 2015, @02:21AM
Lovely Spam! Wonderful Spam! Spam!
(Score: -1, Spam) by Anonymous Coward on Sunday June 07 2015, @02:23AM
Spam! Spam! Glorious Spam!
(Score: -1, Spam) by Anonymous Coward on Sunday June 07 2015, @02:25AM
Spam! Heavenly Spam! Spam!
(Score: 2) by aristarchus on Sunday June 07 2015, @10:21AM
Bloody Vikings!
But I do have to say, in regards to the naming, rehash could be strangely appropriate, since real hash is some form of chopped meat (not necessarily nicely spiced like Spam!), and SoylentNews is people, just like its namesake.
(Score: 1) by Absolutely.Geek on Monday June 08 2015, @09:55PM
In my country "real" hash is illegal and has nothing to do with animal products you insensitive clod.
Don't trust the police or the government - Shihad: My mind's sedate.
(Score: 2) by kaszz on Sunday June 07 2015, @02:53AM
Some articles show up without any comments at all. Only a reload fixes it.
(Score: 2) by NCommander on Sunday June 07 2015, @02:56AM
That was due to the site 500ing when trying to load comments. I'm not 100% I've successfully nailed the cause of this, but I haven't seen it happen since the update.
Still always moving
(Score: 2) by kaszz on Sunday June 07 2015, @09:00AM
It occurred just minutes before posting that comment.
(Score: 0) by Anonymous Coward on Sunday June 07 2015, @03:01AM
Tor traffic is set to bypass HTTPS due to the fact there is no way to prevent a self-signed certificate warning, and by design, tor both encrypts and authenticates hosts when connecting to them. A few lingering issues with the tor proxy
But Tor doesn't help with last-mile attacks between that tor proxy and the server. Personally, I'd be fine with a self-signed cert warning because you only need see it on your first visit, then you can tell your browser to remember it permanently. You can publish the fingerprint of the cert on the non-tor website so anyone can verify it that way if needed.
(Score: 3, Informative) by NCommander on Sunday June 07 2015, @03:07AM
The connection is terminated within our network if you're using our onion site, which is no different that non-tor SSL (we terminate SSL on the load balancer). We've documented and stated before. We've tried before to terminate on the web frontends, but have had odd side effects with that setup.
Still always moving
(Score: 2) by compro01 on Sunday June 07 2015, @03:38AM
It does when it's a hidden service, like SoylentNews is (http://7rmath4ro2of2a42.onion). There isn't an exit node, so there is no possibility of a last-mile attack, as the last layer of the encryption isn't unwrapped until it's within Soylent's network, unlike how it works connecting to a site on the open web via Tor.
(Score: 4, Informative) by NCommander on Sunday June 07 2015, @03:54AM
OP got it in one. When you connect to soylentnews.org directly with Tor Browser, you'll get our standard SSL certificate (this incidentally should work now over tor, though much slower than if going via the onion site) . If you come out via our onion site, you pop out on boron, which is one of our misc boxes, and the connection travels the last mile unencrypted within Linode Dallas data center. While I'd love to do something about that last mile problem, at the moment, its not practical to fix as we don't have complete control of our infrastructure due to being on VPSes :(.
Still always moving
(Score: 3, Funny) by axsdenied on Sunday June 07 2015, @04:09AM
Luckily I am still on IE5 :-)
(Score: 3, Interesting) by NCommander on Sunday June 07 2015, @04:46AM
With the cleaned up logging code, I managed to catch the 500, though I'm at a loss at what caused it; it appears a bunch of things returned a value vs. a HASH when wanted ...
Still always moving
(Score: 1) by barrahome on Sunday June 07 2015, @05:49AM
Nice work NC and SoyLentNews team!
(Score: -1, Troll) by Anonymous Coward on Sunday June 07 2015, @07:05AM
Nice ass kissing, brown mouth.
(Score: -1, Troll) by Anonymous Coward on Sunday June 07 2015, @07:31AM
Do you touch your mother with those fingers?
Perhaps you just touch yourself...
(Score: -1, Troll) by Anonymous Coward on Sunday June 07 2015, @07:47AM
I touch every inch of your mother's skin.
(Score: 2) by NCommander on Sunday June 07 2015, @06:02AM
I think I found the problem. The database starts timing out or returning bad data on high loads (can't tell specifically). I retuned the frontend mysqld to try and improve performance; I forgot that the backend ndbd and frontend ndbd don't share tuning values. Now to wait and see if the 500 errors return.
Still always moving
(Score: 2, Disagree) by Adrian Harvey on Sunday June 07 2015, @10:04AM
I wish you hadn't. This is a public website, my comments etc here are public. I use my real name. Now your site goes slower because there's no caching at my ISP. Why force me into using some encryption I don't need or want?
(Score: 2, Informative) by Refugee from beyond on Sunday June 07 2015, @10:09AM
MitM + zero day vulnerability = ???
Steal cookies, shill the hell out of your account = ???
If you think you have nothing to hide it does not mean it's true. Your BB would like to know what you read. Or maybe censor whatever you read, it's not like HTML manipulation is hard.
Instantly better soylentnews: replace background on article and comment titles with #973131.
(Score: 2) by Justin Case on Sunday June 07 2015, @11:48AM
But I just had a page with no moderation dropdown options. The dropdowns were there but nothing visible in the choices. View source revealed:
<select id="reason_193075" name="reason_193075">
<option value="HASH(0x7f05ff8)"></option>
<option value="HASH(0x7f074c0)"></option>
<option value="HASH(0x6038fc0)"></option>
...
Hit reload and back to normal.
(Score: 2) by NCommander on Sunday June 07 2015, @06:22PM
This is related to the occassional 500s. As best I can tell, either a mySQL query is failing or a load from memcache fails, and somethign goes horridly pearshaped.
Still always moving
(Score: 2) by r00t on Sunday June 07 2015, @05:21PM
Hey guys, not sure how you mitigated FREAK, POODLE , LOGJAM etc but check out your ssl config on https://www.ssllabs.com/ssltest/analyze.html?d=soylentnews.org. [ssllabs.com] I'm getting funky errors about broken encryption in firefox. Qualys is pointing out some problems with the DH mitigation involving logjam.
(Score: 2) by NCommander on Thursday June 18 2015, @12:50AM
Wow, this is a late reply to this, but I regenerated the DH prime a few days after this site upgrade. SSLLabs is showing us with an A rating with no major issues. Once we implement HSTS, we'll go to A+, and hopefully always remain there.
Still always moving