Right, so there's currently a DDoS of our site specifically happening. Part of me is mildly annoyed, part of me is proud that we're worth DDoS-ing now. Since it's only slowing us down a bit and not actually shutting us down, I'm half tempted to just let them run their botnet time out. I suppose we should tweak the firewall a bit though. Sigh, I hate working on weekends.
Update: Okay, that appears to have mitigated it; the site's functional at a reasonable rate of responsiveness.
Update2: Attack's over for now. You may go about your business.
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
(Score: 5, Funny) by Anonymous Coward on Sunday October 30 2016, @01:40PM
For SN, a DDOS is more than 4 people logged on at the same time ;) Tee hee
(Score: 0) by Anonymous Coward on Sunday October 30 2016, @01:44PM
Thanks for the update. I noticed SN was a little slow, so did my usual thing -- rebooted Firefox. Now everything else is snappy.
(Score: 0) by Anonymous Coward on Sunday October 30 2016, @07:49PM
Oh yeah? Well I reinstalled Windoze...
(Score: 2) by Bogsnoticus on Sunday October 30 2016, @11:34PM
I killed all of my old carrier pigeons, and bought new ones. Now SN works like a charm.
Genius by birth. Evil by choice.
(Score: 4, Funny) by kazzie on Sunday October 30 2016, @02:20PM
If I reply to this post, am I making things worse for the server load?
Any comments, please reply below.
(Score: 3, Informative) by The Mighty Buzzard on Sunday October 30 2016, @02:23PM
Nah, we're all good now. The webservers can totally handle this load. It was the firewall at the load balancer that was having issues. It had a rule or three in it that it shouldn't have.
My rights don't end where your fear begins.
(Score: 2) by kazzie on Sunday October 30 2016, @04:54PM
Thanks for the serious feedback, but it shows up a sarcasm fail on my part. :/
(Score: 2) by Gaaark on Sunday October 30 2016, @05:13PM
So, this was actually a GOOD thing! Keeps you on your toes and keeps your skillz up! :)
Congratulations on your fame! May your life be filled with DDOS, as the Chinese say, lol.
--- Please remind me if I haven't been civil to you: I'm channeling MDC. ---Gaaark 2.0 ---
(Score: 3, Funny) by Capt. Obvious on Monday October 31 2016, @03:31AM
I'm not sure, but someone is sure to post a response with the answer. Be sure to reload every few seconds to ensure you learn, as quickly as possible, how not to take actions that make the load worse.
(Score: 3, Insightful) by el_oscuro on Sunday October 30 2016, @11:18PM
Actually, this is one of the fastest and least crappy sites I have visited in years. From an observed performance perspective, it is basically google.
SoylentNews is Bacon! [nueskes.com]
(Score: 0) by Anonymous Coward on Monday October 31 2016, @11:08AM
Soylent News, the site that needs a DDoS to be as slow as the rest of the internet!
(Score: 4, Funny) by coolgopher on Sunday October 30 2016, @01:51PM
Have you been posting unpopular views in the comments section again, TMB? ;)
(Score: 4, Funny) by The Mighty Buzzard on Sunday October 30 2016, @02:02PM
What, me? Never!
My rights don't end where your fear begins.
(Score: 0) by Anonymous Coward on Monday October 31 2016, @11:11AM
I think that might qualify as unpopular view. ;-)
(Score: 2) by WizardFusion on Sunday October 30 2016, @02:09PM
Sigh, I hate working on weekends
It does keep you out of trouble though.!
(Score: 1, Interesting) by Anonymous Coward on Sunday October 30 2016, @02:22PM
Any idea how large the attack was?
Not sure of what units would make sense, how about, "Fraction of the recent attack on Krebs"?
Example: Was this a 0.1 Krebs attack?
https://krebsonsecurity.com/2016/09/krebsonsecurity-hit-with-record-ddos/ [krebsonsecurity.com]
(Score: 4, Funny) by cmn32480 on Sunday October 30 2016, @02:26PM
Big enough that the notoriously slow editorial staff noticed it....
"It's a dog eat dog world, and I'm wearing Milkbone underwear" - Norm Peterson
(Score: 3, Informative) by The Mighty Buzzard on Sunday October 30 2016, @02:27PM
Pretty freaking tiny as far as DDoSes go. Maybe a few dozen hits per second. It's still going on by the way. It just isn't actually big enough to slow us down after removing a couple rules that shouldn't have been in the load balancer's firewall.
My rights don't end where your fear begins.
(Score: 0) by Anonymous Coward on Sunday October 30 2016, @03:18PM
Pretty freaking tiny as far as DDoSes go. Maybe a few dozen hits per second.
A few dozen hits per second? How's that even a DoS? Are you sure it's a DDoS and not some broken javascript/page code or broken browser interpretation of it? I've loaded some sites with noscript on and their pages kept reloading over and over again...
(Score: 4, Interesting) by The Mighty Buzzard on Sunday October 30 2016, @03:32PM
We had a rate-limiting rule that's appropriate for the TOR gateway but extremely inappropriate for the load balancer loaded in iptables on the load balancer. It was causing us to drop outgoing packets over a certain rate. That's a good thing to have on the TOR gateway because it keeps people from slamming our web frontends anonymously (only the TOR gateway would feel the slowdown) but on the load balancer it makes no sense since we've already fulfilled the request at that point.
My rights don't end where your fear begins.
(Score: 3, Informative) by The Mighty Buzzard on Sunday October 30 2016, @03:36PM
But, yeah, it was definitely a DDoS attempt. Requests coming in from a repeating multitude of addresses for comment pages, journal pages, user pages, and the like but no requests for the images or css on the site like browsers would generate.
My rights don't end where your fear begins.
(Score: 3, Insightful) by Unixnut on Sunday October 30 2016, @04:20PM
Amateurs, if you want to DDOS a site you want to pull in all the bandwidth heavy stuff like images in your hit :-P
(Score: 5, Interesting) by blackhawk on Sunday October 30 2016, @09:14PM
That would really just saturate the outgoing bandwidth, and if the site is hosted on Amazon or similar, it's going to take a lot to saturate the bandwidth.
It's more effective to find the pages that take the most server resources to serve up and slam those. You want to target ones that are hitting the database a lot, really tie up those back end resources, that will stop pretty much every page that needs DB access from running in a reasonable time-frame.
A textbook example for me was back in the 00's when I was working with a company who made online software for large companies. There were two teams, I was in charge of one, and the other was writing a CMS for a very popular newspaper.
Their team has followed all the standard textbook code examples, liberally using factories, running all the SQL queries through a small section of code that ensured connection pooling, and using small well defined requests to the DB to markup their pages. Everything was factored so the code was lovely and readable c# and performed exactly the function you'd expect. There was only one problem - when they went to speed test it on a pair of very fast (for the time) web servers, and a massive quad CPU (Xeon) database server...they got unexpectedly (for them) slow results. They called me over to help them speed it up.
I fired up the web testing software and benchmarked the main page and a couple of typical pages and the result was laughable. They could only serve up 4 pages / second on average - so, 2 pages / second / web server!
It took just a couple of minutes to figure out why. Hitting refresh on the main page with the SQL monitor open showed them making something like 30 database calls to mark up that one page. Each call added at least 50-100ms of processing time, despite how simple it was. The menu alone added something like 4 calls to the DB. most of the calls were to data that could be served up a little cold, so rather than getting them to rewrite the whole dam lot, I suggested they identify those parts and use a cacheing scheme on the returned data. That alone had them up to maybe 16 pages / sec. With a few more sensible changes to the simplistic design path they had chosen it was up to 40 pages / sec which was fast enough to ship. It really should have been capable of serving up closer to 200 - 400 pages / sec but the project was well overdue and 40 was the stated target.
A second more salient example is for another database driven website where there were a couple of queries that took a lot more resources to complete that average. The worst took 20 seconds to complete the SQL when it landed on my desk, but a few hours with query planner and some ad-hoc SQL showed it was T-SQL handling null fields very badly. In the end I could run in it perhaps 0.5 seconds.
If you can identify pages with queries like those on them, ones that slam the DB and have long execution times (bonus points for locking the tables), you can cripple most active websites far far more than pulling in some JPG files.
(Score: 5, Informative) by NCommander on Sunday October 30 2016, @10:37PM
In practice, that attack would be extremely difficult to pull off on SN. Right now, only logged in users actually touch our application servers, everything else is shunted into a varnish cache (ACs basically get 5 minute snapshots of the site at any given moment). If push came to shove, we could enable that for logged in users to increase our cache hits at the cost of updates no longer being in real time. Every taking that into account, each application server is backed by an independent mysqld instance which operates as a hot cache against NDBd (which in turn has its own hot caches). The database is small enough (since its 98% text) that between the shards, the entire thing can be stored in memory between two machines, with the option of spinning up additional DB stores as we need them. Furthermore, we always are overprovisioned; our basic rule is no single service can be at 50% average capacity, in case a node suddenly craps out on us which means we're well in excess of what we need at any given moment, and have redundancy always available.
If push came to shove, we're architected in such a way that if we need to spin up additional frontends to weather the storm, we could have them up in an hour or so counting replication, setup and reconfiguration. Same with the backend. We did a lot of code optimizations to rehash to the point as a codebase, I suspect it scales better than the other site given the same hardware.
Still always moving
(Score: 2) by blackhawk on Monday October 31 2016, @03:04AM
It sounds like SN is architected a lot better than the example I gave. To be fair, some of those things wouldn't have been an option at the time, or would have been expensive, but still - a little internal cacheing goes a long way.
Their DB was also mostly text, given it was a CMS, it's just they didn't pay attention to how they were accessing it. Most content was static enough that they could have marked up the pages and then written them to a file cache, and just kept serving them from there - invalidating that cache entry when an edit was made. Ironically, the CMS *was* capable of serving up a static copy of the site - code they had written for another client, but they never put two and two together. I think they were just too blinkered with making it all dynamic content without ever considering if it even needed to be dynamic.
They also never considered options like breaking from third normal form or using temporary tables to get the data moving faster. The lead developer was just too rigid and tunnel visioned on whatever the latest wisdom from the Microsoft bloggers happened to be.
(Score: 3, Informative) by NCommander on Monday October 31 2016, @05:03AM
Slashcode actually originally had this functionality. The database and themes were essentially exported as HTML files and served up to ACs; only accessing the .pl files directly (which happened when logged in) would give you a real-time view (this is what the original "delay in updating static page" meant when you post on the site). Due to coderot, and the fact that generating and syncing this HTML across multiple web frontends was a massive headache, we eventually just migrated everything to using varnish and put a bullet in the static content generation (most of the code is still there but disabled).
One thing that constantly drives me up a wall though is poor use of existing tools. Originally, this site used memcached as a hotcache, and standard MySQL for backend storage, which lead to an entire class of bugs where the two would disagree with each other and occasionally blow crap up. While memcached is very good at what it does, it adds a massive layer of complexity as you essentially have to map structed data to KV pairs, and deal with dirty reads. We had a lot of pain in the earlier days of the site because of this because memcached isn't a distributed datastore, the assumption is you have a central "cache" and then introduce network latency to the mix which actually tends to cause performance to go down the crapper. We got around this by running memcached on each web frontend, but the dirty read problem got a lot worse because now different parts of the stack might have different ideas on what the current state of everything was.
The entire problem got solved by moving everything to MySQL cluster which allowed for in-memory/disk-based durability for CP important data such as posts, stories, and user data. Cluster is amazing for high read performance and allows for easy multi-master operation; data is cached in the frontend mysqld cluster instances, and data is stored in the NDB instance backing it. If we needed even better performance, we'd likely migrate things like templates, statistics and such from MySQL to an AP datastore like Cassandra which is ideal for these sorts of cases where the data is non-essential, but very useful to have "at hand" (I know TMB has experimented with Redis, but I don't think it made it to production). Even without cluster, it would have been possible to use MySQL MEMORY tables as a hotcache backed to InnoDB ones to get the same effect (and we will likely have to do something similar with NDB in the future).
What always staggers me is people praising things like Postgres, but when I look at their code, its basically being used a dumb store with no concept of using FORIEGN KEYS, or stored procedures, or anything that their DB is *good* at, and instead reinventing the wheel.
Still always moving
(Score: 0) by Anonymous Coward on Monday October 31 2016, @11:19AM
Quite obviously you don't have an MBA supervising your performance, or else you'd surely not be allowed to "waste" so many resources. ;-)
(Score: 3, Informative) by NCommander on Sunday October 30 2016, @10:39PM
Oh, and to answer, we're hosted on Linode. Amazon might be better to better wither a DDoS (we've gone into degraded performance because someone blasted Linode's data centers off the net before), but we'd loosing IPv6 capability. We could also deploy a CDN or a LOT of other things before we'd run out of options to weather a storm.
Still always moving
(Score: 2) by The Mighty Buzzard on Sunday October 30 2016, @11:00PM
Heh, right now I'm pretty certain I could knock us off the net entirely with as little as half a dozen hosts. But then I know exactly what hits the DB the hardest and there's a pull request fixing it sitting on github as we speak.
My rights don't end where your fear begins.
(Score: 1, Touché) by Anonymous Coward on Monday October 31 2016, @11:28AM
Given your status as emergency reserve sysop, [soylentnews.org] I guess you could knock SN off the net using a single host — the one you'd use to log in and shut the servers down. ;-)
(Score: 2) by takyon on Tuesday November 01 2016, @02:34PM
Is my slow ass-user page getting fixed?
[SIG] 10/28/2017: Soylent Upgrade v14 [soylentnews.org]
(Score: 2) by The Mighty Buzzard on Tuesday November 01 2016, @03:13PM
No idea, have to ask paulej72. I'm on hold until he gets whatever he wants done taken care of.
My rights don't end where your fear begins.
(Score: 0) by Anonymous Coward on Monday October 31 2016, @12:02AM
Ah the 'its pure' design. Well yeah it is readable. But runs like crap.
It usually takes a mess like that to teach someone that 'yeah 50 round trip calls across the network are not good'. Cache it and aggregate it.
With one system I worked on a stored proc was digging through the same LARGE table 15 times to get different columns. One bit of caching it into a temp table and moving stuff around it went from 1.5 million reads to about 12 reads.
(Score: 2) by NCommander on Monday October 31 2016, @04:48AM
Honestly, you can get both without having a ton of crap as your codebase. Build an interface to do what you want, then slide a caching layer below it. If you've done it right, you get the best of both worlds. While not completely invisible, if you're using caching sanely, and know exactly how it operates, you can drastically reduce cache misses while rarely having to explicately code around the problem of dirty reads (MySQL cluster handles this for us). Rehash's core basically is proof; it's not the most brilliant pile of Perl ever written, but CmdrTaco actually knew what he was doing from a software development perspective. Most of the garbage we stripped out were later additions that were basically tacked on.
Still always moving
(Score: 0) by Anonymous Coward on Monday October 31 2016, @12:04AM
Uh, oh. Somebody's gonna look in my direction now.
(The only thing I don't block is star.png.)
...like browsers would generate
Depends on the browser and the config. 8-)
.
Some Soylentils may rag on the staff from time to time, but our volunteers are pretty great at keeping the site going.
...and the (first) response to Thank You, below, seems quite Funny to me--though those still in the labor force may find it a bit too close to home.
The current mod of Redundant doesn't seem apt.
-- OriginalOwner_ [soylentnews.org]
(Score: 5, Insightful) by SpockLogic on Sunday October 30 2016, @02:35PM
Thought I'd take this opportunity to say "Thank You" to all the unsung heroes who 'hampster' away behind the scenes keeping the site up and running.
Yes I know you don't do it for accolades but all the same "Ta Muchly".
Overreacting is one thing, sticking your head up your ass hoping the problem goes away is another - edIII
(Score: -1, Redundant) by Anonymous Coward on Sunday October 30 2016, @02:53PM
Now everything is working, management has decided to increase shareholder value and reduce staff costs by 33%. Thank you for making this possible. Please bring your written answers to the question "What have I personally done to increase shareholder value this quarter?" to the all-hands compulsory meeting at 9am tomorrow.
(Score: 4, Insightful) by Ethanol-fueled on Sunday October 30 2016, @04:47PM
-- Winston Churchill
(Score: 0) by Anonymous Coward on Sunday October 30 2016, @05:10PM
For some reason, I thought of pop divas that nobody's thought of for awhile (e.g. Janet Jackson), suddenly hitting the top of the Yahoo news crawl for an embarrassing "wardrobe malfunction".
(Score: 3, Funny) by archfeld on Sunday October 30 2016, @05:53PM
I blame you Ethanol, not sure why, but it just seems fitting. What have you been doing lately that would cause this ?
Did you push some old lady into traffic :) But then again maybe it was me, The NSA is investigating me regarding keyword postings and have decided this is a terrorist supporting hub...
For the NSA : Explosives, guns, assassination, conspiracy, primers, detonators, initiators, main charge, nuclear charge
(Score: 2, Funny) by Ethanol-fueled on Sunday October 30 2016, @06:22PM
If its not something Buzzard posted, then it was probably my comment about Hillary arming terrorists with Stinger missiles. It may have even been a friendly doing it as a joke.
(Score: 3, Funny) by frojack on Sunday October 30 2016, @10:23PM
Nah, it was me challenging the IOT source claims in the popular press. All 278 Alexa users decided to sic her on Soylent.
No, you are mistaken. I've always had this sig.