SoylentNews is people
SoylentNews
GitHub has been hit with the largest-ever DDoS attack, and it was only down for a few minutes:
On Wednesday, February 28, 2018 GitHub.com was unavailable from 17:21 to 17:26 UTC and intermittently unavailable from 17:26 to 17:30 UTC due to a distributed denial-of-service (DDoS) attack.
[...] Cloudflare described an amplification vector using memcached over UDP in their blog post this week, "Memcrashed - Major amplification attacks from UDP port 11211". The attack works by abusing memcached instances that are inadvertently accessible on the public internet with UDP support enabled. Spoofing of IP addresses allows memcached's responses to be targeted against another address, like ones used to serve GitHub.com, and send more data toward the target than needs to be sent by the unspoofed source. The vulnerability via misconfiguration described in the post is somewhat unique amongst that class of attacks because the amplification factor is up to 51,000, meaning that for each byte sent by the attacker, up to 51KB is sent toward the target.
[...] Between 17:21 and 17:30 UTC on February 28th we identified and mitigated a significant volumetric DDoS attack. The attack originated from over a thousand different autonomous systems (ASNs) across tens of thousands of unique endpoints. It was an amplification attack using the memcached-based approach described above that peaked at 1.35Tbps via 126.9 million packets per second.
Also at Wired and The Register.
(Score: 2) by BsAtHome on Friday March 02 2018, @01:55PM (22 children)
I still do not understand why the edges have such a hard time implementing reverse path filtering. That would make spoofing IP a lot harder to abuse. There is generally no good reason for a packet to leave your interface that has a reverse path other than your own local network(s). Most L2/3 switches have this functionality readily available and does not cost very much when performed at the edge.
(Score: 2) by c0lo on Friday March 02 2018, @02:17PM (19 children)
How would the internet route around damage if all devices implement reverse path filtering?
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 3, Interesting) by BsAtHome on Friday March 02 2018, @02:29PM (9 children)
You do not do RPF in the backbone because you have multiple routes for transit to consider. The (outer-) edge is entirely different and is not involved in generic routing transit traffic. Your tiny little server or home PC is not part of the transit-network and there is generally no need for it to send with a different source-address than its own.
(Score: 2) by c0lo on Friday March 02 2018, @02:40PM (8 children)
Who exactly are you speaking of?
The guys who left their memcached accessible? If true, how RPF would have helped?
The hackers running the reflection/amplification attack? They do know enough to disable it even if enabled.
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 2) by BsAtHome on Friday March 02 2018, @03:28PM (7 children)
The memcached servers accepted packets with a forged source-address and replied to that address. They only amplified the problem (and were badly configured to allow access in the first place). The originator (original and actual source) packets to the memcached servers should have been filtered at the *source-network* because they are *easily*detectable* forged packets.
This is no rocket science and has been best practice network 101 for a very long time. There are only /very/ rare occasions and server-setups where RPF can be problematic. Your standard "we host a website" junk-machine or your home-PC has no business sending out packets that have the wrong source-address. If your machine sends packets with foreign source-addresses, then it is most likely compromised in one or another way.
(Score: 2) by c0lo on Friday March 02 2018, @03:40PM (5 children)
Since at the origin there be haxors, possibly state-sponsored haxors, why would they filter out those packets they so carefully nurtured?
Is this some sort of a RFC 3514 joke?
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 2) by BsAtHome on Friday March 02 2018, @03:50PM (4 children)
The situation is so problematic not because of the state-sponsored attacks. Those state-sponsored attacks use *other* badly configured networks to cover their tracks!
So, lets get rid of those other badly configured networks first. Then we can _see_ where the original attack came from and point a finger at them!
(Score: 2) by vux984 on Friday March 02 2018, @06:13PM (3 children)
I think I follow your argument. I think you are right.
The final leg of the attack is packets arrive at vulnerable memcached, spoofed to appear to come from github. The vulnerable memcached then hits gitshub with a bunch of garbage.
If the packet with the spoofed origin came directly from china, there's probably nothing that can be done to stop it. (at least not with RPF) given China is going out of its way to allow the spoofed packets to exit their networks, and once they're out onto the backbone there's no easy way to know they didn't come from where they claim to have come from.
However, more often than not, the packet with spoofed github origin will not be coming from China, but rather from a compromised security camera at Jane Q Public's house in Montana or Jorge's Corner store in Mexico, etc. These are not hostile state actors, and the ISPs in these locations there can should be filtering packets originating at their customer endpoints to prevent them from participating in such amplification attacks. It doesn't solve the problem, but it would dramatically dilute the scale.
(Score: 1, Interesting) by Anonymous Coward on Saturday March 03 2018, @05:52AM
I asked a friend of my brother's who is a network admin at my ISP about that once at a family function. Our exchange went roughly as so:
Me: I was reading that an good way to help prevent web attacks is if ISPs filter packets with a source address that didn't originate in their network.
Him: I suppose it would but we don't have the resources to analyze every packet that goes across our network.
Me: But don't you already analyze every packet? I mean, you already do to filter all the incoming packets directed at certain ports (80, 443, 137-139, and some others), all outgoing packets based on what their destination is, and every packet in both directions for size in order to charge them against my data cap.
Him: Yeah, but this is different because... uh... common carrier? If we messed with traffic? ... and... uh... Say is that your mother? (walks away)
(Score: 2) by maxwell demon on Saturday March 03 2018, @02:03PM (1 child)
But wouldn't Jane's security camera send the packet through Jane's home router, whose NAT would replace the origin IP with the IP given to her by her ISP anyway?
The Tao of math: The numbers you can count are not the real numbers.
(Score: 2) by vux984 on Saturday March 03 2018, @07:50PM
The short answer to your question is yes. the NAT router would replace the spoofed source ip address with the nat's public ip.
But what happens when the router gets its response? It sounds like, in some cases the nat box will drop the packet because the sourceip isn't a network it routes for, but apparently quite a lot will actually attempt to deliver it to the original spoofed address (ie now send it out to the default gateway addressed to the the spoofed ip.)
At least if this is to be beleived:
https://security.stackexchange.com/questions/66231/what-happens-when-a-spoofed-source-ip-packet-is-dispatched-out-of-a-private-netw [stackexchange.com]
(Score: 2) by FatPhil on Saturday March 03 2018, @09:26AM
Yes, but *not* doing it is absolutely zero cost, unless you're the victim of one, in which case someone else is to blame. If you don't benefit from doing things the right way, then why would you bother?
It should be a default, to be honest. Every device that knows that it bridges between a small subnet (because it serves DHCP to them, for example, and has a routing table to match) and the wider internet should just do the obvious.
Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
(Score: 3, Insightful) by FatPhil on Friday March 02 2018, @02:30PM (8 children)
Reverse path filtering is about deciding whether a packet you've recieved is sporged.
They're unrelated concepts apart from the aspect that they happen on the internet.
Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
(Score: 2) by c0lo on Friday March 02 2018, @02:45PM (7 children)
You receive an IP packet which passed to 15 routers before reaching you, with the source IP (reply address) being sporged - how would you know it was sporged?
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 2) by VLM on Friday March 02 2018, @03:15PM (3 children)
As the g-g-great-grandpost specified, why don't the edges work properly, and the answer as always is its cheap and lazy.
If your cablemodem doesn't sniff DHCP traffic and try to dynamically insert firewall rules to only forward legit src addrs, then it can not have bugs related to that code nor can it be late to market due to dev and testing time nor will it get blamed for being in a DDOS attack because every other piece of edge gear is trash so its not any worse and its all the big bad hacker's fault anyway plus or minus some victim shaming.
So to circularize the argument and answer g-g-grand OPs post, the edge sux (doesn't block bad traffic) because the edge sux (its gotta be cheap and no one cares if it works "right" as long as the customers pay their bills).
(Score: 2) by c0lo on Friday March 02 2018, @03:25PM
Hang on! You or the OP imply that the haxors would be stopped by their cablemodem?
Especially the gov-sponsored chinese haxors?
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 3, Interesting) by BsAtHome on Friday March 02 2018, @03:40PM (1 child)
The cable-modem is normally not involved in RPF because of dynamic IPs. Therefore, the first convenient place to enable RPF is the router (often L2/3 switch) at the ISP side (the one the cable-modem connects to). There you simply filter the local network segment as valid source.
As a network provider you know where your IP segments are and it is a matter of standard procedure to do proper setup at the right place. Yes, there are a lot who don't care, but that does not mean that we cannot reduce the problem quite easily. This can additionally be solved by ingress/egress verification at the BGP border. Any egress out of the first AS hop is always determinate (which equals the ingress of the second AS).
(Score: 2) by VLM on Friday March 02 2018, @08:58PM
Yes thats the longer more professional answer. With a side dish that adding filtering creates a possible failure mode where you move an IP segment from here to there and forget to update the filtering leading to mysterious phone call complaints from end users.
IPv6 will fix everything when you can allocate such giant chunks of space you'll never wiggle /25 from here to there to make things fit.
(Score: 2) by FatPhil on Friday March 02 2018, @03:37PM
If you're an edge router, that's a very easy question to answer, and if the answer's "no", you don't let it out.
If you're backbone, it's also very easy to answer, the answer's very unlikely to be "no".
Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
(Score: 2) by bob_super on Friday March 02 2018, @05:56PM
You could imagine a protocol where the first packets you receive from a brand-new return IP range get challenged before being forwarded. If the sender pretending to be at that address doesn't respond to a challenge, you discard. The challenge has to be a small packet, on the edge of the network (a tiny ping), to avoid adding congestion.
That wouldn't stop state-sponsored attacks, but script kiddies playing with Memcached would be limited to bugging their neighbors.
(Score: 2) by sjames on Friday March 02 2018, @06:52PM
The idea is for the ISP that owns the edge routers to do the egress filtering. And if they don't, de-peer them.
The closer you get to the core, the harder it is to get right. Right in the core, the list of plausible addresses is nearly the entire address space, so it's not feasible there.
(Score: 3, Interesting) by zocalo on Friday March 02 2018, @03:11PM (1 child)
UNIX? They're not even circumcised! Savages!
(Score: 3, Insightful) by VLM on Friday March 02 2018, @03:20PM
I worked in that cutthroat line of business a long time ago and if you do the right thing, that'll cost time and money, and you'll get replaced in the market by people who save time and money by not bothering.