The Register reports a hack, speculated to be intentional instead of the usual finger fumble, whereby all of Google's traffic was routed for just over an hour to servers in Russia and China.
The Register story: https://www.theregister.co.uk/2018/11/13/google_russia_routing/.
It quotes this update from Google: https://status.cloud.google.com/incident/cloud-networking/18018#18018002
Excerpt from the update:
The issue with Google Cloud IP addresses being erroneously advertised by internet service providers other than Google has been resolved for all affected users as of 14:35 US/Pacific. Throughout the duration of this issue Google services were operating as expected and we believe the root cause of the issue was external to Google. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to help prevent or minimize future recurrence.
As BGP is "broken by design", i.e. assumes trust where there is no longer any, what is perhaps surprising is that it took so long to happen. Does not augur well.
So much for "the internet always routes around damage". Maybe "always" takes time to happen...
Exercise for the reader: is it possible to circumvent this effectively, and if so, how? Has my paranoia-meter misfired, and there's really nothing to worry about?
(Score: 2) by Corelli's A on Tuesday November 13 2018, @06:42AM (2 children)
Mitigation for this problem is straightforward: filter announcements at the edge according to contractually-specified ASNs. Non-edge AS boundaries might be able to filter but in any case ought to be able to detect post-facto and apply financial pressure. This seems to be a political problem, not a technical problem.
(Score: 2) by ledow on Tuesday November 13 2018, @08:15AM (1 child)
Yes, but for a user, there is literally nothing they can do. They are entirely bound by every hop along their route doing this, doing it properly, and never making a mistake or working against them.
Which is a stupid scenario for something so important and so based on random trust.
BGP.
SMTP.
Two huge, unsolved problems that really need to go away forever because they are destroying all semblance of privacy or security for users.
(Score: 0) by Anonymous Coward on Wednesday November 14 2018, @03:38AM
I suppose the best mitigation is using HTTPS Everywhere and HSTS and hoping the website is on one list or the other. With DNSSEC in place and other restrictions like DANE or CAA or key pinning, your attacker would have to jump through much more hoops, some of which might trip automated monitoring, in order to succeed with their attack.
(Score: 1, Insightful) by Anonymous Coward on Tuesday November 13 2018, @07:53AM (2 children)
Don't use google.
(Score: 0) by Anonymous Coward on Tuesday November 13 2018, @10:43AM (1 child)
Yea, is it any worse that the chinese and russian govs got this data instead of google and "five eyes"?
(Score: 4, Insightful) by The Mighty Buzzard on Tuesday November 13 2018, @11:10AM
A lot less bad unless you happen to live in China or Russia, really. They don't have the kind of leverage on western citizens that their own governments can bring to bear.
My rights don't end where your fear begins.
(Score: 1, Insightful) by Anonymous Coward on Tuesday November 13 2018, @10:42AM (1 child)
just like peace: without trust and co-operation, the internet is not possible.
thus the calculation starts if war or peace is more profitable and if the canary in the mine loses its petals?
p.s. i agree with previous poster that some internet stuff is ancient and needs to be replaced. however it is curious, since the internet was/is open that nothing new came along. afterall below port 1024 there are many "protocols" and they all use either tcp, udp or icmp.
during all this time it seems only one protocol was really added and didnt even get a specially assigned port: i am looking at you, torrent, the embodiment of decentralization ...
(Score: 2) by HiThere on Tuesday November 13 2018, @06:13PM
You need to use a bit of systems analysis. If you'd said "trust, but verify" then you'd have a point.
The internet is not just one set of protocols. Some of the ones in use are inappropriate where you can't trust all the players, and a globally distributed system can't trust all the players, so it needs to avoid the protocols that depend on trust being valid. Secure protocols will almost always be either slower, more expensive to use, or both, so there's a case for avoiding them in LANs...but the overhead isn't that high, so it usually won't be a strong case.
Javascript is what you use to allow unknown third parties to run software you have no idea about on your computer.
(Score: 3, Interesting) by The Mighty Buzzard on Tuesday November 13 2018, @11:13AM
Don't assume the word "instantly" into that phrase. Sometimes it's near instant, mostly it's not.
My rights don't end where your fear begins.
(Score: 2) by Bot on Tuesday November 13 2018, @11:14AM
see, e.g. how https://secushare.org/broken-internet [secushare.org] considers BGP...
Account abandoned.
(Score: 1, Interesting) by Anonymous Coward on Tuesday November 13 2018, @12:10PM (2 children)
There are implementations of secure BGP but the will to adopt it, is the biggest obstacle. The ones criying china is rerouting traffic are the same who take advantage of the weakness to reroute adversary traffic, yet another case of keetle call pot black. This is not a technical inpediment and there is no sign of adopting measures to correct this in the near future.
(Score: 1, Interesting) by Anonymous Coward on Tuesday November 13 2018, @03:48PM (1 child)
"but the will to adopt it, is the biggest obstacle"
I wouldn't assume that "will" has anything to do with it. Routing tables are huge processing tasks. Using crypto to authenticate a route is probably several orders of magnitude more CPU load. IOW, you'd need to re-engineer the whole router, and it would still probably converge routes like a pig. And slow convergence creates other problems that can introduce cascade failure modes and Denial of Service attack vectors.
IOW, authentication systems tend not to scale at the rate required to authenticate a full BGP4 routing table on 100k nodes. While somebody may have written the software, making it scale is an entirely different problem.
(Score: 2) by HiThere on Tuesday November 13 2018, @06:18PM
That's a valid point, but wouldn't it be likely to apply to any replacement? Perhaps there could be classes of message with different security requirements, from ROT13 to secure against quantum computers, so only the most sensitive messages would need strong crypto...of course, that singles out just which ones are sensitive.
Javascript is what you use to allow unknown third parties to run software you have no idea about on your computer.
(Score: 1, Informative) by Anonymous Coward on Tuesday November 13 2018, @03:28PM (1 child)
If only one side was at fault, it should have resulted in service disruption. But in this case, the Russians would have been advertising, BUT, the Americans would have been failing to filter that advertisement. So there are actually two failures here.
I recall many moons ago getting a phone call about a false route advertisement. The caller didn't understand what was happening, and once I figured it out, I got the owner of the the offending AS on the phone and bridged him in. Turns out they were competitors and doing it intentionally. It turned into quite a shouting match. This sort of thing is rare, but it does happen. But that was before routing registries and IPv6.
Remember the guy sending the route advertisement is the guy recieving the traffic. So the fault actually resides with the U.S. carrier, not the Russians. You can't count on foreign advertisements to be compliance with route registries, which is why you always filter incoming BGP4 advertisments.
Somebody on the U.S. end fat fingered a route filter, which allowed Russia to be the shorter route. The traffic trunked onto the wrong peer, and bobs your uncle. This shit happens all the time. Usually the middle manager of the side getting pounding with a massive overload, notices and starts screaming like a little girl without knowing that it takes two to create this kind of fuckup.
Of course I'm sure that it won't take long for the news and the FBI to start screaming it was an act of espionage, because some newsy who doesn't understand it claims it is. Congress will issue a law that all people with fat fingers are predisposed to be a commies and we should all be afraid of them. SCOTUS will confirm that the bill of rights really doesn't apply to fat fingers, because they aren't really "people" according to the meaning of the preamble. Lynchings will resume, and all of us fat fingered engineerers will have to leave the country or get sent to gulags.
God damned fat fingered pinko communists. /sarc
SSDD.
(Score: 0) by Anonymous Coward on Tuesday November 13 2018, @03:31PM
Side note:
It could have also been caused by a major outage. Two or three routers crash at a peering point, and maybe Russia is the only remaining route? Could happen in the event of an electrical problem or HVAC problem. It is entirely possible that this was working exactly the way it was SUPPOSED to work.
(Score: 4, Interesting) by Hyperturtle on Tuesday November 13 2018, @04:45PM
"what is perhaps surprising is that it took so long to happen. Does not augur well."
It hasn't augured well for a long time. Here's a list previous auguation situations, from just a quick search. This is not the first time; for those not following this regularly, the list below is from some of the bigger moments in rerouting history.
https://www.wired.com/2014/08/isp-bitcoin-theft/ [wired.com]
https://csecybsec.com/cse-news/experts-detailed-how-china-telecom-used-bgp-hijacking-to-redirect-traffic-worldwide/ [csecybsec.com]
https://securityaffairs.co/wordpress/66838/hacking/bgp-hijacking-russia.html [securityaffairs.co]
http://securityaffairs.co/wordpress/62409/hacking/google-mistakeinternet-outage-japan.html [securityaffairs.co]
https://bgpmon.net/googles-services-redirected-to-romania-and-austria/ [bgpmon.net]
http://germany.timesofnews.com/bgp-attacks-hijack-telegram-traffic-in-iran [timesofnews.com]
Those are not in chronological order, but this has been a problem since before Google. It's just that Google is such an attractive target...
Within the past 5 years it's happened yearly, often to Google. Usually that 8.8.8.8 IP is redirected to some wooden shack on a remote island and then the vans drive off to the airfield and docks with their theoretical bandwidth just as the problems are resolved. It's not just limited to Google nor is it always intentional.
Most of the abuses witnessed are due to a lack of expertise or controls at the ISP; you have to trust your peers, but you can also reject routes you don't expect from them. That can prevent a lot of damage if you only accept what you are supposed to get... it keeps clients from advertising the wrong networks, by design or by accident.
If the redirect comes from a major player, then yes it may be infeasible to filter routes on a giant backbone; redundancy also can mean routes change between providers or links, etc. It can be done, but takes time and money. Often, small regional ISPs are to blame--they do not have time and money to trust but verify their connections and enforce policies reducing the impact of mistakes or bad intent. They just permit any any, participate in the routing exchanges and let the internet sort it out further up or down stream. Sometimes taking action accepts liability, and that is often something to avoid.
It's also harder to defend against some issues... the bad actors are either highly motivated or state-level actors with significant backing, and small regional ISPs don't have much chance to stop anything like that even with time and money.
(Score: 0) by Anonymous Coward on Wednesday November 14 2018, @12:16PM (1 child)
According this article [reuters.com]. This was caused by a network equipment upgrade misconfiguration in Niggeria. Which I don't believe for one second.
(Score: 0) by Anonymous Coward on Wednesday November 14 2018, @02:15PM
This is entirely possible. If the Nigerians send a more specific prefix, and there is disagreement between carriers about the minimum size of advertised blocks, then you have a situation where the smaller block gets passed around to a few small localities on the Internet. Note that this has to do with the number of AS's traversed not the number of routers or geographical distance traversed.
Journalists, politicians and the MIC: "Aaaaaah Russia, China! Commie bastards, drop nukes on those motherfuckers!"
Engineer: "Fuck. I knew when I dropped an Oreo crumb in my keyboard it was going to cause trouble sometime down the road."
Security Blowhards: "Something needs to be done about this, there needs to be more consulting and expensive useless software contracts!"
There are physics problems that dictate why the algo's work the way they do. If there is a better way, there is a lot of money in finding it, yet it hasn't been found. Not that this can be expected to keep people from exploiting public paranoia. In IT we don't often call out the digital version of flat earthers. That is because they outnumber us and are predisposed to violence.
SSDD
(Score: 2) by hendrikboom on Friday November 16 2018, @01:01AM
Send probes randomly to various locations in the world and see how many of them arrive? Check for arrival in the proper place with cryptographic authentication. Might catch large-scale rerouting within a reasonable time. Might require a few secure channels to the distant places you're testing, but inability to set up those secure channels might itself indicate trouble to be investigated.