Another Monday and another BGP (Border Gateway Protocol) misconfiguration causing large parts of the Internet to stop working. More on the cause and effects from the Cloudflare blog How Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline:
[Monday] at 10:30UTC, the Internet had a small heart attack. A small company in Northern Pennsylvania became a preferred path of many Internet routes through Verizon (AS701), a major Internet transit provider. This was the equivalent of Waze routing an entire freeway down a neighborhood street — resulting in many websites on Cloudflare, and many other providers, to be unavailable from large parts of the Internet. This should never have happened because Verizon should never have forwarded those routes to the rest of the Internet. To understand why, read on.
SoylentNews was also affected — alongside other prominent sites.
Related Stories
Also, Linode is planning some server reboots over the next week or so. We will try to give advance notice and keep downtime to a minimum.
Update: Everything seems to have quieted down. Many many thanks to NotSanguine for jumping in and lending his expertise to help identify and isolate where things were borked.
Indications are that a bad BGP (Border Gateway Protocol) route was published causing a relatively small AS (Autonomous System) to have all traffic to/from a large fraction of the internet attempt to go through its routers.
(Score: 5, Informative) by NotSanguine on Tuesday June 25 2019, @06:07PM (3 children)
Verizon definitely screwed the pooch here, in that they apparently hadn't implemented appropriate route filters on a new link that had just gone live.
As much as we (and myself included, Verizon/Bell Atlantic/NYNEX/NY Telephone have sucked for *decades*) love to pile on Verizon, they made a mistake. These are actual humans, and humans make mistakes.
However, AFAIK, Verizon has yet to acknowledge the issue, nor did they have any role in resolving it. There's a pretty good discussion about that over at Hacker News [ycombinator.com].
It's pretty telling that the only response from Verizon that I've seen is over at El Reg [theregister.co.uk], where they quote a response from Verizon:
Given that Verizon didn't actually do *anything* to resolve the issue makes their response even more laughable, given that "some FIOS customers" included (by Cloudflare's estimates) something like 15% of global internet traffic.
How did that old saw go again? "We're the phone company. We don't have to care." I guess that attitude is still alive and well at Verizon.
More's the pity.
No, no, you're not thinking; you're just being logical. --Niels Bohr
(Score: 1) by fustakrakich on Tuesday June 25 2019, @07:46PM
Pretty sad that we give one company so much power over the entire WAN.
A complete redesign is in order to make the service providers into a switching network. Take away their routers.
La politica e i criminali sono la stessa cosa..
(Score: 3, Insightful) by Freeman on Tuesday June 25 2019, @10:02PM (1 child)
That kind of response from Verizon is why pretty much no one likes any of the phone companies. That kind of attitude will guarantee their demise, if / when something comes up that competes for their customers. I for one sure have hopes that Starlink will be part of that answer. Since, Starlink seems to be on track for production deployment.
Joshua 1:9 "Be strong and of a good courage; be not afraid, neither be thou dismayed: for the Lord thy God is with thee"
(Score: 0) by Anonymous Coward on Tuesday June 25 2019, @10:50PM
They suck, slightly, less than AT&T.
MAGA, baby, MAGA.