IBM has blamed a supplier for causing the failure of Australia's online census, which went offline on the very night millions of households were required to describe their disposition.
Big Blue's submission (PDF) to Australia's Standing Committees on Economics, which is conducting an Inquiry into the Preparation, Administration and Management of the 2016 Census by The Australian Bureau of Statistics puts the blame for the failure at the feet of a company called NextGen Networks.
IBM does so because it says it devised a distributed denial of service (DDoS) prevention plan called "Island Australia" that involved "blocking or diverting international traffic intended for the eCensus site before it reaches the site, while leaving the system free to continue to process domestic traffic."
"This method was chosen because the primary risk of DDoS attacks of sufficient size to disrupt site availability was considered to be from foreign sources."
IBM's submission says two carriers were chosen to bring traffic to the Census site, Telstra and NextGen. Both were informed about "Island Australia" and how to implement it. But come Census day, IBM says "a Singapore link operated by one of NextGen's upstream suppliers (Vocus Communications or Vocus) had not been closed off and this was the route through which the attack traffic had entered the NextGen link to the eCensus site."
Big Blue's document says Vocus 'fessed up to the error on Census night.
[...] In a delicious irony, NextGen's submission also notes its recent acquisition by none other than Vocus. Which will make life interesting at the first all-hands meeting once the acquisition closes.
The Inquiry will issue a report on November 24th. The Register's Australian outpost has laid in copious stocks of popcorn ahead of the report's release.
Previous reporting:
Australian Census: Hacked or Just Ill-Prepared?
Related Stories
Australian Census Attacked by Hackers
The Australian census website was shut down by what authorities said was a series of deliberate attacks from overseas hackers.
Millions of Australians were prevented from taking part in the national survey on Tuesday night. The Australian Bureau of Statistics (ABS) had boasted only hours before that its website would not crash.
The prime minister assured the public that their personal information was not compromised. Debate about privacy concerns has been raised despite assurances from the government that security would not be compromised. Prime Minister Malcolm Turnbull said that public's personal information was safe and and stressed the "unblemished record" of the ABS.
"The one thing that is absolutely crystal clear is that there was no penetration of the ABS website," Mr Turnbull said.
"What you saw was the denial of service attack or a denial of service attempt which, as you know, is designed to prevent access to a website as opposed to getting into the server behind it. Some of those defences failed, frankly."
[Continues...]
(Score: 3, Informative) by RamiK on Saturday October 22 2016, @09:11AM
They should have had their IP range in black lists days in advance and test the international availability on their own.
Besides, Australia's ISPs usually don't shape traffic ( https://wiki.vuze.com/w/Bad_ISPs#Australia [vuze.com] ) and don't have the capabilities (hardware or personal) for that sort of work. They intentionally kept much of their equipment dumb to avoid government log requests so doing this sort of filtering could mean sending technicians to physically access each switch's serial port with a laptop and input the ip range.
compiling...
(Score: 2, Informative) by Anonymous Coward on Saturday October 22 2016, @10:23AM
(Score: 2) by RamiK on Saturday October 22 2016, @02:49PM
First off, you're right that list isn't good. I wanted to paste the link to the response the ISPs made to a data request a few month ago (not sure if it was ACCC or ACMA or whatnot) but couldn't find it.
Either-way, I WAS talking about first mile ISPs. I figured it's obvious enough considering international interconnect was what the article was talking about and even specified the those companies.
That said, last I heard, they don't deploy layer 2+ switches unless specifically asked by customers* to avoid legal requirements and technical complications. That's to say, they don't go into the packets. They only rate limit.
Though admittedly, I don't have any way to verify what they were saying... And I'm guessing it's plan dependent regardless.
* And specifically just the business \ cloud customers and usually not the consumers ISPs. Vocus even has a special DDOS protection plan that is based on rate limits rather then packet analysis.
compiling...
(Score: 2) by driverless on Sunday October 23 2016, @12:16PM
It was the dog! The dog did it, I swear! I had my homework all done but then the dog ate it! It's not my fault!
(Score: 2, Interesting) by bug1 on Saturday October 22 2016, @01:48PM
"Vocus does not agree that the fourth DDoS attack was the cause of the site becoming unresponsive. The fourth attack comprised of attack traffic which peaked at 563Mbps which is not considered significant in the industry, and lasted 14 minutes. See report from Arbor Networks which indicates that it is materially below the mean attack size (https://www.arbornetworks.com/images/documents/WISR2016_EN_Web.pdf)."
"The cause of the census website being unreachable was IBM employee’s falsely identifying normal traffic patterns as data exfiltration, and manually turning off their Internet gateway routers"
"Vocus was not, at any time prior to 9 August 2016, invited to participate in any testing of IBM's DDoS mitigation strategy, or given any detail of what testing was undertaken. In fact, Vocus was not informed of IBM’s DDoS mitigation strategy, Island Australia or its specific requirements, until after the fourth attack."
http://www.aph.gov.au/DocumentStore.ashx?id=9f58f27a-9b42-4de1-80f7-60e17edfb153&subId=414847 [aph.gov.au]
(Score: 2) by Mykl on Sunday October 23 2016, @03:52AM
While all of the players involved are still claiming that it was a DDOS that brought about #CensusFail, the reality is much simpler.
IBM's Stress and Volume testing assumed a uniform distribution of Census entries [smh.com.au] over a 12+ hour period, when in reality most people jumped on to do the census 1-2 hours after dinner. So the DDOS was really from the unsuspecting user base.