https://motherboard.vice.com/read/a-loud-sound-just-shut-down-a-banks-data-center-for-10-hours
ING Bank's main data center in Bucharest, Romania, was severely damaged over the weekend during a fire extinguishing test. In what is a very rare but known phenomenon, it was the loud sound of inert gas being released that destroyed dozens of hard drives. The site is currently offline and the bank relies solely on its backup data center, located within a couple of miles' proximity.
"The drill went as designed, but we had collateral damage", ING's spokeswoman in Romania told me, confirming the inert gas issue. Local clients were unable to use debit cards and to perform online banking operations on Saturday between 1PM and 11PM because of the test. "Our team is investigating the incident," she said.
The purpose of the drill was to see how the data center's fire suppression system worked. Data centers typically rely on inert gas to protect the equipment in the event of a fire, as the substance does not chemically damage electronics, and the gas only slightly decreases the temperature within the data center.
(Score: 2) by quietus on Monday September 12 2016, @04:23PM
The Bank said it required 10 hours to restart its operation due to the magnitude and the complexity of the damage. A cold start of the systems in the disaster recovery site was needed.
Run that by me again? Either these guys had a split-brain situation, or they had no L2 datacenter redundancy -- xconnects, VPLS are around since 2007/2009 -- or somebody just forgot to synchronize the security certificates on their exterior facing firewall cluster (and somebody else panicked).
(Score: 1) by nitehawk214 on Monday September 12 2016, @06:16PM
Seriously. Completely off backup site? No way. Maybe the article writer just did not understand what "cold start" meant and decided to use a word that sounds technical.
"Don't you ever miss the days when you used to be nostalgic?" -Loiosh