Stories
Slash Boxes
Comments

SoylentNews is people

posted by CoolHand on Monday September 12 2016, @01:32PM   Printer-friendly
from the booms-and-bangs dept.

https://motherboard.vice.com/read/a-loud-sound-just-shut-down-a-banks-data-center-for-10-hours

ING Bank's main data center in Bucharest, Romania, was severely damaged over the weekend during a fire extinguishing test. In what is a very rare but known phenomenon, it was the loud sound of inert gas being released that destroyed dozens of hard drives. The site is currently offline and the bank relies solely on its backup data center, located within a couple of miles' proximity.

"The drill went as designed, but we had collateral damage", ING's spokeswoman in Romania told me, confirming the inert gas issue. Local clients were unable to use debit cards and to perform online banking operations on Saturday between 1PM and 11PM because of the test. "Our team is investigating the incident," she said.

The purpose of the drill was to see how the data center's fire suppression system worked. Data centers typically rely on inert gas to protect the equipment in the event of a fire, as the substance does not chemically damage electronics, and the gas only slightly decreases the temperature within the data center.


Original Submission

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 4, Funny) by weeds on Monday September 12 2016, @02:14PM

    by weeds (611) on Monday September 12 2016, @02:14PM (#400713) Journal

    I experienced a Halon dump test once (so this is truly anecdotal.) The dump was done from the ceiling, nothing under the floor or anywhere else.

    The test was done with the full staff in the room. Doing it this way increases the concentration of the Halon in the room (displacement) and makes the test more likely to pass.

    All of the systems were running. (This will get you a laugh - Perkin Elmer 32 bit systems - 3220 and 3240 with CDC 9766 Drives - 300 MB!)
    None of the equipment was damaged and these drives were pretty sensitive. The only negative was the gunk that came out of the pipes and made some unsightly marks on the ceiling. Staff enjoyed a few minutes of having voices like James Earl Jones.

    Starting Score:    1  point
    Moderation   +2  
       Interesting=1, Funny=1, Total=2
    Extra 'Funny' Modifier   0  
    Karma-Bonus Modifier   +1  

    Total Score:   4  
  • (Score: 1) by Delwin on Monday September 12 2016, @04:23PM

    by Delwin (4554) on Monday September 12 2016, @04:23PM (#400786)

    You must be talking about Helium, not Halon. Halon is heaver than air. It will also kill you quite quickly.

    • (Score: 1, Insightful) by Anonymous Coward on Monday September 12 2016, @04:35PM

      by Anonymous Coward on Monday September 12 2016, @04:35PM (#400792)

      Considering the voice like James Earl Jones part, I doubt he was referring to helium.

    • (Score: 2) by bob_super on Monday September 12 2016, @05:21PM

      by bob_super (1357) on Monday September 12 2016, @05:21PM (#400817)

      Helium rises, which is often counterproductive when trying to displace O2 to stop a fire.
      Regardless of which gas, and considering equipment racks are taller than me, a properly designed system should be killing humans if it's doing its job. If you're a basketball player, you may get to die from the fire smoke rather than from the fire suppressant.

      • (Score: 3, Informative) by VLM on Monday September 12 2016, @07:37PM

        by VLM (445) on Monday September 12 2016, @07:37PM (#400891)

        a properly designed system should be killing humans if it's doing its job.

        Rather pessimistic. Assuming good mixing you'll get high as a kite via the usual solvent inhalant effect when its pretty effective at stopping fire (a couple percent)

        Of course you don't set a system off for fun, usually, implying there's a killer fire, so you get the heck out because of the fire not necessarily the halon or halon replacement.

        A good engineering rule of thumb is half the atmosphere is below 18kft and 18kft mountains or aircraft flying is no laughing matter for random sick civilians but its hardly guaranteed death and that's 50% depletion of O2 via 50% halon in the air.

        Probably 5% or so in the air might put out a fire if in some miracle it were perfectly mixed, 10% or so will make you high as a kite and lay down and die in the fire (I mean, there is a fire, after all), and nobody spends enough money to put way more than 50% in the air. I mean, even before it was banned, it wasn't cheap.

        Like I wrote I've worked at sites that had a halon dump, nobody dies although I heard its utterly terrifying and the mess was impressive.

        Of course, 50% of O2 displaced from air can't kill healthy hiker me, but if I huffed lungfulls of smoke to cause smoke inhalation problems (because you're doing a halon dump because of the fire...) then maybe 50% halon could kill a no longer healthy dude. Like 50:50 halon and plastic smoke probably wouldn't be good for you even if 50:50 halon and air would be fine. Then again at 50% plastic smoke intake you're probably dead already. Hmm.

      • (Score: 2) by sjames on Monday September 12 2016, @09:42PM

        by sjames (2882) on Monday September 12 2016, @09:42PM (#400951) Journal

        No. It should not be killing humans. Some places use a reduced oxygen environment (15% rather than the usual 20) to prevent fire. It's perfectly safe for any reasonably healthy person to be in the room for a while as long as they don't do heavy labor.

        • (Score: 2) by bob_super on Monday September 12 2016, @09:49PM

          by bob_super (1357) on Monday September 12 2016, @09:49PM (#400955)

          One could easily argue that being in a server room during an unexpected halon release, especially if there's a real fire, would be a stress level that qualifies as hard labor.
          (I did give VLM a +1 for his detailed response. I thought the O2 displacement was higher than 5%)

    • (Score: 2) by weeds on Monday September 12 2016, @08:01PM

      by weeds (611) on Monday September 12 2016, @08:01PM (#400903) Journal

      Halon - Not at 100% concentration, just enough to put out a fire. That won't kill you. I was in the room too.
      Denser than air = lower voice.

    • (Score: 2) by tangomargarine on Monday September 12 2016, @08:43PM

      by tangomargarine (667) on Monday September 12 2016, @08:43PM (#400914)

      Supposedly halon extinguishes fires at a low enough concentration that you can stay in the room for at least some amount of time and still be okay. Not that you'd probably *want* to, but yeah.

      --
      "Is that really true?" "I just spent the last hour telling you to think for yourself! Didn't you hear anything I said?"
  • (Score: 3, Informative) by Hyperturtle on Monday September 12 2016, @08:49PM

    by Hyperturtle (2824) on Monday September 12 2016, @08:49PM (#400918)

    I am seriously doubting it was Halon in the dump of 'inert gas', and wonder about your co-workers.

    Halon is a toxic chemical and is a central nervous system irritant. Wikipedia has all the relevant details.

    https://en.wikipedia.org/wiki/Bromotrifluoromethane [wikipedia.org]

    I would expect the gas to be Inergen instead -- also in canisters, also a pain clean up after. All of comments of the mess the gas makes seem to be more in line with Inergen from what I understand -- I've been through a few of those.

    Every incident of its deployment involved user error. One time from the company testing the smoke detectors; the guy in one case was throwing his full body weight against a pair of double doors that were trying to open from the pressure--security doors (like fire doors with badge access to a datacenter types of doors) blown off their hinges. The guy was bruised from head to toe. And the mainframe printer room -- it was like a tornado got in there and threw all the toner (which was in giant multi gallon bottles) into the air and all over the place. Tractor feed and laser printer paper all over the place as well. The print operators were pretty traumatized as there was no warning.

    Not only did the place go down, but on top of the service disruption, it costs hundreds of thousands of dollars; one time it was caused by the company providing the Inergen gas. It cost thousands just to recertify the system -- tens of thousands. Remember, stuff that sat unused for years suddenly had a huge gas discharge blown through at high pressures and blew the caps off things and cracked other stuff.

    Then another time a contractual foreign national that followed in the badge swipe of another person from the same firm had never seen the red button on the wall before, and pressed it. There is no un-press the button feature--there is no bypass like how there is with an actual smoke or fire alarm. The system presumes if you lifted the shield and pressed the button, you mean business. Again, six digits of damage. Another time someone didn't think a cigarette would set it off.

    The gas has to be replaced, and the system it connects to has to be inspected and certified.

    Then everything the gas gets sucked into or condenses on -- servers, fans, air filters, vent grilles, the space under raised floors... all gets covered in dust and much and stuff that got blown around after years of being still.

    The fact that disk drives now are now too sensitive to withstand this sort of air pressure (which can tear doors off their hinges) tells me that the damages nowadays are much higher...

    • (Score: 4, Informative) by tangomargarine on Monday September 12 2016, @09:11PM

      by tangomargarine (667) on Monday September 12 2016, @09:11PM (#400930)

      For example, Halon 1301 total flooding systems are typically used at concentrations no higher than 7% by volume in air, and can suppress many fires at 2.9% v/v. By contrast, carbon dioxide fire suppression flood systems operate from 34% concentration by volume (surface-only combustion of liquid fuels) up to 75% (dust traps). Carbon dioxide can cause severe distress at concentrations of 3–6%, and has caused death by respiratory paralysis in a few minutes at 10% concentration. Halon 1301 causes only slight giddiness at its effective concentration of 5%, and even at 15% those exposed remain conscious but impaired and suffer no long-term effects. (Experimental animals have also been exposed to 2% concentrations of Halon 1301 for 30 hours per week for 4 months, with no discernible health effects.) Halon 1211 also has low toxicity, although it is more toxic than Halon 1301, and thus considered unsuitable for flooding systems.

      However, Halon 1301 fire suppression is not completely non-toxic; very high temperature flame, or contact with red-hot metal, can cause decomposition of Halon 1301 to toxic byproducts.

      https://en.wikipedia.org/wiki/Halomethane#Fire_extinguishing [wikipedia.org]

      --
      "Is that really true?" "I just spent the last hour telling you to think for yourself! Didn't you hear anything I said?"
    • (Score: 2) by weeds on Tuesday September 13 2016, @02:55PM

      by weeds (611) on Tuesday September 13 2016, @02:55PM (#401320) Journal

      Well, my friend, you can doubt it all you want.
      I spec'd the system, I selected the system, I purchased the system, I supervised the installation of the system, and I was in the room for the test.
      Those cylinders that said HALON on them were connected and I had the pleasure of sounding like Michael Clarke Duncan for the few minutes we were in there.
      As you may have deduced from my mention of the computers and drives, this was in the early 1980's.
      As I recall, the percentage was something like 4% and once the meter (on the floor, since Halon is heavier than air) registered that, the doors were opened.
      I can say for certain that no one exposed died during the test or within the next 5 years that I worked there.

      And thank you tangomargarine for the details.