Stories
Slash Boxes
Comments

SoylentNews is people

posted by CoolHand on Monday September 12 2016, @01:32PM   Printer-friendly
from the booms-and-bangs dept.

https://motherboard.vice.com/read/a-loud-sound-just-shut-down-a-banks-data-center-for-10-hours

ING Bank's main data center in Bucharest, Romania, was severely damaged over the weekend during a fire extinguishing test. In what is a very rare but known phenomenon, it was the loud sound of inert gas being released that destroyed dozens of hard drives. The site is currently offline and the bank relies solely on its backup data center, located within a couple of miles' proximity.

"The drill went as designed, but we had collateral damage", ING's spokeswoman in Romania told me, confirming the inert gas issue. Local clients were unable to use debit cards and to perform online banking operations on Saturday between 1PM and 11PM because of the test. "Our team is investigating the incident," she said.

The purpose of the drill was to see how the data center's fire suppression system worked. Data centers typically rely on inert gas to protect the equipment in the event of a fire, as the substance does not chemically damage electronics, and the gas only slightly decreases the temperature within the data center.


Original Submission

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 5, Interesting) by VLM on Monday September 12 2016, @01:51PM

    by VLM (445) Subscriber Badge on Monday September 12 2016, @01:51PM (#400698)

    inert gas

    They probably mean halon.

    I've personally seen the aftermath of two halon dumps, and obviously the journalist has not, and has not even talked to anyone.

    First small stuff blows everywhere and clogs every fan so I bet a significant fraction of dead drives were baked in "fresh" dust. You have to shut down the power and replace or at least inspect all the filters. Even if you think you have a "clean" data center.

    Second the suppression "experts" have a fine line between quickly flooding out a fire vs what feels like a compressed gas tank explosion. Yes I have personally seen raised floor tiles that popped loose and lightweight cables (aka fiber) that has been physically moved or stretched underneath the floor. Say what you want about bus and tag connectors nothing short of a lumber axe can damage them but you take an escon fiber cable and stretch it with the force of two fingers and you need a new escon cable. That stuff's all obsolete now, probably.

    Third is halon and halon dumps are incredibly expensive and the circular firing squad will be out in force, so right when you need your best ops crew checking air filters for individual devices and large coolers, and reinserting cables that have been yanked out, and fixing floor and ceiling tiles that have been blown out, they'll all be sitting in a departmental meeting for hours on the topic of conserving shareholder value and who did what when and is there any way we can blame this on the intern or the guy who retired last month and it was all the other departments fault and did anyone have a maintenance notification open that we can blame or ... and this can go on for hours or days and multiplied by the staff qty and pay rate you can exceed the cost of wasted halon in terms of meeting wasted labor costs.

    Once in my career I got to be the guy holding the halon override button while people ran around in circles trying to verify there is or is not a fire. So that was kinda stressful.

    Starting Score:    1  point
    Moderation   +4  
       Insightful=1, Interesting=2, Informative=1, Total=4
    Extra 'Interesting' Modifier   0  
    Karma-Bonus Modifier   +1  

    Total Score:   5  
  • (Score: 4, Funny) by weeds on Monday September 12 2016, @02:14PM

    by weeds (611) on Monday September 12 2016, @02:14PM (#400713) Journal

    I experienced a Halon dump test once (so this is truly anecdotal.) The dump was done from the ceiling, nothing under the floor or anywhere else.

    The test was done with the full staff in the room. Doing it this way increases the concentration of the Halon in the room (displacement) and makes the test more likely to pass.

    All of the systems were running. (This will get you a laugh - Perkin Elmer 32 bit systems - 3220 and 3240 with CDC 9766 Drives - 300 MB!)
    None of the equipment was damaged and these drives were pretty sensitive. The only negative was the gunk that came out of the pipes and made some unsightly marks on the ceiling. Staff enjoyed a few minutes of having voices like James Earl Jones.

    • (Score: 1) by Delwin on Monday September 12 2016, @04:23PM

      by Delwin (4554) on Monday September 12 2016, @04:23PM (#400786)

      You must be talking about Helium, not Halon. Halon is heaver than air. It will also kill you quite quickly.

      • (Score: 1, Insightful) by Anonymous Coward on Monday September 12 2016, @04:35PM

        by Anonymous Coward on Monday September 12 2016, @04:35PM (#400792)

        Considering the voice like James Earl Jones part, I doubt he was referring to helium.

      • (Score: 2) by bob_super on Monday September 12 2016, @05:21PM

        by bob_super (1357) on Monday September 12 2016, @05:21PM (#400817)

        Helium rises, which is often counterproductive when trying to displace O2 to stop a fire.
        Regardless of which gas, and considering equipment racks are taller than me, a properly designed system should be killing humans if it's doing its job. If you're a basketball player, you may get to die from the fire smoke rather than from the fire suppressant.

        • (Score: 3, Informative) by VLM on Monday September 12 2016, @07:37PM

          by VLM (445) Subscriber Badge on Monday September 12 2016, @07:37PM (#400891)

          a properly designed system should be killing humans if it's doing its job.

          Rather pessimistic. Assuming good mixing you'll get high as a kite via the usual solvent inhalant effect when its pretty effective at stopping fire (a couple percent)

          Of course you don't set a system off for fun, usually, implying there's a killer fire, so you get the heck out because of the fire not necessarily the halon or halon replacement.

          A good engineering rule of thumb is half the atmosphere is below 18kft and 18kft mountains or aircraft flying is no laughing matter for random sick civilians but its hardly guaranteed death and that's 50% depletion of O2 via 50% halon in the air.

          Probably 5% or so in the air might put out a fire if in some miracle it were perfectly mixed, 10% or so will make you high as a kite and lay down and die in the fire (I mean, there is a fire, after all), and nobody spends enough money to put way more than 50% in the air. I mean, even before it was banned, it wasn't cheap.

          Like I wrote I've worked at sites that had a halon dump, nobody dies although I heard its utterly terrifying and the mess was impressive.

          Of course, 50% of O2 displaced from air can't kill healthy hiker me, but if I huffed lungfulls of smoke to cause smoke inhalation problems (because you're doing a halon dump because of the fire...) then maybe 50% halon could kill a no longer healthy dude. Like 50:50 halon and plastic smoke probably wouldn't be good for you even if 50:50 halon and air would be fine. Then again at 50% plastic smoke intake you're probably dead already. Hmm.

        • (Score: 2) by sjames on Monday September 12 2016, @09:42PM

          by sjames (2882) on Monday September 12 2016, @09:42PM (#400951) Journal

          No. It should not be killing humans. Some places use a reduced oxygen environment (15% rather than the usual 20) to prevent fire. It's perfectly safe for any reasonably healthy person to be in the room for a while as long as they don't do heavy labor.

          • (Score: 2) by bob_super on Monday September 12 2016, @09:49PM

            by bob_super (1357) on Monday September 12 2016, @09:49PM (#400955)

            One could easily argue that being in a server room during an unexpected halon release, especially if there's a real fire, would be a stress level that qualifies as hard labor.
            (I did give VLM a +1 for his detailed response. I thought the O2 displacement was higher than 5%)

      • (Score: 2) by weeds on Monday September 12 2016, @08:01PM

        by weeds (611) on Monday September 12 2016, @08:01PM (#400903) Journal

        Halon - Not at 100% concentration, just enough to put out a fire. That won't kill you. I was in the room too.
        Denser than air = lower voice.

      • (Score: 2) by tangomargarine on Monday September 12 2016, @08:43PM

        by tangomargarine (667) on Monday September 12 2016, @08:43PM (#400914)

        Supposedly halon extinguishes fires at a low enough concentration that you can stay in the room for at least some amount of time and still be okay. Not that you'd probably *want* to, but yeah.

        --
        "Is that really true?" "I just spent the last hour telling you to think for yourself! Didn't you hear anything I said?"
    • (Score: 3, Informative) by Hyperturtle on Monday September 12 2016, @08:49PM

      by Hyperturtle (2824) on Monday September 12 2016, @08:49PM (#400918)

      I am seriously doubting it was Halon in the dump of 'inert gas', and wonder about your co-workers.

      Halon is a toxic chemical and is a central nervous system irritant. Wikipedia has all the relevant details.

      https://en.wikipedia.org/wiki/Bromotrifluoromethane [wikipedia.org]

      I would expect the gas to be Inergen instead -- also in canisters, also a pain clean up after. All of comments of the mess the gas makes seem to be more in line with Inergen from what I understand -- I've been through a few of those.

      Every incident of its deployment involved user error. One time from the company testing the smoke detectors; the guy in one case was throwing his full body weight against a pair of double doors that were trying to open from the pressure--security doors (like fire doors with badge access to a datacenter types of doors) blown off their hinges. The guy was bruised from head to toe. And the mainframe printer room -- it was like a tornado got in there and threw all the toner (which was in giant multi gallon bottles) into the air and all over the place. Tractor feed and laser printer paper all over the place as well. The print operators were pretty traumatized as there was no warning.

      Not only did the place go down, but on top of the service disruption, it costs hundreds of thousands of dollars; one time it was caused by the company providing the Inergen gas. It cost thousands just to recertify the system -- tens of thousands. Remember, stuff that sat unused for years suddenly had a huge gas discharge blown through at high pressures and blew the caps off things and cracked other stuff.

      Then another time a contractual foreign national that followed in the badge swipe of another person from the same firm had never seen the red button on the wall before, and pressed it. There is no un-press the button feature--there is no bypass like how there is with an actual smoke or fire alarm. The system presumes if you lifted the shield and pressed the button, you mean business. Again, six digits of damage. Another time someone didn't think a cigarette would set it off.

      The gas has to be replaced, and the system it connects to has to be inspected and certified.

      Then everything the gas gets sucked into or condenses on -- servers, fans, air filters, vent grilles, the space under raised floors... all gets covered in dust and much and stuff that got blown around after years of being still.

      The fact that disk drives now are now too sensitive to withstand this sort of air pressure (which can tear doors off their hinges) tells me that the damages nowadays are much higher...

      • (Score: 4, Informative) by tangomargarine on Monday September 12 2016, @09:11PM

        by tangomargarine (667) on Monday September 12 2016, @09:11PM (#400930)

        For example, Halon 1301 total flooding systems are typically used at concentrations no higher than 7% by volume in air, and can suppress many fires at 2.9% v/v. By contrast, carbon dioxide fire suppression flood systems operate from 34% concentration by volume (surface-only combustion of liquid fuels) up to 75% (dust traps). Carbon dioxide can cause severe distress at concentrations of 3–6%, and has caused death by respiratory paralysis in a few minutes at 10% concentration. Halon 1301 causes only slight giddiness at its effective concentration of 5%, and even at 15% those exposed remain conscious but impaired and suffer no long-term effects. (Experimental animals have also been exposed to 2% concentrations of Halon 1301 for 30 hours per week for 4 months, with no discernible health effects.) Halon 1211 also has low toxicity, although it is more toxic than Halon 1301, and thus considered unsuitable for flooding systems.

        However, Halon 1301 fire suppression is not completely non-toxic; very high temperature flame, or contact with red-hot metal, can cause decomposition of Halon 1301 to toxic byproducts.

        https://en.wikipedia.org/wiki/Halomethane#Fire_extinguishing [wikipedia.org]

        --
        "Is that really true?" "I just spent the last hour telling you to think for yourself! Didn't you hear anything I said?"
      • (Score: 2) by weeds on Tuesday September 13 2016, @02:55PM

        by weeds (611) on Tuesday September 13 2016, @02:55PM (#401320) Journal

        Well, my friend, you can doubt it all you want.
        I spec'd the system, I selected the system, I purchased the system, I supervised the installation of the system, and I was in the room for the test.
        Those cylinders that said HALON on them were connected and I had the pleasure of sounding like Michael Clarke Duncan for the few minutes we were in there.
        As you may have deduced from my mention of the computers and drives, this was in the early 1980's.
        As I recall, the percentage was something like 4% and once the meter (on the floor, since Halon is heavier than air) registered that, the doors were opened.
        I can say for certain that no one exposed died during the test or within the next 5 years that I worked there.

        And thank you tangomargarine for the details.

  • (Score: 2) by DeathMonkey on Monday September 12 2016, @05:24PM

    by DeathMonkey (1380) on Monday September 12 2016, @05:24PM (#400821) Journal

    They probably mean halon.
     
    Possibly, since it's in Romania, but in the US halon has been banned since it's a CFC. There are still some grandfathered installs around but you can't build a new one.

  • (Score: 3, Informative) by Webweasel on Monday September 12 2016, @05:29PM

    by Webweasel (567) on Monday September 12 2016, @05:29PM (#400827) Homepage Journal

    Your a bit out of date:

    The production of Halon 1301 was banned in 1993, a direct result of its negative effects on the ozone layer.

    https://nationalfireinc.com/suppression-systems/inergen-ig-541.html [nationalfireinc.com]

    We use IG-541 Inergen in our data centers, and have had an accidental release.

    No residue on anything the only problem was it cut the power. Servers were more or less fine, nothing major.

    This is UK however, no idea how things are in the US. I'd guess most systems have been replaced now and would be surprised to see halon anywhere.

    --
    Priyom.org Number stations, Russian Military radio. "You are a bad, bad man. Do you have any other virtues?"-Runaway1956
    • (Score: 3, Informative) by sjames on Monday September 12 2016, @09:57PM

      by sjames (2882) on Monday September 12 2016, @09:57PM (#400962) Journal

      It's production was banned, but many systems installed before that still have it. For years after, there was a grey market for halon in tanks removed from decommissioned or updated data centers.

  • (Score: 2) by Fnord666 on Monday September 12 2016, @09:02PM

    by Fnord666 (652) on Monday September 12 2016, @09:02PM (#400924) Homepage

    They probably mean halon.

    No, they mean inert gas.

    INERGEN® is a mixture of 52% Nitrogen, 40% Argon and 8% CO2. However, in the event of a fire, when INERGEN® is discharged, it mixes with the air present in the room to create a mixture that comprises of 67.3% Nitrogen, 12.5% Oxygen, 17% Argon and 3.2% Carbon Dioxide.

    This discharge mixture makes INERGEN® singularly unique. No other inert gas has the unique ability to rapidly extinguish a fire yet at the same time provide a safe environment for any person within the occupied area by actually decreasing cardiac distress and maintaining arterial blood oxygenation and mental performance in low oxygen levels.

  • (Score: 2) by driverless on Tuesday September 13 2016, @06:46AM

    by driverless (4770) on Tuesday September 13 2016, @06:46AM (#401151)

    This is Romania though, here's what really happened:

    1. Drill occurs, data centre is evacuated.
    2. After awhile, staff re-enter. Find back door onto loading dock open and tumbleweeds blowing through where the server racks used to be.
    3. Lots of new servers suddenly turn up at Piața Aurora-Flavia market in Timişoara.
    4. Operators blame cosmic rays^H^H^H^H^Htermites^H^H^H^HLOUD NOISES!