Stories
Slash Boxes
Comments

SoylentNews is people

posted by janrinok on Wednesday January 05 2022, @12:41PM   Printer-friendly
from the BIG-oops! dept.

This HPE software update accidentally wiped 77TB of data:

We covered this story here University Loses Valuable Supercomputer Research After Backup Error Wipes 77 Terabytes of Data. I, like some others, suspected finger trouble on the part of those doing the backup, but the company writing the sofware have put their hands up and taken responsibility.

A flawed update sent out by Hewlett Packard Enterprise (HPE) resulted in the loss of 77TB of critical research data at Kyoto University, the company has admitted.

HPE recently issued a software update that broke a program deleting old log files, and instead of just deleting those (which would still have a backup copy stored in a high-capacity storage system), it deleted pretty much everything, including files in the backup system, Tom's Hardware reported.

As a result, some 34 million files, generated by 14 different research groups, from December 14 to December 16, were permanently lost.

In a press release, issued in Japanese, HPE took full responsibility for the disastrous mishap.


Original Submission

 
This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 3, Interesting) by tangomargarine on Wednesday January 05 2022, @04:06PM (6 children)

    by tangomargarine (667) on Wednesday January 05 2022, @04:06PM (#1210151)

    some 34 million files,

    Yeah, like how they used to say "computer capable of storing ten million pieces of information" "so it's got a 10 MB hard drive"

    Wait, what? They lost two days worth of files?

    Sure. That’s annoying. It’s definitely a setback. And it absolutely shouldn’t have happened.

    But we’re not talking about years of work lost. Not months. Not even weeks. It’s two days.

    The headlines on this incident would have you believe this is the burning of the Lighthouse of Alexandria or the Sack of Baghdad level cultural loss.

    Depends on the context. My sister was doing experiments with mice for her PhD and she said that if something happened to her mice before the experiment was done, it would cost $5000 a mouse instead of $50 to replace them and start over, because the specific genetic breed she was using was no longer in demand from the breeding facility.

    Even aside from missing an obviously critical event ("we were studying the build-up and actual event of this supernova and we've still got the last 3 months but we lost the initial 2 days of the nova itself"), it's always possible that the data as a whole is worse without a continuous sampling.

    And it's 14 different research groups. So the odds are higher at least 1 of them was significantly impacted.

    The headlines on this incident would have you believe this is the burning of the Lighthouse of Alexandria or the Sack of Baghdad level cultural loss.

    Which of these headlines is implying that?

    This HPE software update accidentally wiped 77TB of data

    University Loses Valuable Supercomputer Research After Backup Error Wipes 77 Terabytes of Data

    "Valuable" is hardly comparing anything to the Library of Alexandria, and "77TB" is just a fact. It sounds like *you're* the one overreacting.

    --
    "Is that really true?" "I just spent the last hour telling you to think for yourself! Didn't you hear anything I said?"
    Starting Score:    1  point
    Moderation   +1  
       Interesting=1, Total=1
    Extra 'Interesting' Modifier   0  
    Karma-Bonus Modifier   +1  

    Total Score:   3  
  • (Score: 2) by maxwell demon on Wednesday January 05 2022, @06:04PM

    by maxwell demon (1608) on Wednesday January 05 2022, @06:04PM (#1210192) Journal

    "computer capable of storing ten million pieces of information" "so it's got a 10 MB hard drive"

    A piece of information could be a bit, in which case it would be more of a 1.44 MB floppy.

    --
    The Tao of math: The numbers you can count are not the real numbers.
  • (Score: 2) by krishnoid on Wednesday January 05 2022, @06:29PM (3 children)

    by krishnoid (1156) on Wednesday January 05 2022, @06:29PM (#1210201)

    Average mass of adult mouse: 30g (on the heavier side)
    Cost of one gram of gold: ~$60
    Replacing one mouse: $5e3 / 30g x $60 = $1800 ~= 2.77 times its mass in gold.

    Sounds like a great investment. In a research lab though, would you buy mouse futures or insurance? It's not like they live that long, so you'd figure they'd either rebreed their own or pay the research breeder a little more to keep a line open for a couple years. I don't know how these "lines" are kept "open" when it comes to rodent husbandry, though.

    • (Score: 2) by tangomargarine on Wednesday January 05 2022, @07:33PM (2 children)

      by tangomargarine (667) on Wednesday January 05 2022, @07:33PM (#1210217)

      My numbers may not be accurate, but it was at least an order of magnitude.

      I think the issue is, if they have an active breeding facility at the supplier, all they need to do is dip into the cage to get some mice out, and ship them to you. When there isn't enough demand for a genetic line, they freeze some of its DNA for storage...so if you want that same strain again, they have to use petri dishes to "spin up" the breeding population again before it becomes self-sustaining, which will take longer and be more inconvenient for them.

      I think the strain she was working on may have affected the fertility of the mice involved, too.

      --
      "Is that really true?" "I just spent the last hour telling you to think for yourself! Didn't you hear anything I said?"
      • (Score: 2) by krishnoid on Wednesday January 05 2022, @07:42PM

        by krishnoid (1156) on Wednesday January 05 2022, @07:42PM (#1210224)

        Are you serious? Seems like a great B-story premise for a science-fiction epic, with humans instead of mice.

      • (Score: 1, Informative) by Anonymous Coward on Wednesday January 05 2022, @11:04PM

        by Anonymous Coward on Wednesday January 05 2022, @11:04PM (#1210325)

        That is about right. Getting a specific mouse line repeated can easily cost multiple orders of magnitude than ordering them in the first place. The suppliers can easily get you more popular one or the ones they have on hand, but if you go outside of that Oh boy can it get expensive. If they have to dip into their embryo stock, replicate the growth environment, and all the other work that goes into creating a mouse line, and they need to do it in a rush so your experiments aren't delayed/corrupted, and they don't have a large number of orders to spread out the cost, it adds up really quick per mouse.

  • (Score: 0) by Anonymous Coward on Friday January 07 2022, @09:44AM

    by Anonymous Coward on Friday January 07 2022, @09:44AM (#1210801)

    I had experience when working on a project with free electron laser our team got four nights to use the beam and boy it would be a shame to loose data from very expensive research equipment if some bug would wipe out our data. LHC at CERN maitenance is $3000000 a day on average, so two days might be very costly at research institutions.