SoylentNews

hubie [soylentnews.org] writes:

When it comes to recording, tracking, and storing data, the concept of "permanence" can be a lot trickier than you'd expect [gizmodo.com]:

More and more, our data—private photos, work documents, confidential information, and so on—are migrating to digital clouds and platforms. In a more recent and perhaps controversial turn of events [gizmodo.com], some gaming firms are pushing for digital downloads in lieu of physical CDs.

One recurring argument against digital options has been that it's not as "permanent" as their physical counterparts. After all, aren't physical storage systems safer from the whims of a tech firm wiping your cloud storage—or, even worse, forcing you to choose a subscription plan?

But CDs break. Books burn. USBs go defunct. Personally, I—a stickler for physical media—have a very bad track record of breaking my game chips and scratching my movie discs. There are real challenges in dealing with physical storage systems, too, and big players like Microsoft are investing significant resources to overcome the inevitable decay [gizmodo.com] caused by, well, the passage of time.

[...] The following responses may have been lightly edited for length and clarity.

Mohiuddin Ahmed, Computer scientist, Adelaide University, Australia.

Data storage is a contextual matter. Given the transient nature of everything around us, permanent data storage is some sort of a myth at this stage. That said, multiple copies of data and moving them to a new storage every couple of years can be thought of as a practical solution toward permanent data storage.

Francesca Musiani, Research director, Center for Internet and Society at the National Center for Scientific Research, France.

I find the idea of "permanent" data storage quite elusive, not only in our digital "era." Every medium we have ever invented eventually fails, from stone inscriptions that erode to paper burning, CDs' "disc rot" and solid-state drives that gradually lose stored charge. Further, even if a storage medium could survive for centuries, the devices and software needed to read it may not; a perfectly preserved floppy disk is of little use if no one can build or maintain a compatible drive. In this sense, data loss is a techno-historical problem, I would say: preserving information requires preserving the means of interpreting it.

[...]

Jack Cushman, Director, Library Innovation Lab at Harvard Law School; creator of the Public Data Project.

I know of one data storage form that can be permanent: a library. Or rather, a world full of libraries. Libraries aren't perfect storage devices (nothing is), but they are fiercely committed to self-repair. A good library is a collection of people, technologies, and practices that pass an obligation—to remember the things we must remember—from the last generation to the next.

Libraries, rather than better hard drives, are what solve correlated failure, cataloged in the LOCKSS (Lots Of Copies Keep Stuff Safe) threat model. A correlated failure is when you make really great, durable copies of the data you care about, and then they all get hit by an asteroid. Shouldn't have had them all in one physical location. Or a hacker marks them all for deletion (they were all behind one sysadmin), or a government orders them all destroyed (all in one regulatory regime), or they all rot at the same time (all from one flawed batch), or the money to store them dries up (all on one funding source).

On a long enough timeline any copy will break. You need multiple copies with different vulnerabilities so they don't all break at the same time. Then you need to repair the broken ones. A worldwide network of libraries does the repair work, whether the objects are 30 medieval Magna Carta manuscripts [harvard.edu] or 300,000 government datasets [harvard.edu].

[...]

Kevin Curran, Computer scientist and cybersecurity researcher, Ulster University, U.K.

No form of digital storage media is truly reliable for passive, long-term archival use. Entropy eventually wins, failures occur faster than expected, and the only practical solution is active management—periodic verification, rewriting data, and maintaining multiple redundant copies on fresh media. The most common form of storage which is hard drives—should not be viewed as archival media. Hard drives should be treated as consumables.

A recent Iron Mountain report highlighted that one-fifth of 1990s era hard drives sent to them were entirely unreadable, even under ideal storage conditions. Quite simply, standard hard drives were not designed for long-term archival use. You can never decouple the magnetic disks from the reading hardware inside so if either fails, then the whole drive dies. Hard drives magnetic charge fades over time, bearings seize, and there is often no partial recovery option. There are methods to try to combat this such as periodic full rewrites to refresh the magnetic domains but how many would ever do this?

[...]

Adnene Guabtni, Principal research scientist, Commonwealth Scientific and Industrial Research Organisation, Australia.

Data loss, whether from hardware failure, natural disaster, cyberattack, human error, or simply a discontinued cloud service, has plagued computing since its inception. Solving it requires a multi-faceted approach combining data replication and distribution, decentralization, and data sovereignty. First, data replication and distribution across multiple cloud storage services reduces the risk of data loss from a single point of failure. This approach costs more, since the user must subscribe to multiple providers. Yet, it doesn't protect against a global outage or a coordinated cyberattack affecting several providers at once.

[...]

Melanie Hubbard, Head of Digital Scholarship & Data Services, Boston College Libraries

When we talk about data these days, we typically mean data that is born-digital or digitized. It is the digital aspect that adds to its instability, though it is also what makes it more discoverable, accessible, usable, and manageable. From the moment something is made digital, it becomes harder to grasp, like sand sifting through fingers, as the risks of degradation, context loss, format obsolescence, container vulnerability, and shifting technology loom. For instance, I can't plug in a FireWire drive or load a CD directly on my laptop anymore. I can plug in my USB drive, but the data is corrupted.

The closest we come to data permanence is when it is in a pure analog format, such as books, punchcards, and stone tablets, but these are not practical for data storage and access at scale, and analog offers no guarantees. After all, ancient libraries stand in ruins today, if they stand at all. And even when technological limitations don't cause data loss, there are always the risks posed by politics, corporate greed, the laws of thermodynamics, or natural disaster.

Original Submission