SoylentNews is people
SoylentNews
https://www.xda-developers.com/your-unpowered-ssd-is-slowly-losing-your-data/
SSDs have all but replaced hard drives when it comes to primary storage. They're orders of magnitude faster, more convenient, and consume less power than mechanical hard drives. That said, if you're also using SSDs for cold storage, expecting the drives lying in your drawer to work perfectly after years, you might want to rethink your strategy. Your reliable SSD could suffer from corrupted or lost data if left unpowered for extended periods. This is why many users don't consider SSDs a reliable long-term storage medium, and prefer using hard drives, magnetic tape, or M-Disc instead.
Unlike hard drives that magnetize spinning discs to store data, SSDs modify the electrical charge in NAND flash cells to represent 0 and 1. NAND flash retains data in underlying transistors even when power is removed, similar to other forms of non-volatile memory. However, the duration for which your SSD can retain data without power is the key here. Even the cheapest SSDs, say those with QLC NAND, can safely store data for about a year of being completely unpowered. More expensive TLC NAND can retain data for up to 3 years, while MLC and SLC NAND are good for 5 years and 10 years of unpowered storage, respectively.
The problem is that most consumer SSDs use only TLC or QLC NAND, so users who leave their SSDs unpowered for over a year are risking the integrity of their data. The reliability of QLC NAND has improved over the years, so you should probably consider 2–3 years of unpowered usage as the guardrails. Without power, the voltage stored in the NAND cells can be lost, either resulting in missing data or completely useless drives.
This data retention deficiency of consumer SSDs makes them an unreliable medium for long-term data storage, especially for creative professionals and researchers. HDDs can suffer from bit rot, too, due to wear and tear, but they're still more resistant to power loss. If you haven't checked your archives in a while, I'd recommend doing so at the earliest.
(Score: 1, Touché) by Anonymous Coward on Sunday November 30, @11:05AM (3 children)
Still the winner for long term storage.
(Score: 4, Touché) by Anonymous Coward on Sunday November 30, @01:28PM (2 children)
(Score: 2) by aafcac on Sunday November 30, @08:37PM
They do, but you can also just punch 1s and 0s into metal and have it last rather a long time. But, I think that in most cases, it makes more sense to prioritize ease of data verification over absolute longevity as the media might last for centuries, but will there still be any hardware with which to read the media a couple decades down the line?
Regardless of medium, you are going to want to read and verify the data periodically anyways, so might as well copy it to a newer storage technology from time to time anyways.
(Score: 3, Insightful) by jb on Monday December 01, @07:42AM
Not for reliability. Drop a 30 year old tape on a hard floor and it bounces. Drop a 30 year old clay tablet on the same floor and I'd be willing to bet it would shatter.
Humans make clumsy mistakes. It's nobody's fault, we just do. Any useful backup medium needs to be resilient against that sort of thing.
(Score: 5, Interesting) by JoeMerchant on Sunday November 30, @02:00PM (1 child)
SSDs are charge storage devices, the longevity of the data depends on high impedance (resistance) barriers between cells.
When they were brand new (10-ish years ago) SSDs were rated to hold data for 10 years "when new."
As they are used (mostly write cycles) those resistive barriers break down - a very pro- business feature ensuring replacement sales. After they wear down, data retention times dwindle to a year of power off, then to a month or less.
Of course this wear doesn't happen evenly, so much has been done around "wear leveling algorithms," which, of course, are not all equally successful. Some models of SSD exhibit worn out behavior much faster than others. In addition to wear leveling, there are also error correcting schemes which prevent data loss, up to a point.
One warning sign of an SSD "on the edge" is very slow read and write performance, particularly slow boot times.
One way to mask these issues is to keep your SSDs continuously powered, of course this means that when you finally do power it off, your data is gone the next time you power it up.
Some newer SSDs are longer wearing, most newer SSDs are just bigger and cheaper, but that's a good thing too because if you have a 4TB SSD that you use for the same data activity as a 100GB SSD, the 4TB drive will wear out 40x slower.
🌻🌻🌻🌻 [google.com]
(Score: 2, Interesting) by Anonymous Coward on Monday December 01, @01:18AM
The temperature when the data was written and the temperature when it was stored also matters according to stuff 10 years ago:
https://archive.is/1lW1R [archive.is]
Note: the article linked to was about stuff past endurance ratings.
(Score: 4, Interesting) by mrpg on Sunday November 30, @04:51PM (11 children)
What is the correct way to stop data degradation (the decay of electromagnetic charge in a computer's storage)? Rewrite all the data in a SSD (and also HDD)? I already have multiple backups, I ask about data degradation.
(Score: 1, Informative) by Anonymous Coward on Sunday November 30, @05:30PM
The other little surprise is there are a finite number of rewrites on a cell.
Doing too many writes will just reduce cell's usable lifetime.
(Score: 3, Interesting) by aafcac on Sunday November 30, @09:01PM (8 children)
That should be necessary, from what I understand, the SSDs will transparently move data around from time to time to avoid that issue, the stated capacity will oftentimes not include some extra space to deal with issues that arise from the data shuffling. The better strategy is to have good backup strategies and data integrity software like checksum or par2 to identify when rarely used, but important, files are starting to go bad. Par2 is great because it also allows you to repair some of the damage that might occur. I like ZFS, but from what I understand, it doesn't necessarily play well with flash memory.
(Score: 5, Interesting) by JoeMerchant on Sunday November 30, @10:02PM (7 children)
I have wanted to like ZFS for a very long time now.
Finally built a system with ZFS on the boot volume, ran it for a year and it imploded while half full - total meltdown of the filesystem, I could probably have salvaged most of the data if it was important but this was a "daily driver" system that didn't really have anything unique on it, all backed up elsewhere, so I just reformatted in disgust to ext4 and that system has been running strong for about a year since then now.
ZFS has been "not quite ready for production use" on Ubuntu for what seems like a decade now.
🌻🌻🌻🌻 [google.com]
(Score: 3, Interesting) by aafcac on Sunday November 30, @11:18PM (6 children)
I liked ZFS on FreeBSD, but I'm not ready to really trust it on Linux even though it's probably fine. I was mostly using it with redundancy and without a lot of the more complicated things, so it was fine, but I'm likely to give it a go in the near future as I've got an external HDD case and I'm likely to primarily use it with video files from ripped DVDs, so replacing them wouldn't be any real effort if for some reason something does happen.
(Score: 3, Interesting) by JoeMerchant on Monday December 01, @01:51AM (3 children)
I get the impression it may work better for data volumes - somehow the load of being the root of the whole system boot volume just messed with ZFS a little too hard.
🌻🌻🌻🌻 [google.com]
(Score: 4, Informative) by remote_worker on Monday December 01, @04:36AM (2 children)
I haven't tried ZFS on linux, but given that the licenses aren't compatible enough for ZFS to be in a distributed kernel I'd expect a few wrinkles with it. In particular, ZFS eats a lot of VM (as an ARC), and if the kernel isn't properly set up to handle that I would expect things to fall apart.
Where ZFS shines is FreeBSD (where ZFS is integrated into the kernel) and ZFS on root. Upgrades are no-risk:
1) make a zfs snapshot of your pre-upgrade system. Use bectl (bectl create ...) for the root (boot environment), and plain ZFS snapshots of any non-root filesystems that are going to be upgraded. This is fast. Leave the active boot environment alone, that's the one we want to upgrade.
2) perform the upgrade, by whichever method turns your crank
3) try the upgrade out.
4) if the upgrade is good, delete the snapshots from (1) and get on with your day :).
5) if the upgrade is bad, use the bootloader to boot the snapshot from (1), or if you can get to a shell in the upgrade, activate the root snapshot with "bectl activate ..." and reboot, then use plain ZFS to rollback to the snapshots from (1).
6) Once you're back in the pre-upgrade system you can delete the snapshots from (1) or leave them while you try some other upgrade approach.
One other thing that ZFS involves is that you should do a periodic "scrub" of every pool. I do it weekly, via crontab, at a time I'm pretty sure I'm going to be sleeping :). I expect monthly would be OK as well, but weekly is easier to fit into my timetable. The "zpool scrub ..." starts a background process that cleans up and checks a ZFS pool. My largest pool is a bit over 6Tb of spinning rust (mirrored drives), about 40% used, and scrub runs in 3.5 hours. The pool is perfectly usable while the scrub is running, but possibly a bit slower. A "zpool status ..." gives you the status of the scrub and lets you know if your drive is starting to show errors. If I paid more attention to the status I could probably have avoided the failures I talk about below :).
I have my ZFS root on a small drive or SSD, but all my user data on a separate mirrored ZFS pool, and do daily backups using borgbackup of both the root pool and the user data pool to another machine. I tend to prefer spinning rust for the user data, both becasue it's still cheaper, and because it doesn't have the decay issues of SSDs, but SSDs are nicer for booting.
I've had user data drives fail and root drives fail. I've never had ZFS fail. The failures in the mirrored pool were stressful because of the time it took to get replacement drives, but otherwise painless (not a single file lost, and the only downtime was the physical drive swap). I didn't lose any files in the root drive failures either, but things got slightly more complicated because a simple retsore from borgbackup to a new drive doesn't make the restored data bootable. Making things bootable again depends on how you like to set things up so I won't waste your time with details, but it isn't difficult.
Really, if you can handle all the variations in the different Linux distributions, FreeBSD is easy. There may be more new stuff going from Linux to FreeBSD than from one Linux distro to another, but none of it is hard stuff, just different stuff :).
(Score: 2) by JoeMerchant on Wednesday December 03, @12:55AM (1 child)
So, perhaps I didn't "scrub" my filesystem like I should have. I was doing an eval for a field deployment (thousands of machines in the field, nobody maintaining them) so, either I would have to setup scripts / cron jobs / whatever to do that for me, but it seems a bit absurd in this day and age that a filesystem that needs that kind of maintenance doesn't have something developed and ready to go in the default configurations.
🌻🌻🌻🌻 [google.com]
(Score: 1) by remote_worker on Monday December 08, @04:00AM
It could be needing a scrub, or it could be the the virtual memory needs got too big as the amount of data grew. A ZFS on linux developer might know, but I don't.
However, you're right about the missing default, it does seem that there should be something for a default scrub setup. I didn't find out about scrub for several months after I started using ZFS. I didn't get a crash, or even a performance slowdown, but I might have if I'd gone longer.
A big field deployment is not the place you'd want to try out a different OS as well as a different FS :). I was living with a dual-boot setup on my desktop until I felt comfortable, so one machine and no travel or remote access issues.
(Score: 5, Informative) by Unixnut on Monday December 01, @12:15PM (1 child)
Seconded on not trusting ZFS on Linux. I've used ZFS for years on FreeBSD and it has been a godsend for data integrity. I've not lost a byte of data in the last decade despite multiple power/system/drive array failures thanks to ZFS.
Having had such success I tried bringing ZFS to Linux servers a few years ago, and it really was not a good idea. For one thing you get no support for the configuration (from example RedHat, if you are trying this in an enterprise Linux environment), secondly it feels clunkly, like ZFS is not so much integrated into Linux as much as "bolted on" in places. This includes bypassing and hacking around the Linux VFS Layer (as ZFS is not just a "filesystem" but a complete vertically integrated storage subsystem).
As a result at best I found ZFS on Linux to be clunky and fragile, and at worst it breaks in some horrific way that makes it impossible to recover without data loss. Perhaps this is due to the licencing incompatibilities preventing proper kernel integration, or it is just something about the way the Linux kernel is designed that prevents it to be reliably integrated.
Either way, for serious/production work on Linux you are better off with MDADM/LVM and ext4. If you want ZFS and all the goodies it brings then best use an OS that has it properly integrated and supported in the kernel.
(Score: 2) by aafcac on Tuesday December 02, @04:11PM
Having messed around with former ZFS based disks that I had decommissioned, one of the things that I came to realize is just how fragile they can be if they get damaged. It hasn't been an issue for me, as I've been generally using the disks as part of a zmirror or raidz configuration and haven't lost data due to that, but there's weird stuff that can happen to the label that makes it a right pain to work with if you don't already know how to do it and are prepared.
That's really the biggest worry I have as it can be relatively hard to push systems like this hard enough to really establish that they're reliable without putting data on there that could be annoying to have to restore. I'm curious about Btrfs which does play a bit better with Linux.
Side rant: But really, the whole thing is the result of some rather poor decisions made decades ago without any real consideration of the consequences. In practice, the difference between GPL and something more permissive like the BSD or MIT license is pretty much non-existent if you don't have attorneys, and even if you do have attorneys, the difference is still rather small. Sure, the GPL theoretically prevents people from taking the code and distributing binaries without providing the source in the same place, but it also means that code can't be easily contributed the way that it would be with a more open license.
I do like modern Linux in a lot of ways, but there's an awful lot of boneheaded decisions being made lately like things involving having so many different ways that programs can be packaged/managed, SystemD everything, Wayland and things that seem to be more or less the same sort of nonsense that MS used to engage in because everybody is special and we need to change things because we need to change them. Never mind that it's absolutely absurd to have to reload system services to issue a mount -a after modifying the fstab or that 17+ years for the nonsense that is Wayland is way too much time when all the necessary bits of Linux to be a functional OS took a fraction of that time.
(Score: 2, Interesting) by Anonymous Coward on Monday December 01, @01:23AM
According to a 10 year old article, if you're going to store SSDs, write at high temperature, store at low temperature, rewrite once in a while before the data fades too much:
https://archive.is/1lW1R [archive.is]
Note the link is about SSDs that have past their TBW but the principle should still be the same, just the actual numbers different.
Also that was 10 years ago, stuff might have changed since...
(Score: 2) by Uncle_Al on Sunday November 30, @05:35PM
The failure mode of a magnetic storage device is rarely magnetic decay.
In the case of disks, it is almost always problems with the motors or heads failing over time.
With tape, it is the fact that you have physical contact with the read/write head an all of the
associated problems with that, and any mechanical parts in the transport.
And remember.. those decaying flash cells aren't just in storage devices. There is flash firmware
and even cells in flash-configurable options on other chips.
(Score: 4, Interesting) by pTamok on Sunday November 30, @08:47PM (10 children)
AFAIK (and I don't know very much), USB 'thumb' drives and mini- and micro-SD cards use stored charge as well, so ought to be subject to similar degradation. I suspect it is a rare USB drive that executes a comprehensive transparent data re-write strategy when it is plugged in.
Many people claim magneto-optical disks are good for data retention: my experience is that the drives for reading and writing them become unavailable (I have an old SCSI one that is effectively useless). Archival quality CDs and DVDs are also meant to be good - but expensive.
For seriously long data retention, the Rosetta project went for etched Nickel disks. [rosettaproject.org]
(Score: 0) by Anonymous Coward on Monday December 01, @01:29AM (9 children)
I have a similar feeling about magnetic tape. I've heard stories about backup tapes that could only be read by the drive that wrote them...
Whereas HDDs come with the drive that wrote the data...
(Score: 0) by Anonymous Coward on Monday December 01, @08:08AM (2 children)
Nowadays basically all tape drives follow the LTO standards and are interoperable with other LTO drives of the same generation. But long-term compatibility is still a problem as the standards are updated every couple years and new drive models are not generally backwards compatible with old tapes (most generations could read one or two prior generations of tapes, but the latest LTO-10 is fully incompatible with all previous versions.
The people maintaining tape archives are presumably expected to switch to the latest formats as they become available and migrate all data which should be retained.
If you have a large tape archive and don't do this, you could end up like the BBC where they had something like a quarter million D-3 tapes as the only archival copies of things, and it was unknown if there were even enough working tape heads (which wear out over time) in the world to read them all.
(Score: 0) by Anonymous Coward on Monday December 01, @08:49AM (1 child)
Meanwhile I can easily use a SATA HDD from more than 10 years ago. So IMO if HDDs can do the job, it's better to use HDDs instead of tapes. Just store them well and don't drop them...
(Score: 2) by aafcac on Tuesday December 02, @04:15PM
HDDs aren't perfect, but they are cheap, reasonably durable, reasonably large and you can automate most of the process of verifying, repairing and transferring the data from one drive to a new drive as appropriate. We may someday get to the point where the interconnection is stable enough that that isn't needed, but we likely won't know that we're there until a rather long time after we hit that point. We may well be there with SATA as that hasn't been changing much in terms of wiring or chips other than more or less the same thing, but faster, but I'm not ready to make the assumption that we're not going to switch to something at some point. Especially with things like nVME chips being a common thing these days.
(Score: 2, Interesting) by pTamok on Monday December 01, @08:21AM (5 children)
> I've heard stories about backup tapes that could only be read by the drive that wrote them...
In the days of 9-track ½-inch tapes [wikipedia.org], that was a common occurrence as a result of head misalignment. I have war-stories about that, which I won't bore people with here.
(Score: 5, Interesting) by Unixnut on Monday December 01, @12:29PM (4 children)
I've had it happen with LTO tapes. Usually how it turns out is that a LTO drive falls out of misalignment (either due to age, lack of maintenance, or someone dropped it at some point and told nobody). The drive can write and read back its tapes absolutely fine, but if the drive fails and you switch it for another one, you find that all your tapes are unreadable.
Many moons ago this happened at a company I was a sysadmin at. We diligently did all the backups to LTO-3 tapes with verification step, marked them and put them in the offline storage vault for years. Then there was a disaster in the server room (it was in the basement, and a sewage pipe of the building burst flooding the basement. The smell was... indescribable) which took out all the servers and the LTO tape drive.
However when the dust settled and we were setting up the new server room, we found that the new LTO-3 drive could not read our tapes. Not a single tape of all the ones we had in the vault were readable. 3 years of monthly backups and not a single one could be read.
It was a major issue, in the end the company had to bring in a LTO specialist from the manufacturer who took one of our backup tapes, opened up the LTO drive and with an oscilloscope started deliberately misaligning the head until it could read our tapes.
It cost a huge amount in consulting fees but we got our data restored. Only saving grace was none of the IT team could be blamed, as all of us were hired within the three years, meaning if anyone had damaged the LTO drive, it was more than three years ago and were probably long gone.
A lesson in that you need at least two LTO drives, and you should verify on a different LTO drive to the one that wrote the tape, as the chance of both of them being misaligned in exactly the same way is infinitesimally small.
As for my personal back-ups. I use HDD's. I've managed to recover data off drives that were sitting in my attic for ~30 years (old Quantum fireball 4GB). It took a few taps to get the bearings to unstick and start spinning, but once up and spinning all the data was there, and I had no errors or access issues. In fact since then the drive spins up and works fine, just I can't find a use for it (seriously, I have USB keys that hold more data nowadays).
SSDs are good for daily high speed access, but I see no reason to not use the right technology for the job. For bulk and long term/archival storage HDD's and magnetic media work, Each has its niche.
(Score: 2, Insightful) by pTamok on Monday December 01, @02:22PM (3 children)
That sounds very, very similar to a couple of my war stories.
> you should verify on a different LTO drive to the one that wrote the tape
This is a/the key point. When verifying your backup, check also that it is readable in a different reader. Always.
(Score: 4, Interesting) by Unixnut on Monday December 01, @03:16PM (2 children)
No doubt, every war story I've heard follows a similar pattern. Usually involving prior bad decisions, lack of funding/interest from those in charge (who are not technically savvy) and an unfortunate disaster. I've been through a fair few of them. The "basements flooding" disaster were quite common, because back then the basement was the one place nobody wanted to work or be in but is a space which the company paid for, so logically it was the best place to put servers as they did not care about the smell or lack of natural light and thereby save money on office rental space.
The irony of the situation in my war story, is that after all that cost and rigmarole, the beancounters actually rejected our request to buy a second LTO drive, pointing out we now have a new one that has just been verified as "calibrated", so everything is now fine.
Back then I could not understand the short sightedness, but nowadays I have learned that there is a difference between capital and operational expenditure when it comes to tax write-offs and accounting, so perhaps it was the financially better off decision they made for the company.
Head misalignment is a common issue with devices where the recording medium is not always matched to the recording/playback equipment. I've had same issue above happen with my personal zip drives, and even with minidiscs.
It is also the reason I prefer to archive/backup on HDD's. The drive and heads are always aligned because they are built into the same enclosure. As long as I have an interface that can connect to the drive (currently SATA/SAS) I don't have to worry about misalignments, and so far the commonly mentioned failure modes of drives used for backups (e.g. stuck bearings) has not happened once.
Saying that, a lot of companies now just "back up to the cloud" so I don't think many even bother thinking about it anymore, but during the 2000's and before it was still an issue that many SME's had to think about. Probably why "Cloud" became so popular, as they didn't need to worry about this, or even need an IT employee anymore.
(Score: 2, Interesting) by pTamok on Monday December 01, @07:37PM (1 child)
Heh. Basement floodings are commonplace.
How about flooding an umpteenth floor data-centre in a high-rise office building*? That wasn't my story, but one I read. I thought it was in the The RISKS Forum Mailing List [seclists.org], but I wasn't able to find it the last time I searched for the source. There are some humdingers in that list.
As for 'backing up to the cloud', if your business relationship with the cloud provider fails, you lose access to the backups. This is generally not good for business. Similarly, there are various technical failures that can give you a really bad day. It's always DNS, unless it is a lost encryption key.
*Grundfos: Water distribution in high-rise buildings [grundfos.com]
Olympian Water Testing: Why do Tall Buildings have Water Towers on their Roofs? [olympianwatertesting.com]
(Score: 3, Interesting) by aafcac on Tuesday December 02, @04:22PM
I used to work security in a highrise where one of the standpipes storing water for the sprinklers had broken. That was before my time, but one of those systems can flood an area with an almost unimaginable amount of water in a matter of a few minutes if it breaks. But, a more common issue is for just one or a few of the sprinkler heads to start dumping water because they got triggered.
Water damage as you suggest is the biggest source of property damage in this context. Between plumbing disasters, literal floods and storm surges it's a lot.
My slightly off-site backups for the largest files I have go into a water resistant ammo can. It's not perfect, but if the sprinklers in the storage area do go off, it's good enough for that. If I were storing them there longer term, I'd probably throw a few desiccant packets in with the drive and use my vacuum sealer without the vacuum to seal it off. Which is fine, as that might be an issue in case of a fire, but if it's a fire hot enough to melt the vacuum seal plastic in a problematic way, it's likely hot enough to destroy the disk.