SoylentNews is people
SoylentNews
from the someday-coming-to-a-phone-near-you? dept.
For those Linux folks out there, imagine merging LVM2, dm-raid, and your file system of choice into an all powerful, enterprise ready, check-summed, redundant, containerized, soft raid, disk pool, ram hungry, demi-god file system. The FreeBSD Handbook is a good start to grep the basic capabilities and function of ZFS[*].
The Ars reports:
A new long-term support (LTS) version of Ubuntu is coming out in April, and Canonical just announced a major addition that will please anyone interested in file storage. Ubuntu 16.04 will include the ZFS filesystem module by default, and the OpenZFS-based implementation will get official support from Canonical.
...
ZFS is used primarily in cases where data integrity is important—it's designed not just to store data but to continually check on that data to make sure it hasn't been corrupted. The oversimplified version is that the filesystem generates a checksum for each block of data. That checksum is then saved in the pointer for that block, and the pointer itself is also checksummed. This process continues all the way up the filesystem tree to the root node, and when any data on the disk is accessed, its checksum is calculated again and compared against the stored checksum to make sure that the data hasn't been corrupted or changed. If you have mirrored storage, the filesystem can seamlessly and invisibly overwrite the corrupted data with correct data.
ZFS was available as a technology preview in Ubuntu 15.10, but the install method was a bit more cumbersome than just apt-get install zfsutils-linux. I for one am excited to see ZFS coming to Linux as it is a phenomenal solution for building NAS devices and for making incremental backups of a file system. Now I just wish Ubuntu would do something about the systemD bug.
[*] According to Wikipedia:
ZFS is a combined file system and logical volume manager designed by Sun Microsystems. The features of ZFS include protection against data corruption, support for high storage capacities, efficient data compression, integration of the concepts of filesystem and volume management, snapshots and copy-on-write clones, continuous integrity checking and automatic repair, RAID-Z and native NFSv4 ACLs.
ZFS was originally implemented as open-source software, licensed under the Common Development and Distribution License (CDDL). The ZFS name is registered as a trademark of Oracle Corporation.
OpenZFS is an umbrella project aimed at bringing together individuals and companies that use the ZFS file system and work on its improvements.
(Score: 0) by Anonymous Coward on Friday February 19 2016, @05:25AM
A single flipped bit in RAM in a single file can hose an entire ZFS filesystem. This is because the error is spread across the whole filesystem through the checksums used to verify file integrity. ZFS is designed not to trust HDDs and always trust the rest of the computer.
Now while I'd love to see ECC RAM in every box, commodity hardware just doesn't support it.
https://duckduckgo.com/?q=zfs+ecc+ram [duckduckgo.com]
(Score: 1) by Francis on Friday February 19 2016, @05:39AM
That's not true. A single flipped bit isn't going to take down an entire zfs file system. It would be nice to see a credible source on the matter.
(Score: 1, Touché) by Anonymous Coward on Friday February 19 2016, @06:30AM
sudo cat $GPP > matter.c
(Score: 3, Informative) by Anonymous Coward on Friday February 19 2016, @06:54AM
This question was answered in a nuanced manner on the BSD Now show "ZFS in the trenches" and different developers have different opinions but all recommended ECC if possible. In the episode Josh Paetzel of FreeNAS explains that if the ZFS spacemaps were corrupted due to memory errors you could be in a worse situation compared to less sophisticated filesystems that have corrupt data an fsck would destroy in an attempt to recover the rest [youtu.be].
(Score: 2, Insightful) by pTamok on Friday February 19 2016, @10:58AM
Is there a text article somewhere that goes into these issues?
I'm afraid I have an old-school preference for text over video. It suits the reading and comprehension process I have honed over a lifetime.
I am trying very hard not to make a value-judgement about the utility of video as an information transfer mechanism, and I know I'm probably partially ossified in my ways.
(Score: 2, Insightful) by Francis on Friday February 19 2016, @08:55PM
Not just that, you can skim through quickly to get a rough sense of what's going on before going into things in depth. Greatly improves the comprehension.
(Score: 2) by darkfeline on Saturday February 20 2016, @01:33AM
I used to think that, until I discovered the joys of playing YouTube videos using mpv at 1.33x speed or faster. You can nudge the speed faster or slower depending on the content, but I've found the average informational video works well at 1.33x.
Join the SDF Public Access UNIX System today!
(Score: 0) by Anonymous Coward on Friday February 19 2016, @08:05AM
Now while I'd love to see ECC RAM in every box, commodity hardware just doesn't support it.
To be fair plenty of commodity hardware does. But you got me thinking so I thought I'd check against my semi-recent motherboards.
My GA-Z77X-D3H (Intel 1155) motherboard doesn't, but my GA-MA770T-UD3P (AMD AM3+) does.
(Score: 2) by ledow on Friday February 19 2016, @11:14AM
A single flipped bit in RAM could hose your EFI and kill your laptop, if recent events are anything to go by.
A single flipped bit in RAM is BAD. As in most likely you'll blue-screen or kernel-panic no matter what the OS or filesystem, and then you're into data loss and corruption that no RAID BBU will save you from.
Though ZFS might be technically worse in that scenario, it's already game-over time for continuing operations, it probably means you're going to lose all service to that PC, have a disk check on next boot at minimum and maybe/maybe not lose the database you were writing to no matter what the filesystem.
Either way, you want to not be relying on that never happening. So ECC RAM or redundant storage and servers at minimum. Once you get there, ECC RAM is bog-standard and a pitiful percentage of the overall cost.
Home PC's? Yeah, maybe that's a problem. But to be honest, you're looking at a "power-user" use case to be using Ubuntu LTS anyway, especially if you're using ZFS, and especially if you're using it on a PC without ECC.
So, really, it's kind of a niche worry.
I speak as someone who had to have a nice guy from IBM/Lenovo visit recently with a very expensive (per GB, was free replacement for me) RAM chip when one of our blades BSOD'd for safety when it detected an ECC error.
Services were affected for about a minute until the replicas took over. We didn't trust the server that were serviced and made it pull back all the replicas from the other machines once it was back up, and checked them. And I'm part of a two-man team in a school, so it's hardly top-end stuff.
A single-bit flip in your RAID card onboard RAM could do untold damage.
A single-bit flip in your RAM isn't that much better.
If it matters to you, only run with ECC and monitoring tools enabled.
(Score: 2) by pendorbound on Friday February 19 2016, @01:56PM
In theory sure, ECC is nice, and ZFS makes some assumptions about RAM integrity. But in practice?
I've run ZFS for about 8 years with around 8TB pool size on non-ECC RAM with no troubles. OS was OpenSolaris, then FreeBSD, and finally Gentoo Linux for about the last four years. No lost pools, and no ZFS scrub errors that weren't immediately attributable to failing drives.
I've moved to ECC for the last year or so now because I got a deal on a server on eBay I couldn't turn down, but non-ECC RAM isn't going to make all your pr0n evaporate overnight when a cosmic ray hits your motherboard...
(Score: 2) by zeigerpuppy on Friday February 19 2016, @10:03PM
You've been lucky, there's good reasons why ZFS is very good at protecting data when using ECC RAM but very poor at doing so without.
https://pthree.org/2013/12/10/zfs-administration-appendix-c-why-you-should-use-ecc-ram/ [pthree.org]
(Score: 2) by zeigerpuppy on Friday February 19 2016, @10:21PM
This is the clearest description of why ZFS needs ECC that I have seen.
https://pthree.org/2013/12/10/zfs-administration-appendix-c-why-you-should-use-ecc-ram/ [pthree.org]
I've been running ZFS in two servers for the last 3 years.
It's been possible to install ZFSonLinux with little difficulty for at least this long (my system is based on Debian - no systemd)
It is awesome and I've had no major issues but potential users should also be aware that not all hard disk controllers are created equal.
ZFS must access the drives at a low level to do its checksumming.
Some controllers and drivers get in the way of this.
I have had problems with supermicro onboard SAS controllers for instance.
The LSI SAS controllers are awesome and quite cheap too.
Here's a bit more discussion: http://zfsguru.com/forum/buildingyourownzfsserver/21 [zfsguru.com]
In summary, ZFS is fantastic, reduces administration overhead and is mature on Linux. However, it requires a level of understanding that is not for the casual user. ECC is required and a good understanding of how caches and hardware commands interact. Also, if you're using SSD as an intents log, it needs to be battery backed up (Intel S3x00 for instance)
By all means use it, but let's put this issue to rest, ZFS should not be considered safe on any old hardware.