SoylentNews is people
SoylentNews
John Paul Wohlscheid over at It's FOSS takes a look at the ZFS file system and its capabilities. He mainly covers OpenZFS which is the fork made since Oracle bought and shut down Solaris which was the original host of ZFS. It features pooled storage with RAID-like capabilities, copy-on-write with snapshots, data integrity verification and automatic repair, and it can handle files up to 16 exabytes in size, with file systems of up to 256 quadrillion zettabytes in size should you have enough electricity to pull that off. Because it started development under a deliberately incompatible license, ZFS cannot be directly integrated in Linux. However, several distros work around that and provide packages for it. It has been ported to FreeBSD since 2008.
(Score: 4, Insightful) by pendorbound on Tuesday September 11 2018, @04:44PM (7 children)
ZFS is "Free" as in speech. It's just that its CDDL license is not compatible with GPL. You could make an argument that GPL is less free than CDDL since it's a restriction in the GPL which makes it impossible to use with CDDL, not the other way around. CDDL code is freely used in BSD-licensed software without any legal issues.
The fact that Sun staff are on-record saying that CDDL was written based on MPL explicitly because it made it GPL-incompatible and limited ZFS' usage with Linux kind of muddies the situation. Still, the CDDL license is free as in speech according to both the FSF and OSI. https://en.wikipedia.org/wiki/Common_Development_and_Distribution_License [wikipedia.org]
I went through a similar filesystem evolution as you did and landed on ZFS. I've been using it in home-production since 2008. Its built-in snapshot and checksumming support have saved my bacon several times in the face of both hardware and operator errors. I'd highly recommend giving it a look. The one caveat is look at the minimum hardware requirements, balk at how insane they are for a filesystem, and then make sure you comply with them anyways. ZFS does not operate well in RAM constrained environments.
(Score: 3, Informative) by VLM on Tuesday September 11 2018, @07:00PM (5 children)
The whole 1GB ram per 1TB of storage thing? Not really needed. Dedupe is memory hungry. If you're trying to do an office NAS for many (hundreds?) of users simultaneously, you'll want more than 1G per 1T for caching reasons.
You can't hurt anything by having more ram, thats for sure.
I'm typing this on a freebsd box with 32 GB of ram for fast compiles and only 250GB x2 of storage so its very nice, but not necessary.
(Score: 2) by Entropy on Tuesday September 11 2018, @07:44PM (4 children)
No. That's for dedup. (Deduplication) which is by no means necessary. I think ZFS doesn't work well if you have 512M of RAM or something, but over a gig should be fine.
(Score: 2) by pendorbound on Tuesday September 11 2018, @07:58PM
It's life & death critical for dedupe(*), but you're still going to want much more RAM than normal for ZFS. ZFS' ARC doesn't integrate (exactly) with Linux's normal file system caching. I've seen significant performance increases for fileserver and light database workloads by dedicating large chunks of RAM (16GB out of 96GB on the box) exclusively for ARC. It'll *work* without that, but ZFS is noticeably slower than other filesystems if it doesn't have enough ARC space available. Particularly with partial-block updates, having the rest of the block in ARC means ZFS doesn't have to go to disk to calculate the block checksum before writing out the new copy-on-write block. Running with insufficient ARC causes ZFS to frequently have to read an entire block in from disk before it can write an updated copy out, even if it was only changing one byte.
(*) Source: Once tried to enable dedupe on a pool with nowhere near enough RAM. Took over 96 hours to import the pool after a system crash as it rescanned the entire device for duplicate block counts before it was happy the pool was clean. Had to zfs send/receive to a new pool to flush out the dedupe setting and get a usable system.
(Score: 0) by Anonymous Coward on Wednesday September 12 2018, @10:18AM (1 child)
My home network media backup ZFS box had 2GB ram, I ran into occasional stability issues (unexplained random reboots) it now has 4GB and runs solid 24/7, i'd say throw as much RAM at it as your motherboard can handle.
Once I can source a cheap secondhand multi cpu server with over 32GB ram then I'll move the current disk pool over to it and consider firing up the (probably now badly needed) dedupe facility, the SD cards from the various family digital cameras and phones get regularly backed up to the server, and the networked home directories live there as well, so no doubt there are multiple copies of the same images and music files lurking on it.
(Score: 0) by Anonymous Coward on Wednesday September 12 2018, @10:42AM
I should have added in there, that this was during the testing phase, I ran the thing for a month and seriously hammered it, upped the beastie to 4GB, hammered it again for another couple of weeks before finally going live with it.
(Score: 2) by VLM on Wednesday September 12 2018, @11:09AM
Dedupe is almost never necessary. Under really weird conditions if you're running over 1000 almost identical virtual compute nodes (maybe a webhosting farm using virtualization?) then you can save some cash on storage. But under normal conditions you're basically trading high speed ram which is money and heat and energy intensive for slightly lower bulk storage which is cheap and getting cheaper; generally not a win.
A good analogy is dedupe is kinda like the old windows "autoexec on media insertion" which sounds nifty but turns out to be not so great overall.
(Score: 2) by hendrikboom on Tuesday September 11 2018, @09:33PM
I found the following:
https://www.reddit.com/r/DataHoarder/comments/5u3385/linus_tech_tips_unboxes_1_pb_of_seagate/ddrngar/ [reddit.com]
And also https://www.reddit.com/r/DataHoarder/comments/5u3385/linus_tech_tips_unboxes_1_pb_of_seagate/ddrh5iv/ [reddit.com]
So I now wonder what the *real* limits are on home-scale systems. In particular, suppose I have only a few terabytes. And a machine with only a half gigabyte of RAM. And used for nothing more bandwidth-intensive than streaming (compressed) video over a network to a laptop.
What I like about ZFS is its extreme resistance to data corruption. That's essential for long-term storage. My alternative seems to be btrfs. Currently I'm using ext4 on software-mirrored RAID, which isn't great at detecting data corruption.
-- hendrik