John Paul Wohlscheid over at It's FOSS takes a look at the ZFS file system and its capabilities. He mainly covers OpenZFS which is the fork made since Oracle bought and shut down Solaris which was the original host of ZFS. It features pooled storage with RAID-like capabilities, copy-on-write with snapshots, data integrity verification and automatic repair, and it can handle files up to 16 exabytes in size, with file systems of up to 256 quadrillion zettabytes in size should you have enough electricity to pull that off. Because it started development under a deliberately incompatible license, ZFS cannot be directly integrated in Linux. However, several distros work around that and provide packages for it. It has been ported to FreeBSD since 2008.
(Score: 2) by VLM on Wednesday September 12 2018, @11:09AM
Dedupe is almost never necessary. Under really weird conditions if you're running over 1000 almost identical virtual compute nodes (maybe a webhosting farm using virtualization?) then you can save some cash on storage. But under normal conditions you're basically trading high speed ram which is money and heat and energy intensive for slightly lower bulk storage which is cheap and getting cheaper; generally not a win.
A good analogy is dedupe is kinda like the old windows "autoexec on media insertion" which sounds nifty but turns out to be not so great overall.