The nice feller over at phoronix brings us this handy to have bit of info:
It turns out the RAID5 and RAID6 code for the Btrfs file-system's built-in RAID support is faulty and users should not be making use of it if you care about your data.
There has been this mailing list thread since the end of July about Btrfs scrub recalculating the wrong parity in RAID5. The wrong parity and unrecoverable errors has been confirmed by multiple parties. The Btrfs RAID 5/6 code has been called as much as fatally flawed -- "more or less fatally flawed, and a full scrap and rewrite to an entirely different raid56 mode on-disk format may be necessary to fix it. And what's even clearer is that people /really/ shouldn't be using raid56 mode for anything but testing with throw-away data, at this point. Anything else is simply irresponsible."
Just as well I haven't gotten around to trying it then.
(Score: 5, Informative) by NCommander on Sunday August 07 2016, @02:54PM
btrfs amazes me in the fact its a pile of crap that has been in development for almost a decade. It's the Duke Nukem Forever of filesystems.
To put this in context, ZFS development was started in 2001, and shipped in 2005, and at the time of its release, ZFS was likely the most advanced filesystem in the world. btrfs on the other hand manages to be NIH to a horrifying extent (Oracle owns ZFS and is also the chief contributor to btrfs), and manages to fuck its strengths.
On the fly conversion of RAID levels is supposed to be a killer feature of btrfs. To hear its not only broken, but broken to the extent that the on-disk format needs to change well. That's a lot of fail.
Still always moving
(Score: 4, Interesting) by Post-Nihilist on Sunday August 07 2016, @03:43PM
The number of paid full time engineer working on ZFS was borderline ridiculous1 Compared to ZFS, brtfs is resources starved since the development started. It's no wonder that they are progressing so slowly...
1-I knew this as I was working as a campus ambassador for SUN at that time. My job was to shill for SUN by giving OpenSolaris DVD and T-Shirt and organizing technical talk with SUN's engineers. But most importantly, the job was about filling reports about filling reports for 4 different managers. It is no wonder that SUN imploded on its own mass just before getting sold to Oracle....
Be like us, be different, be a nihilist!!!
(Score: 4, Interesting) by NCommander on Sunday August 07 2016, @05:15PM
Sun's management was famously clueless. I've liked how the (in)famous USENIX talk on the subject, https://www.youtube.com/watch?v=-zRN7XLCRhc [youtube.com] which had a rather epic rant on the subject, describing the situation akin to Somalia warlords.
That being said, I miss Sun. I rather deal with incompetence vs. malice, and that's about the best way I can describe Oracle's practices. At leas Sun both gave back, defined open standards, and from a technical standpoint, pretty damn decent.
Still always moving
(Score: 3, Interesting) by Post-Nihilist on Sunday August 07 2016, @07:01PM
Sun's management was well past the cluelessness point, they were deep into the Kafka zone. When I got bored by that chearleading job, I did a small experiment: Instead of quitting I kept submitting reports on how i spent my time fillings the report requested by the other managers, naively thinking that they would catch on the silliness of the situation. I was so naive...all I managed to achieve is getting some praise for my accurate record keeping, at which point I formally resigned.
Be like us, be different, be a nihilist!!!
(Score: 3, Insightful) by fleg on Monday August 08 2016, @04:13AM
>Sun's management was famously clueless.
i was there in the late 80's/early 90's in the uk, they were fine up until around the time they merged their uk and european headquarters into one building, suddenly there were managers everywhere and it went downhill fast in terms of culture. it was also around that time they unbundled the c compiler from the OS, which a lot of the old hands were very unhappy about. i think the writing was on the wall from that point.
(Score: 2) by Post-Nihilist on Monday August 08 2016, @04:43AM
Let's have a drink to a decade of driftings managers
Be like us, be different, be a nihilist!!!
(Score: 2) by fleg on Monday August 08 2016, @05:29AM
mines a large single malt!
does the underbelly of your ivory tower have a bar?
(Score: 3, Funny) by driverless on Monday August 08 2016, @06:23AM
It is no wonder that SUN imploded on its own mass just before getting sold to Oracle....
So Oracle essentially bought a neutron star?
(Score: 4, Funny) by Anonymous Coward on Sunday August 07 2016, @05:00PM
btrfs- "I Can't Believe It's Not Better"®
Well actually I can... ;)
(Score: 0) by Anonymous Coward on Monday August 08 2016, @10:32AM
I always read "bitrot-filesystem"...
(Score: 4, Interesting) by Marand on Sunday August 07 2016, @05:54PM
(Oracle owns ZFS and is also the chief contributor to btrfs)
Sounds like this might be part of the problem. Oracle started Btrfs in 2007, then acquired Sun (and ZFS) in 2010. Once Oracle became the owner of the filesystem they were trying to ape, what reason was there to continue putting heavy development into Btrfs?
Sure, Oracle kept some token effort going on there, but the incentive to make it usable for their needs died once they became owner of the filesystem they were trying to ape, and most everyone else seems to be fine using ZFS, ext3/4, or one of the others like XFS.
(Score: 1) by Francis on Sunday August 07 2016, @09:33PM
Apart from the nutters that insist upon using draconian licensing schemes to control developer behavior, was there every really any point to the filesystem? It seems to me that the problem with Linux filesystems isn't that there are too few of them, but that there's so many that they often times aren't completed before being incorporated. I'm still allergic to EXT4 because when I first tried using it, it would eat the entire install if the system crashed. And the system crashed every time I loaded it, so I'd have to reinstall every single time I'd want to use it.
Needless to say, I stopped using it and avoid it like the plague, but having a ton of filesystems available that aren't necessarily read for primetime isn't really a good idea. On some level if a filesystem isn't production ready after a certain number of years, it's probably not going to be ready ever. Either it's too niche to attract developers or it's gone off the rails by trying to do too much.
(Score: 0) by Anonymous Coward on Sunday August 07 2016, @09:57PM
crashing an ext4 filesystem by having your seagate harddrive overheat does the same thing.
And yes I have tried with ext2/3 and the same thing doesn't happen... HOWEVER running ext3 on the ext4 driver DOES cause similiar corruption.
So now all my archival partitions stay ext2 and only my OS drives end up ext4 (it does offer better performance usually, but the reliability is definitely shite. Plus it is unreliable thanks to feature changes if you ever need to use it in a system with an older kernel! Same deal with ext2 if you aren't careful about mkfs options!) Another issue to be aware of: the ext3 driver breaks on certain options depending on if you're using an i686 vs an x86_64 driver. The ext4 driver will load the i686 formatted partitions on x86_64, but will if you switch to an i686 kernel, whereas the ext3 driver (when still present in the kernel) would load the partition on i686 but fail if you attempted to load it on x86_64 with an invalid filesystem options error. Why is this important? Because it meant either changing the fstab entry before boot, or manually setting the filesystem driver on the kernel commandline to ensure it would boot smoothly. Forget to do it, especially on a remote system, and you end up with your server hung.
(Score: 2) by Marand on Sunday August 07 2016, @11:10PM
Apart from the nutters that insist upon using draconian licensing schemes to control developer behavior, was there every really any point to the filesystem?
It started inside Oracle, so the reasoning was probably as simple as "Oracle wanted ZFS but Sun has it." No idea why they didn't just abandon ship as soon as they bought Sun, though; maybe a passionate dev in the company convinced someone to keep working on it fort he future or something.
The appeal for others was either "ZFS licensing isn't compatible with the Linux kernel but we want ZFS features" or "we want stuff that ext[2|3|4] doesn't have", but it's apparently not been enough. It may well be a good long term goal, and if that's what someone wants to work on good for them, but it still isn't ready for everyday use and I'm amazed people actually suggest it for such. :/
I've got nothing against developing new filesystems, and I don't think it's a problem that the kernel supports a bunch of niche ones, because use cases are different and sometimes a good specialised FS isn't suitable for general use (or vice-versa). The problem here with btrs is that filesystems are too damn important to put into production use when still immature, and for whatever reason people seem to think it's production-ready when it's clearly not.
(Score: 3, Flamebait) by Entropy on Sunday August 07 2016, @09:38PM
ZFS has everything BTRFS has, but in a vastly superior interface. Last I checked each snapshot in BTRFS needed to be mounted(why?) making your mount screen look like a tragedy. Also writable snapshots? Why would I ever want to write to a snapshot?! The point of a snapshot is it's frozen.
Of course ZFS has had a superior version of raid5 forever now, block level replication based on snapshots, and physical devices(zvols) that can be used as virtual machine hard disks.(that can also be replicated!)
No doubt BTRFS could one day achieve all those things, of course...but that day is far in the future.
(Score: 0) by Anonymous Coward on Sunday August 07 2016, @02:57PM
This is why I laugh at idiots that think using RAID means you don't need backups.
(Score: 2, Insightful) by Anonymous Coward on Sunday August 07 2016, @03:06PM
Yes, RAID is not a replacement for backups. Both strategies should be used. I don't see how that's relevant.
Only a fool would use btrfs for production. Has btrfs ever had a stable release?
(Score: 2) by VLM on Sunday August 07 2016, @04:48PM
Has btrfs ever had a stable release?
Yeah, it shipped under the code name "ZFS on FreeBSD" back in '07
Oh Snap that was brutal. Someday though BTRFS in like 2027 might catch up to ZFS from 2007, well, someday. Maybe.
Seriously though, systemd, best thing ever, without that I wouldn't be using freebsd and discovering zfs and pf and stuff.
(Score: 1, Touché) by Anonymous Coward on Monday August 08 2016, @06:47AM
Only a fool would use btrfs for production. Has btrfs ever had a stable release?
Maybe somehow these bunch have a stable btrfs release: https://www.synology.com/en-global/dsm/Btrfs [synology.com]
:p
(Score: 0) by Anonymous Coward on Sunday August 07 2016, @03:34PM
Reminds me of Joshua Woodruff ... and Zuora.com, where that very claim was asserted, by management.
Idiots. Working there was like strewing pearls before swine.
(Score: 2) by rleigh on Sunday August 07 2016, @06:38PM
Yep. I argued with my immediate boss about this at a previous job. He wouldn't have any of it, after all, the systems they had sold were all working fine and no customer had complained. A while later, a thunderstorm over a customer's shop caused a lightning strike, which went through the phone line, modem, serial port, mainboard, through the HDD cables and blew up the *same* chip on the HDD controller board on both HDDs--I saw the charred remains personally when we got them sent back. Total unrecoverable dataloss, even after sending the drives to a specialist data recovery firm. In the next revision of the hardware, we replaced one HDD with a USB port and had the customers do a nightly full backup and rotate them; they could then transfer that data elsewhere as they liked. But it took a huge disaster which showed up the incompetence, along with the financial/legal consequences, to effect any change; simple reason and logic was not sufficient, sadly. Before that, they were convinced that two disks in the same system were a backup, and told customers this. After, they finally realised how uniquely vulnerable the data is when it's all in that one place. They used to tell people to remove one (in a caddy) and put it in a safe; but when it's in place during regular operation there's simply no recovery from failure.
(Score: 2, Informative) by Anonymous Coward on Monday August 08 2016, @07:33AM
Morons.
In my experience, the primary use of backups is not hardware faults (including lightning-induced hardware faults), but human or software error overwriting or deleting stuff.
In this situation, RAID will ensure that the mistake is propagated to every drive within milliseconds.
Snapshots can remedy the human factor somewhat (as long as the person making the mistake doesn't have root access), but it doesn't protect against software problems (including controller and file system drivers).
(Score: -1, Flamebait) by Anonymous Coward on Sunday August 07 2016, @03:12PM
Just imagine what would happen if GitHub imploded. Goodbye Linux. Goodbye open source. So many careers ruined.
(Score: 2) by hendrikboom on Monday August 08 2016, @02:30AM
Well, for every project that's actually under development (and few usable ones aren't), the developers will have their own git repositories. So all would not be lost. What would be lost is the ease of finding projects.
-- hendrik
(Score: 2, Interesting) by Anonymous Coward on Sunday August 07 2016, @03:30PM
This should be the title:
Why? Because it is not just the RAID code that is hosed, it is ALL of BTRFS that is hosed.
Personal experience, large disk array, running RAID1 (mirroring). Filesystem would run for months no problem, then, suddenly, hose itself, with no recovery beyond re-init and reload. It did this twice, after #2, it was replaced by ext4. The ext4 filesystem's been running for several years now, never hosed itself
Prior to the second hosing of the large disk array, a workstation system was setup with BTRFS in plain mode (no raid, nothing special, just a filesystem). Came home from work one day, and it too had hosed itself with no recovery beyond a restore from backup. Well, the restore from backup was into ext4 as well, and that FS ran until the disk holding it gave up (hardware failure)
So, after three hosings, this Linux user will never use BTRFS for anything ever again. And I recommend you do not use it for anything either. It is, at best, still pre alpha quality code
(Score: 1, Informative) by Anonymous Coward on Sunday August 07 2016, @06:01PM
Different AC here, but I've had similar experiences with btrfs killing itself. It was on a fresh install of opensuse, I believe, and it didn't last a week before it wouldn't boot. Nothing fancy, no raid or the like, just pure defaults, hence why I used it at all. Didn't inspire confidence in the file system or the distro that picked it.
(Score: 2) by rleigh on Sunday August 07 2016, @06:48PM
You most likely hit the unbalancing issue which makes it go read-only. I've seen this repeatedly. A rebalance would fix it, though it takes ages and kills the system's performance while it grinds away. Note I use "issue" rather than "bug" because it's more of a design flaw, though the implementation is undoubtedly also defective. Search for "btrfs read only balance" for a whole lot of detail about it. When I initially read that SuSE was going to default to Btrfs my initial reaction was utter disbelief; how on earth they justified doing that I hate to think.
(Score: 0) by Anonymous Coward on Sunday August 07 2016, @08:54PM
It's also default on SUSE commercial distro that is supported until 2030 or something, so .... but you can also select Ext4 or XFS too. But some influential people were pushing for it so SUSE is stuck with it now - hopefully it's not too buggy!
(Score: 0) by Anonymous Coward on Monday August 08 2016, @10:45PM
Commercial distro, eh? As in commercial support? Great way to get people to call you for support would be to have the file system hose itself seemingly at random, especially at the times they are must desperate.
(Score: 0) by Anonymous Coward on Sunday August 07 2016, @09:12PM
Nope, in my SuSE case, it was a single disk VM and mounting it into another VM with btrfs support reported errors when running through the diagnostic procedure on the wiki. But, if you have stupid things pop up like rebalancing in normal use in less than a week on a single disk after light use, assuming that was even the problem instead of actual corruption, then btrfs is total garbage and SuSE is too for having garbage defaults.
(Score: 2) by rleigh on Sunday August 07 2016, @06:23PM
I've had bad experiences with it since the start. I wrote up some of the issues I had in this related thread: https://news.ycombinator.com/item?id=12232907#12233154 [ycombinator.com] These horrific problems are by no means isolated instances. Even if the RAID1 code is now fixed, I've lost all trust in it. There's just too much stuff which is fundamentally broken, and that's just not acceptable in a filesystem. I'm simply not prepared to lose any more data or downtime to it. I had high hopes for it, but it's turned into a seriously bad joke. Too many times people have told me, "oh, you need to upgrade the the latest kernel for $fix". How many times do you consider it acceptable for me to lose my data? Sorry, but it's not ready for production use, and it never has been.
Over the last 2.5 years, I've been using ZFS on FreeBSD. What an absolute revelation and joy to use after 15 years of mdraid and LVM (and Btrfs). I wish I'd discovered it years before; I've got systemd to thank for that, and I'm genuinely happy that it gave me the push to test the waters outside the (increasingly insular) Linux sphere.
But ZFS is getting much better supported on Linux as well. With Ubuntu 16.04, it's possible to boot directly to a root filesystem on ZFS, with /boot on ZFS. It's still a little rough--not supported directly by the installer--but all the pieces are there in GRUB, the initramfs, the init scripts etc. With a little pain and a few tries and failures, I got it booting directly with EFI and GRUB2. The only missing piece to get this generally usable is an option in the installer like you have with FreeBSD, and then it will be a piece of cake to get up and running.
To be fair though, this isn't all the fault of the Btrfs developers. The number of uninformed fanboys parrotting how great it was and how we should all be using it belied the reality that those of us who heavily tested it for years discovered to our cost. Not so long ago the slightest criticism or caution was jumped upon in some quarters as though it was some sort of betrayal. No, it was simple common sense borne out of actual informed real-world experience with it! Blind faith in it won't make it magically reliable and stop it toasting all your data! I think that these people did a great disservice to anyone who followed their advice, particularly if they suffered from dataloss.
For anyone interested in trying out ZFS on Linux as a rootfs (sorry about the mangled spacing, it should be more readable). Note it also includes a zvol as the swap device.
% lsb_release -cr
Release: 16.04
Codename: xenial
% sudo zfs list
NAME USED AVAIL REFER MOUNTPOINT
fdata 134G 315G 96K /fdata
fdata/old-root-backup 8.85G 315G 8.85G /fdata/old-root-backup
fdata/rleigh 156M 315G 96K /fdata/rleigh
fdata/rleigh/clion 156M 315G 156M /fdata/rleigh/clion
fdata/schroot 201M 315G 96K /fdata/schroot
fdata/schroot/sid 200M 315G 200M /fdata/schroot/sid
fdata/vmware 125G 315G 125G /fdata/vmware
rpool 21.9G 85.7G 96K none
rpool/ROOT 7.53G 85.7G 96K none
rpool/ROOT/default 7.53G 85.7G 7.25G /
rpool/home 308K 85.7G 96K none
rpool/home/root 212K 85.7G 132K /root
rpool/opt 1.74G 85.7G 475M /opt
rpool/opt/steam 1.27G 85.7G 1.27G /opt/steam
rpool/swap 8.50G 93.7G 510M -
rpool/var 4.06G 85.7G 96K none
rpool/var/cache 4.06G 85.7G 4.02G /var/cache
rpool/var/log 3.06M 85.7G 2.95M /var/log
rpool/var/spool 168K 85.7G 104K /var/spool
rpool/var/tmp 200K 85.7G 128K /var/tmp
(Score: 3, Interesting) by frojack on Sunday August 07 2016, @06:49PM
Same here. With the Second data loss (slow learner) I kicked BTRFS off my machines.
It solves a lot of problems that no one new existed, and tries to be the systemd of file systems.
You never know how much free space you actually have. And deleting data can actually take more space in unexpected ways as the cows come home to roost. (yeah, mixed that metaphor on purpose).
It really is a mess, and Opensuse, true to form, pushed it as the default filesystem. For For Joe User, there is zero advantage. For Big Data user there is every reason to avoid this crap.
ext4 and xfs for me. (And software raid).
No, you are mistaken. I've always had this sig.
(Score: 2) by darkfeline on Sunday August 07 2016, @09:02PM
For a different anecdotal viewpoint, I have been using BTRFS (without RAID) for six months now and have had zero issues, with constant writing and deleting of large media files with LUKS encryption.
Join the SDF Public Access UNIX System today!
(Score: 5, Interesting) by rleigh on Sunday August 07 2016, @09:34PM
This is a "relatively safe" mode of operation, since you're not using any of the RAID code, and it's also much better tested in this mode. But, the thing with filesystems is that it's all fine when there are no problems, since if there are no faults you're not exercising any of the failure codepaths. The true test is when you have hardware glitches on the disk, faulty connectors, or memory errors etc. It's these failure codepaths which are where Btrfs fails so spectacularly, the very parts we absolutely require to work correctly. And having been exposed to a whole bunch of them, I can tell you right now that it's unsuitable for production use if you really care about your data, because relying on Btrfs is like playing Russian roulette! Lightweight use will *seem* OK, and for the most part *will* be OK. Start loading it with multiple parallel users, concurrent snapshots, and it will fail *really* quickly: total time from clean new filesystem to broken and unbalanced for me, the record is: every 18 hours with a few tens of concurrent snapshots and jobs. This is utterly thrashing the filesystem, so with lighter use that time will be significantly more than 18 hours: a week, several months or several years. But the fact that it will simply stop working at some undefined future point is absurd. That's not hardware related; it's a massive design and implementation *fail*. I can thrash an ext3, xfs or other "simple" filesystem in a similar manner for *years* with every expectation that it will continue to work absent any hardware problems. Filesystems demand perfection in their implementation more than any other piece of software; we entrust them with our data. Btrfs has failed at this, badly, ever since its creation. I've been using it since the start. We hoped it would stabilise. It didn't. It's still not trustworthy, and "no issues for six months" really means little when you look at the shocking flaws in it.