Maybe because XFS is well proven in certain critical environments, especially ones which might involve frequent manipulations of large quantities of small files (think mailserver or database). Indeed, some developers will tell you in the docs that they only support XFS or ext4 and that you're on your own if you choose something else.
Maybe because ZFS and BTRFS still have their bugs, small and big (hello RAID5 in BTRFS ;-) ) and can be heavily dependent on particular implementations.
Maybe because in a virtualised environment, you loose many of the benefits of ZFS or BTRFS (they like to see the raw disk, not some abstract notation).
Maybe because the combination of XFS/ext4 plus LVM is more than good enough for most people as that route still provides snapshots etc.
Maybe because BTRFS needs babysitting[1] (don't know about ZFS, I suspect it might).
Maybe because not everyone feels the need to "keep up with the Jonses".
As far as any of your reliability concerns, my experience with ZFS has been that it is amazingly reliable.
If I had to pick one place to put precious data that I needed to access for 10-20 years, ZFS would be it. Of course, I wouldn't put it in one place, so if I picked places to put that data, ZFS would definitely be one of them.
I've been using ZFS for over a decade, both for my personal storage systems and for work (largely backup servers), and despite heavy use and abuse, I've never once had data loss while using it. I've had some white-knuckle times, but in the end ZFS was able to recover all the data.
I say "abuse" because I was able to write a stress test program based on my usage that uncovered several ZFS+FUSE issues early in that port.
Likewise. Been using ZFS for over 10 years. Been through intermittent controller faults, backplane faults, cabling faults on a variety of ZFS, ext, and NTFS systems.
The one time I lost data from a ZFS system (which was a faulty controller silently writing garbage to multiple disks in the array), ZFS told me exactly which few files were uncorrectable, and I was able to restore them from backup.
Not one single second of doubt as to which storage system I want my data on.
That you're equating ZFS and BTRFS, and tarring ZFS with the issues of BTRFS, clearly shows how much you don't know about it. ZFS was production-stable more than 10 years ago.
> With btrfs you're still waiting for stuff to be properly implemented.
Redhat gave up on waiting, and is removing btrfs as a supported option in RHEL. Mostly, I assume, because the btrfs team has yet to ship something anyone in enterprise would be comfortable betting their customers' data on.
RedHat is not "just waiting" company for core component but hiring maintainers and strongly contributing company if needed. it's happened on XFS adoption.
I think the biggest issues with using ZFS as a linux user come down to
A) ZFS-on-Linux was always behind BSD ZFS until recently.
B) The license issue, so you have to compile it as a separate module using DKMS, which means having a compiler, the linux headers, etc etc. No where near as low friction as just choosing ext4 or btrfs.
> ZFS-on-Linux was always behind BSD ZFS until recently.
Sure, but the sort of people who need ZFS are the same sort of people who are doing their storage on a SAN, and so the sort of people for whom choice of OS (for the SAN itself) can be contingent on choice of storage layer, rather than the other way around.
This is half the reason ZFS-on-Linux didn't have much momentum: most ZFS deployments were happy to just run BSD on their SAN, then serve SAN volumes out over the network (or into a hypervisor cluster) for Linux systems to consume.
> ZFS-on-Linux was always behind BSD ZFS until recently.
FreeBSD was never really working on ZFS in a way ZOL was. Whole reason for the switch is because main contributor upstream used by FreeBSD stopped using ZFS.
I think there is a reason the part you quoted was written in the past tense. The situation has changed. But these things have momentum and people might have adopted FreeBSD years ago because of the former status quo, where FreeBSD had it integrated in the default install and Linux distros did not.
I think it's a disservice to even mention btrfs and zfs in the same breath. ZFS is a battle tested, enterprise ready, production grade filesystem and lvm.
BTRFS is a science experiment by comparison. They tried to copy zfs features because Oracle won't release it under GPL and did a fairly poor job of it. The fact we're this many years later and their RAID5 code still has total data loss bugs pretty much sums up btrfs.
Anecdotal, but I had a power outage at work last year and my relatively freshly-installed 18.04 system, for which I chose btrfs as the root filesystem, completely corrupted.
I managed to extract most of my data to another disk on another system, but it was pretty shocking. I haven't seen unfixable filesystem corruption like that since ext2 in 2001. I'd completely forgotten it was a thing, but there I was in 2019 trying to use a second linux box to salvage my data.
Of course, whenever I mention this the btrfs zealots always come out of the woodwork saying "you must not know what you're doing if you broke btrfs" or "you obviously didn't really try if you couldn't get it fixed" or a half-dozen other excuses, but in the end I've just completely given up on btrfs. It's just not worth the hassle.
Do people really create zpools within a VM for more than playing around?
The best practice is to point a VM at a zvol the way you might use LVM on bare metal to back storage for VMs.
Right. The storage pool for VMs should be zfs (if you’re going to use zfs as an assumption) while the VMs should be something else unless you’re doing something specific to use zfs in the VM. Zfs is engineered to almost turn a system into a file system server in exclusion so use it for its strengths appropriately.
I would avoid lumping BTRFS in with ZFS. The two are very different.
> Maybe because XFS is well proven in certain critical environments, especially ones which might involve frequent manipulations of large quantities of small files (think mailserver or database). Indeed, some developers will tell you in the docs that they only support XFS or ext4 and that you're on your own if you choose something else.
XFS is great, except it doesn't do any of the things that ZFS does well. For example: snapshots (to the fantastic extent that ZFS does), multi-volume support, shared storage pooling, etc. The only thing XFS offers that ZFS doesn't is support for reflink copies on Linux (i.e. "cp -R --reflink=always src dst" for near-instantaneous copies of large directories with no extra disk usage).
> Maybe because ZFS and BTRFS still have their bugs, small and big (hello RAID5 in BTRFS ;-) ) and can be heavily dependent on particular implementations.
ZFS has been, in my experience, amazingly bug-free over the last however long I've been using it (5 years in production, more personally). ZFS has working and reliable RAID support; only BTRFS is lacking it.
BTRFS, on a stock Ubuntu install with default settings, corrupted after a power outage, and could only recover most of my data, but not all. I haven't had filesystem-related data loss since ext2 in 2001.
Also, for everyone except people running Solaris, there is (now) effectively one implementation, and it works fine.
> Maybe because in a virtualised environment, you loose many of the benefits of ZFS or BTRFS (they like to see the raw disk, not some abstract notation).
You still get point-in-time snapshots on a copy-on-write filesystem, which can support different block sizes per subvolume, on-the-fly compression, and incremental or full exports, all from a shared, extendable storage pool. Those alone are huge enough to justify it for me.
ZFS wants to see the underlying disks so that it can make sane judgements about block allocation, alignment, etc. On a VM, that shouldn't matter, because your host filesystem/SAN/etc. should be doing that anyway.
> Maybe because the combination of XFS/ext4 plus LVM is more than good enough for most people as that route still provides snapshots etc.
LVM snapshots are awful from a performance standpoint.
In ZFS, when you create a snapshot, any new data is written to new blocks (read the old block, make the change, write it to the new block), so your snapshot points to the old blocks on disk.
From what I can tell[1], LVM snapshots mean that when you write data to a block, LVM reads that block, writes it somewhere else, and then updates the original block in-place. This makes every write a synchronous write, because you have to write the old data to its new block, sync to make sure it actually gets stored, and then do your new write.
BTRFS snapshots cannot be recursive (i.e. you cannot snapshot /data/ and /data/gitlab and /data/svn and /data/backups atomically). They argue that this is a feature, but I would consider it a massive bug. LVM snapshots by their nature are not recursive because LVM has no concept that I can find of nesting.
> Maybe because BTRFS needs babysitting[1] (don't know about ZFS, I suspect it might).
ZFS, as far as I can tell, does not need babysitting. I used it for storing backups at my last job; we had a QNAP NAS exporting over iSCSI, ZFS was using the iSCSI storage, and we wrote data to it. Nothing ever went wrong, and I literally never checked on it unless the power went out or something of the sort. It was three years before I ever got around to setting up monitoring for it, because literally nothing ever went wrong so I completely forgot that there was anything to monitor. Eventually I did set it up, but it never went off. Literally set it and forget it.
In summary: ZFS is fantastic, and basically never needs to be taken care of. It does its thing and it works great, and that's it. BTRFS has corrupted itself arbitrarily after a power outage (which I thought we'd solved with ReiserFS back in 2001), LVM snapshots are slow and awful, and the tooling for LVM and BTRFS is a usability disaster while also losing a lot of features that ZFS has (like on-the-fly compression, block deduplication, etc.).
ZFS is great but CoW makes performance bad on some situation in theory. So there are some situations better to use XFS (without using LVM snapshot all time).
Did you include BTRFS in here just to mentioned all weak points of BTRFS and act like they also true for ZFS? Because that's not true at all.
> Maybe because in a virtualised environment, you loose many of the benefits of ZFS or BTRFS (they like to see the raw disk, not some abstract notation).
ZVOL exist...You can snapshot, close, send/recv it just like dataset. You can fine-tune it like any other dataset. ZVOL works just fine for VMs.
> Maybe because ZFS and BTRFS still have their bugs, small and big (hello RAID5 in BTRFS ;-) ) and can be heavily dependent on particular implementations.
Everything has bugs. Again, you put BTRFS here just to drive your point like if it's try BTRFS then it's true for ZFS?
> Maybe because BTRFS needs babysitting[1] (don't know about ZFS, I suspect it might).
Same as above.
> Maybe because the combination of XFS/ext4 plus LVM is more than good enough for most people as that route still provides snapshots etc.
Very different kind of snapshots...
I've been using ZFS on my desktop and laptops for over a decade now. You gotta make really kick-ass FS for me to switch off ZFS even for desktop.
Why people aren't just using ZFS (or BTRFS) ?
Well...
Maybe because XFS is well proven in certain critical environments, especially ones which might involve frequent manipulations of large quantities of small files (think mailserver or database). Indeed, some developers will tell you in the docs that they only support XFS or ext4 and that you're on your own if you choose something else.
Maybe because ZFS and BTRFS still have their bugs, small and big (hello RAID5 in BTRFS ;-) ) and can be heavily dependent on particular implementations.
Maybe because in a virtualised environment, you loose many of the benefits of ZFS or BTRFS (they like to see the raw disk, not some abstract notation).
Maybe because the combination of XFS/ext4 plus LVM is more than good enough for most people as that route still provides snapshots etc.
Maybe because BTRFS needs babysitting[1] (don't know about ZFS, I suspect it might).
Maybe because not everyone feels the need to "keep up with the Jonses".
[1] https://www.youtube.com/watch?v=8YUC-r1aXAc