Correct me if I'm wrong, but I can't add vdevs to a RAIDZ (RAID5/6ish) system in ZFS, which are probably the most common configurations for home NAS systems.
I'd love to run ZFS, but I can't because I need to be able to add more drives as I buy them.
But it doesn't look like there's been any movement on an implementation and it seems like it's high effort and mostly home users who want this, not enterprises who might be willing to pay for.
correct, the number of devices in a raidz1/2/3 vdev cannot change. However you can replace all of the drives one at a time with higher capacity drives and most zfs installs will grow your vdev.
You could also add another raidz vdev if you have the space in the server, eventually they should level out depending on workload.
"Best" solution, read most expensive, would be to copy the data over to a new server you've configured for the new load/capacity.
I add two at a time without much problem, mind you I have a rather extreme setup with an external 12-bay enclosure but I feel most people building a NAS instead of buying an off-the-shelf system from Synology/Drobo aren't looking to cut costs or corners.
A 4TB Seagate IronWolf drive costs $129 off Amazon, buying two to add as a new mirrored vdev to my TrueNAS box isn't outrageous.
You're limited then to only mirroring, you can't do RAID6 or similar on a massive array where you're only giving up 2/10 drives space but can have a 2 disk failure. Seems like a lot of wasted disks when the only thing I'm dependent on the NAS for is not losing my TV collection. Right now I know if that if I go out and buy 2 more 4 TB disk I get 8 TB more in my NAS and at the price of an only slightly increased risk of having more than 2 drives fail at once. That's probably my favorite feature of RAID6 for a home NAS.
I actually went custom because those off-the-shelf boxes are either very expensive or have weak CPUs so can't be used for video transcoding very well - it's cheaper to build it yourself, much cheaper if you already have old hardware to dedicate to the task.
Different use cases I suppose, my FreeNAS box stores my video collection and all the usual stuff but I've also got all the VM's that run my home network stored on it - performance+resliver times are a lot more important to me than storage efficiency.
All of my virtual disks are hosted off my FreeNAS, I've got a direct 10GbE link between it and my oVirt host - raw throughput isn't so much the issue most of the time as IOPS are, I've got a local gitlab instance, OpenShift, some PostgreSQL databases, etc. and they like to hammer the crap out of my storage when in use.
Having an L2ARC helps out quite a bit, but only having 32GB of memory and wanting to keep most of it for the L1ARC means I still hit my spinning disks regularly (and mirrored vdev's help read IOPS tremendously in this case).
Reducing the cost of a custom system isn't irrational.
My own personal budget is very limited, buying two IronWolf HDDs in this case is easily a good chunk of my monthly income. If I can build a NAS that can expand with single drives as needed, it's more cost effective for me.
And I imagine a lot of others have the same problem.
In the end, by buying 2 drives when a single drive could have solved the problem equally well: expanding your space by 4TB, is wasting money. Period.
I'd love to run ZFS, but I can't because I need to be able to add more drives as I buy them.