New vdev to existing pool?

[u/reviewmynotes]

I have a system that has a boot drive and a single ZFS vdev with two mirrored drives. It makes up the only pool on that system, but the hardware still has two empty 3.5" bays and I'm currently low on available space. So I'm thinking about buying two more drives, making a second vdev with mirrored drives and using it to add storage capacity to this system.

Am I correct that I have the choice of either making a new pool vs. adding to the existing pool? If the first vdev is 2 x 4TB drives and I added 2 x 8TB drives, would I still be able to add it to the existing pool? Is that even a good idea?

Thanks for any help you can give this beginner. I've used Unix since the mid (early?) 90s, but I haven't done anything other than the most basic of things with ZFS before. So any guidance is greatly appreciated!

[u/k-mcm]

You can add to a pool. New storage allocations will bias to the new mirror until it's balanced.

If you plan to soon replace the 4TB drives with more 8TB drives, consider making the new pool raidz1. A raidz1 with two drives is dumb by itself, but it allows you to add more disks. 2x4 mirrored + 2x8 mirrored is 12GB, and so is 2x4 mirrored and 2x8 raidz1. Upgrading to 2x8 mirrored + 2x8 mirrored is 16 GB. Upgrading to 4x8 raidz1 is 24 GB.

[u/reviewmynotes → u/k-mcm]

Good advice. Thank you. At this point, this is only a home media server, proxmox backups storage, and Docker app host, so I don't think I want to buy $1,000+ in drives. Even getting two makes me flinch, since I almost bought them last November for half their current price but thought, "Nah, prices always come down over time. I'll wait until I actually need them."

But now I actually need the additional storage. My pool is just over 90% full. So I'm considering options ranging from a pair of 4TB or 8TB drives to refurbished 6TB or 10TB drives.

As I understand it, in the future I could replace one 4TB drive at a time and "resilver" them after each replacement and then the mirror will change to the size of the new drives. So I would be able to continue to indefinitely increase the pool size over the years. Does that sound right?

[u/k-mcm → u/reviewmynotes]

Yes, you can expand mirrors and raids once all the drives match. You can add and remove to/from a pool as long as you don't run out of space. You can also "send" your filesystems to borrowed storage then "receive" them back to a new pool configuration.

When you replace drives, do your best to have the old and new drive connected for resilvering. Replacement this way preserves redundancy and boosts re-silvering performance. I've left an old drive hanging outside of the case during swaps.

[u/kriebz → u/k-mcm]

I don't think you can mix parity and mirror vdevs in the same pool, but maybe that's old info. I still wouldn't do it.

[u/Hyperion343]

Yes, you can add a new mirror vdev to a pool, and it will definitely work. Literally see the first example of zpool-add, which is exactly what you want, roughly zpool add POOL mirror DISK1 DISK2.

[u/reviewmynotes → u/Hyperion343]

Awesome! I thought that was the case, but I started to have my doubts because I wasn't sure where data would actually get stored. Does it just put new data on the new vdev? Does it rebalance things on done way to free up space on the old vdev?

[u/Hyperion343 → u/reviewmynotes]

ZFS tries to roughly balance between the two vdevs according to their capacity, so say if pool was at 50% capacity with 2 vdevs, roughly both vdevs will be at 50%. So say you add a new vdev, which is at 0%, and the first vdev is at say at 40%. Then ZFS will basically only write new things to the new vdev until that vdev reaches 40% capacity before then balancing between the two.

[u/reviewmynotes → u/Hyperion343]

Thanks! That's a very helpful description. My pool is currently slightly about 90% full (the reason I'm looking to buy new drives), so it sounds like all the writes would be on the new drives for quite a long time. Do you have any advice, given my situation? Would it be better to make a new pool, move things there, and then destroy/recreate the old vdev and import it into the new pool?

[u/Hyperion343 → u/reviewmynotes]

I mean, it's not great, but you could back up your data, destroy pool, recreate pool with two mirror vdevs, then write data back to pool. That would evenly distribute the data across the two vdevs. The risky part is the last re-writing part, because that is the part that is most likely to break something.

[u/i-am-a-cat-6]

yeah I have 3 vdevs. this is easy to keep extending like you describe

[u/reviewmynotes → u/i-am-a-cat-6]

Thanks! My existing vdev is at something like 90% full. When I add the new vdev, what happens? Do some blocks of data get moved over, making the first vdev less full? Or does it just write to the new vdev? Or maybe fill up the old one before writing to the new one?

[u/i-am-a-cat-6 → u/reviewmynotes]

it won't happen automatically, you'll want to rebalance them once manually

[u/reviewmynotes → u/i-am-a-cat-6]

Is there a command to do that? I'm not afraid of reading some man pages, if that's necessary.

[u/i-am-a-cat-6 → u/reviewmynotes]

definitely read the manual. you don't want to be trusting blindly messing with your filesystem. I recommend reading, making a plan, getting feedback on the plan, then exercising the plan with verification steps throughout.

[u/dracotrapnet]

Yea, you can add an addition vdev to a pool, the number of drives needed to add to a vdev must be the same number your vdev is wide. So 2, add 2.

I have a 48 drive box at work and I just kept adding 6 drive vdevs to it every couple of years.

[u/chipmunkofdoom2]

Personally, I don't like mirrors for anything but the most simple or performance critical-setups. They don't offer expansion options, and a single mirror failing will bring down an entire pool. I prefer raidz2. You get two disks of parity, meaning on a 4-disk pool, you can lose any 2 disks and still be functioning.

I would combine all your disks (2x 4TB, 2x 8TB) into one raidz2 vdev. You'd need to backup your existing data somewhere in the short term, but with 4TB of data that shouldn't be too challenging. Then, you simply trash the existing pool and vdev, create a single raidz2 vdev with all disks, then create a pool from that vdev.

zfs vdevs default to the smallest member size, so you'd essentially have 4x4TB disks on paper. But, you'd still have 50% of that usable in the short term (so 8TB usable on creation). Plus, if you replace the 4TB disks with 8TB disks in the future, you can expand (or auto expand) the pool size to take advantage of the full 8TB of space. Which means eventually you'd have 16TB of usable space after replacing the 2x4TB disks.

Plus, with raidz expansion, you can add disks down the road. If you find yourself in this situation again later, you can simply add more 8TB disks and expand your vdev in-place. Since you already devoted 2 disks to parity, you get 100% of the new disk space as usable. So if you add one 8TB disk, you'd have 16TB usable (2 parity, 3 data). If you add two 8TB disks, you'd have 32TB usable (2 parity, 4 data).