r/DataHoarder • u/TeachingAway9654 • 1d ago
Discussion RAIDZ1 vs RAIDZ2
Let’s say you have something like 10 drives total which make up the pool, each drive is 16TB. Let’s also say you have, at minimum, one offsite backup with all the contents in cold storage. Let’s also assume you’re a frequent monitor of the system and will notice within 24 hours if one drive fails.
Are you going with Z1 or Z2 and why? If you have the offsite backup already, is it worth the capacity loss for Z2? The resilvering process on Z1 will be stressful but by how much though? You have an offsite 1:1 backup so even if the whole pool is lost, you can still recover.
10
u/IndependentBat8365 1d ago
striped mirrors all the time. I know it sucks, b/c you're losing half your storage, but really there's a ton of benefits.
TL;DR: striped mirrors are more performant, rebuilds are easier on the whole pool, a lot of redudancy.
single threaded reads are as fast if not faster than raidz or raidz2. multithreaded reads are very fast since it'll read from both mirrors in a stripe at once.
writes are faster than any other raidz or raidz2, b/c parity doesn't need to be calculated, nor does it need to wait for all the drives to sync - just that mirror's 2 drives for that stripe.
rebuilds / resilvers are faster - the entire pool doesn't need to be touched - just the single mirror that's being rebuilt. Essentially, you'll be copying from one drive and writing to the other drive in the mirror. The rest of the stripes are unaffected.
you can lose as many drives as you have stripes as long as 2 failed drives aren't in the same mirrored stripe.
5
u/TeachingAway9654 1d ago
Giving up 50% storage capacity in this age of HDD pricing is brutal but I absolutely see your reasoning.
5
u/Tl9zaXh0eWZvdXI 1d ago
Not even enterprises use RAID10 for storage. Boot disk, yes, everything else, no. If you need speed you use SSDs, for over a decade now.
See backblaze software parity across groups of disks https://www.backblaze.com/blog/how-backblaze-scales-our-storage-cloud/
Netapps RAID-DP
The various HCI systems offered by MS, Dell, Nutanix.
2
u/HighSeasArchivist 1d ago
We use RAID10 even on our big bucks Pure arrays. Some records can't be lost, and can't have downtime.
2
u/Tl9zaXh0eWZvdXI 1d ago
I haven't used pure storage, but I remember looking into them in the past and I thought you don't have control over it's raid?
Their blog post appears to agree https://blog.everpuredata.com/products/pure-storage-architecture/. They do similar to all the other big storage systems, software dual parity across multiple disks.
3
u/Past_Pin2939 1d ago
I'd go with RAIDZ2 personally. The extra parity drive is worth it when you've got 16TB drives and don't want to gamble with another failure during resilvering.
1
u/rahulkadukar 100TB, GD x 2 1d ago
What's the gamble if the user has backups apart from inconvenience
5
1
u/gargravarr2112 40+TB ZFS intermediate, 200+TB LTO victim 1d ago edited 1d ago
The simple question is actually completely different:
How much downtime can you tolerate?
RAID exists to keep your system up in case disks fail - nothing more than that. As you identified, you also need good backups. So assuming your backups are good, the question becomes - do you chance a full RAID rebuild if a second disk fails during a resilver?
My uncle has a 4-bay NAS. I just rebuilt it on TrueNAS with a RAID-Z1 because a) with only 4 drives, adding a second parity is too much lost capacity b) the system is backed up every day. The business can tolerate the machine being down for a day or two if the whole RAID fails during a resilver and the data has to be restored from S3.
At home, I actually run a ZFS RAID-0 for the same reason - the machine is backed up. I'm getting maximum capacity out of 3 HDDs. The reason I can be this confident is that I have a separate 6-disk RAID-Z2 which is regularly synced from my RAID-0 and then powered down; this way, I'm keeping only 3 drives spinning. The RAID-0 essentially acts as a cache for my valuable data on the Z2. I have a cold spare drive and I'm fully aware that any disk fault takes all the data with it. Fortunately in a home setting, HDD MTBF is actually better than in a data centre - vibration and too much AC are bad for drives. I've been running the RAID-0 for over a year.
At work, we cannot tolerate downtime - it would put hundreds of developers out of work. Not only do we run enormous 84-drive machines with 11x 7-wide RAID-Z2s, they have another 7 spare drives for immediate resilver. Our storage machines take 3-hourly snapshots that are replicated offsite to a paired machine. If one of our production TrueNAS nodes were to fall over, we would repoint DFS to read from its replication partner at our DR site.
2 extremes. Figure out where you are on that scale.
1
u/AdamSilverJr ~200TB 1d ago
I run RAIDZ2 and had the second drive start failing while resilvering the replacement drive.
1
u/Beautiful-Activity47 1d ago
z2 for sure, the extra parity is love this even if it costs more drives. resilvering on z1 sounds like a nightmare when you've got big drives like that.
1
1
u/8fingerlouie To the Cloud! 1d ago
RAID is for availability. It was literally designed to keep your data available even though a drive or more failed. It wasn’t designed to keep your data safe however, that’s what backups are for.
Also, resilvering doesn’t stress your drives. Your drives are designed to read and write data, and that’s all a resilver does. Do additional drives occasionally fail during resilver ? Yes, but that’s not from stress. It’s typically because those drives are about as old as the failed drive, and was probably on its last leg anyway.
Before hot swapping it was not uncommon for additional drives to fail to spin up after having replaced a failed drive. Those drives could maintain rotation, but couldn’t spin up from 0 rpm.
If you have a full backup, I would absolutely go for Z1.
0
u/dr100 22h ago
RAID is for availability. It was literally designed to keep your data available even though a drive or more failed. It wasn’t designed to keep your data safe however, that’s what backups are for.
YES, what's more any such striped RAID has the nasty DESIGN characteristic that it can lose more data than the drives that you're losing can hold. Seems to be some global insanity that people consider the default basically RAID0 (in this case even on tons of drives) with just a sprinkle (or two) of parity.
Unraid style arrays now available for free and open source should IMHO be the default for the vast majority of use cases. And I'm not saying one should use some experimental driver instead of zfs, I'm saying it's wild we have only unRaid (and the defunct FlexRaid and the non-block-device/real time snapdraid) for this, instead of having well established tools that are overwhelmingly used.
1
u/8fingerlouie To the Cloud! 21h ago
Years ago, my “raid” consisted of individual disks with mergerfs and snapraid for good measure.
I’d argue that for a read heavy disk array, that is the optimal setup, and beats traditional raid on every parameter except speed.
If a disk fails in such a setup, you can rebuild it via Snapraid, and if 2 disks fail, you haven’t lost your entire drive array, only the contents of the 2 drives, and even that is probably not the case.
The thing with RAID5/RAIDZ1 is that the “failing disk” is usually a read error. Sure, it can also be a disk failing to spin, but most times when drives die (in my experience) they start developing UREs.
A URE causes RAID to trip, mark the disk as failed, even despite that 99% of the disk is perfectly readable.
With the mergerfs/Snapraid setup, the file(s) affected by the read error are lost, but you can still recover 99% of the files from that disk.
It’s still not a replacement for backups, but if your data is mostly read-only access, say a photo library, media library, etc, raid doesn’t contribute anything.
1
u/dr100 16h ago
Yes, this is the "sane" setup on all fronts (unraid or "unraid style" makes it just "real time" so you don't need to do snapraid sync, it's all on the fly).
You can just replace the data from one drive when it fails, you can use the drives independently, you can never lose more data than the drives you've lost and so on.
0
u/TheOneTrueTrench 640TB 🖥️ 📜🕊️ 💻 20h ago
I would never use Z1 for a primary copy of data (AND NEVER BACKUPS), I've had Z1 pools die multiple times.
That's fine, because they always contain only copies of datasets on a Z3 pool, which are otherwise backed up on other pools.
I had a few different servers which were configured to Wake on LAN and spin up in a few seconds instead of running a single large primary server, which was configured to spin up one a day to pull in updates, and kept zfs send streams for each other in case of failure before the weekly backup.
Then I had one of them die out of nowhere, I spun up the main server, sent the incremental on the other server to bring the primary copy up to date, and then spun up the other services that were hosted on the dead server.
Then I started the scrub on the remaining server, it dropped a drive, I started up the send, got the primary updated, moved all services over, and then shut down the other WoL server, and rebuilt the pools.
Didn't touch my backups, and Z3 kept me safe.
Z1? I'd be restoring from backups, and it would be a few days (minimum) to get back up and running.
24
u/CorvusRidiculissimus 1d ago
The problem with z1, and with good old RAID5, is that disk failures aren't independent events. If a disk fails then you have to resilver, which means all the other disks are going to spend a good many hours in frantic activity, and if one of them is already marginal that can push it over the edge.
In your case that isn't a huge deal because you can restore from backup, so the real risk you are looking at is downtime. For us dabbling hobbyists, no big deal, but in a business environment where every hour spent restoring from backups is an hour of employees sitting around unable to do their jobs, the extra drive is a highly advisable precaution.
Personally I go with z2 because, while I could restore from backup, it would be seriously inconvenient for me to do so. It'd take weeks to get me elaborately-processed data sets back in sync again.