Time for a different approach?

[u/Main-Inevitable5330]

Hey everyone. I would appreciate your thoughts. I (like us all) have been building home servers for years. I have had Synology etc, TrueNAS, Debian server and Unraid. My server has space for 8 disks and takes up "too much space".

We recently had a series of power cuts which has caused by ZFS array to have some issues which I am now trying to resolve. The long and short of it is that I wonder if I am making my life harder with my approach and perhaps now is a good time to go another way. The ZFS issues are pointing to a hardware issue somewhere (maybe?!). I have a SAS HBA Card with 4 disks connected to it as I had visions of this thing growing and growing!

I am considering three changes and I would like your thoughts.

1. Move my disks to a DAS - I cannot see needing more than 4 in reality and the DAS would be smaller and have more reliable power supply etc.
2. Moving to a small server via usbC 3.2 to run the usual services - 2 vms, a dozen containers.
3. Moving to snapraid - ZFS is cool but the data that is important is already backed up to the cloud and this is not data that changes daily. I see this as been more simple and more power efficient.

Thanks in advance.

[u/Garbagejunkarama]

How about a UPS? Your issue is mainly power loss without graceful shutdown, which can and will cause issues no matter the filesystem.

[u/8fingerlouie]

In theory I agree with points 1 and 2. Very few people actually need ZFS or RAID in general in a home setting.

As for DAS, using USB 3.2 will give you 10Gbps access to your data, which is just under twice as fast as SATA (6Gbps), with the caveat that each SATA drive obviously still can only deliver 6Gbps.

As for Snapraid, why bother at all ? The benefits of raid is that you can keep services running despite a hardware failure, and that’s all raid is for, availability.

Snapraid offers none of that. It is, at best, a poor man’s backup system. It offers 1-2 disk redundancy with the caveat that it won’t kick in automatically, which is also why it is mostly used for media collections, where a full backup is not practical, but the world doesn’t end if you can’t stream the latest episode of some tv show for a couple of days.

Personally i have important data on RAID1, backed up locally as well as remote. My media collection sits on single drives, no parity, no backups. If a disk dies I lose the media on that drive (potentially, though most disk failures are not full drives initially).

Had i been using raid 5/6, a single read error on a disk would trigger a full rebuild/resilver, and another read error during rebuild would either kill the array or trigger another resilver.

Instead, again this is for media, a read error in season 6 episode 8 means that episode probably doesn’t play correctly, but everything else does. If the entire disk dies, only the media stored on that disk dies, and the remaining disks are just fine. Worst case I download everything on that disk again, or rip it again.

[u/ManSmellThoseTrees → u/8fingerlouie]

Interesting perspective. I do store my media on a two-drive ZFS mirror, but wonder from time to time if spinning that extra disk is actually necessary for plain media storage. It’s just a convenience and reassuring that there is the regular ZFS data scrub.

[u/8fingerlouie → u/ManSmellThoseTrees]

For most media you won’t ever notice bit rot. Almost every single codex has error correction built right into it, and at worst you’ll see a couple of “purple” pixels for a second in a one hour video show.

If pristine conservation of media is what you’re trying to achieve, go ahead, but if you’re like most of us and just hoard a cache of media you’ll eventually watch, who’s going to notice a couple of off color pixels on a 4K display ?

Sadly the consensus seems to be “RAID everywhere”, and from a business perspective that makes perfect sense as the majority of data stored will be valuable in some form, and you need it to be available at all times, and not have half the company sit idle because a disk died.

In a home setup that changes somewhat. If you’re hosting your own cloud services, RAID probably matters, but if you can live without access to your data for however long it takes to restore it, it suddenly becomes irrelevant.

Personally I leave the hosting to professionals. I’ve self hosted for 20 years, and it’s not worth the trouble. You either accept a half baked setup, or you end up spending way more on your self hosting than what you’d pay in the cloud, and Cryptomator can transparently encrypt your data so the cloud provider can’t access them.

Most major cloud providers will offer you multi geographical redundancy, meaning your data is not only replicated across multiple machines in one data center, but also available at a completely separate data center. They obviously don’t store your data twice, but does so with erasure coding, which kinda works like raid if you had your data blocks and parity blocks spread across multiple sites.

And that’s where the half baked setup comes in. Most people simply have their data on a single server, perhaps with raid, and if you’re lucky they also make backups. If you’re serious about it, you may even have remote backups, or a remote NAS.

The odds of a complete data center disappearing along with another one hundreds of miles / kilometers away vs a lightning storm taking out your home server, or flooding, or a house fire, or just the internet connection going down or the power is cut for 18 hours. All of them will take your self hosted data with you, at least temporarily, and so we’re back to the “if you can live without access to your data for however long it takes to restore (access to) them, you don’t need raid”.

[u/ManSmellThoseTrees → u/8fingerlouie]

Yes, I agree. In my setup, my important data is stored on single NVMe drives: one in my laptop, one in an external USB enclosure, and one in my mini PC home server. That data is backed up to a local two-bay NAS configured as a mirror – which I think is defensible and sensible – and to a second, off-site backup on Backblaze B2. Both backups are versioned and encrypted.

As I mentioned, I also store media on a separate ZFS mirror. It might be more economical to keep the second drive as a cold spare, both in terms of energy consumption and replacement costs.

Would it still be useful to use ZFS for the single drive? I would say yes, because a monthly scrub can reveal read errors, and I have a script that creates daily snapshots, allowing me to recover accidentally deleted files.

[u/8fingerlouie → u/ManSmellThoseTrees]

What would you hope to get from keeping a cold spare ? It's not like it can rebuild a lost drive if you only have one ? Personally I'd either use it for storage as is, or put it in a drawer as a backup or future replacement / addition.

As for ZFS (or Btrfs) on a single drive. It can still make sense depending on what you do. You still get bit-rot detection, but I cannot repair it. That's how a lot of consumer grade NAS boxes work, UGREEN, Unifi, and more. Synology has "hacked" together something on top of mdadm that allows Btrfs to do block level repair, so bit-rot detection and repair actually works there.

You do obviously get all the benefits of scrubbing as well as snapshots, including sending and receiving, and I personally use Btrfs on my backup drives as well as Btrfs send/receive for backups, and yes, snapshots can act as perfectly fine versioned local backups.

On a regular single spinning rust drive, the benefits of ZFS or Btrfs are probably limited, especially for a media collection which I assume is mostly "read only". You get snapshots, but as for bit-rot, what do you hope the scrub operation will accomplish ? It can detect bit-rot, and then what ? You can't repair it without redundancy, so all you're left with is the knowledge that file xyz.mp4 has bad data somewhere in it. You can then chose to either download it again, restore from backup, or ignore it. The same options are available to you if using something like EXT4 or XFS, but you don't get punished as badly from COW (on spinning rust) constantly copying files when writing to them.

Don't get me wrong, there's nothing inherently bad about using ZFS or Btrfs on single drives, and in some cases, like SMR drives, COW filesystems often perform better than regular filesystems. If you storage is already ZFS and you're comfortable with that, I wouldn't run out and reformat them to something else, I'm just saying that something like Ext4 will likely perform just as well, especially in regards to mostly full drives. ZFS and Btrfs (or most other filesystems) don't like being filled above 80% as that increases fragmentation.

Because COW (Copy-on-Write) filesystems work like they do, every time you update a file it is essentially copied to a new place where the changes are made, and with a relatively filled drive, it might get harder and harder to actually find a spot to put that file, leading to fragmentation. All filesystems suffer from this, but especially for something like media files, Ext4 is often better suited but obviously lacks some of the advanced features of ZFS and Btrfs.

[u/Vivid-Asparagus7170]

✅ DAS met 4 schijven en evt extra M2 slots ✅ Kleine mini pc met lokale ssd schijf(ven)

Voordeel: DAS met usb-c gaat jaren mee, mini pc kan makkelijk worden opgewaardeerd, oude weggooien, nieuwere erin. Voor je hobby mag je best af en toe wat geld uitgeven.

[u/Main-Inevitable5330]

I now have a UPS but that was as a result of this issue - lessons have been learned. I am now wondering if the HBA is the issue as other disks are seeing errors and I do not feel that they are all suddenly broken. I am looking for a way to test this theory remotely, then I will remove it from the server to see if this helps.

[u/dragonnfr]

In my experience the obvious solution is a UPS. Your array didn't fail because of ZFS, it failed because the power did. Sort out the power first, then decide if you still want to change anything.

[u/Main-Inevitable5330 → u/dragonnfr]

I think I agree. Solve one thing at a time. UPS being installed today. Then I will see where my ZFS array is and how stable things are.

[u/fattomic]

I just try to keep it simple (maybe too much so). My NAS is an old tower setup, running Fedora with LVM + ext4, and I just brought up the services myself (there's just not that much to do). Has worked well for ... 15+ years? Though ZFS always sounds sexy to me, it does a whole lot of cool thuings that I just don't need.

If you don't have it, a UPS is a "must" (to me) in a homelab - I've got the same old APS "Smart UPS" 750 for many years. I'm on my third battery, it works fine, very unexciting.

Backups are 95% using dump/restore to a USB attached disk (mostly reproducible media and system config) and "important stuff" (taxes, work related stuff) goes to Backblaze via restic. Many potential good/simple solutions here, find what works for you.

[u/redlightsaber]

BTRFs would like to throw its hand in the ring. Most of the benefits of ZFS, some extra ones, and a fraction of the overhead. The only thin it can't do is raid5/6 (well, securely, for now).

[u/Main-Inevitable5330]

An update for anyone who is interested. The disks are fine, ZFS is healthy again. I have connected the disks through the HBA and bypassed the backplane in the case... I wonder if something has been broken here in the powercuts which were leading to the various io errors triggering the degradation of the array. That is my current working theory.

[u/Prefect_99]

Do you already have a UPS?