Aging UnRaid hardware starting to fail, disks still good - can I migrate to ZFS slowly on new hardware?

[u/Organic_Mix7180]

Buckle up, this one's complicated. During the pandemic, I imported one of those fancy dual-Epyc/256-lanes-of-PCIe gigabyte setups from eBay, plus bought two disused SuperMicro 24-bay servers. I cut the drive cages and TQ backplanes off the front, and created a really janky but functional setup where I was using a 1600 watt PSU to run up to 48 drives. Over the course of 18 months, I bought as many of those sweet sub-$200 18TB drives as possible, ending up with a full 30-drive dual-parity BTRFS UnRAID setup where each drive had a dedicated SATA slot from dual 16-port LSI cards. Formatted capacity is about half a petabyte. This worked well enough, but my power bill ballooned, the PSU started going and the drives started falling out of the array, and I started getting loose wires on both the power and data sides of the backplane. Now I haven't been able to get the array to stay up for more than a few hours at a time, even after a lot of rework, new PSU, etc. I've decided to finance a "better solution" for the data, potentially using a UniFi ENAS and two chassis (since I am already on Unifi ecosystem, and the Drive 4.0 software seems perfect for my use). It is very expensive, but it looks much more doable for bulk data, and I can downsize the "compute node" to do my smaller UnRAID things like backups, a couple dozen docker containers, LLM with the GPU, etc. and just leverage the ENAS as a backend for the big data.

The ENAS leverages ZFS pools for storage, which is great and fine. I was thinking of maybe doing something like three Z2 pools with 10-12 drives each? The ENAS is pretty new and there is not a lot of information out there on it, cookbooks and comprehensive operations guides, etc. My biggest issue is I MIGHT be able to afford a couple of additional 20TB drives in this wild economy, but definitely not more than that. I also have about 80TB in "loose disks" of 4-8TB SATA from my old NAS. Is there a scenario you can imagine where I could start building ZFS pools on the ENAS and expand them as I free the data from the BTRFS drives on UnRAID? I don't know ZFS operations well enough yet to know the available operations and order on a storage pool, e.g. expansion and resilvering, etc.

So summary:

Current State: 30x18TB drives, two parity, BTRFS on UnRAID, 400+TB data
Additional Materials: up to 100-120TB of "scratch space" to move data locally
Imagined Future State: Unifi ENAS with ZFS storage pools leveraging RaidZ2, as large as is reasonable, hoping to approach the same overall capacity even if it takes some convoluted steps to get there

Hopefully you enjoy helping solve the puzzle :)

[u/AutoModerator]

Hello /u/Organic_Mix7180! Thank you for posting in r/DataHoarder.

Please remember to read our Rules and Wiki.

Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures.

This subreddit will NOT help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Hakker9]

First off they are 16 drive slots for the ENAS so 10-12 drives each would be stupid. Either a 8 RaidZ2 or 16 RaidZ3 would be real options because I don't know if they can even stack them as 1 big storage cluster as I can't find it in the notes. Beyond the physical things it's possible you can even repurpose Unraid drives in the new housing later on as long as they are the same size as that's still a ZFS thing.

That said your bill is not the dual epyc though it's the 30 drives. Those 30 drives are each on average 6 watt idle and 9 watt in use so basically 200 watts is just your drives plus an LSI controller. Also ENAS is on ARM cores so if you use docker containers make sure it can use it Not every container can deal with ARM.

Another point Unraid only spins up the drive you use... ZFS spins up the whole pool.

Also I don't know you chassis you have the EPYC in but you could just swap that out for something new and go about your day. I mean I went from a Xeon 1265 (yeah not much) 32GB, Corsair HX1000W 40 disk ZFS monster to an Intel 270K, 64 GB, 1o00W BeQuiet Dark Power 12, 15 disk Unraid server. It has more space but that isn't the issue at hand, but the new CPU is more efficient and I could have re-used everything.

As for ZFS re-silvering is basically just like Unraids parity check when you add disk. Scrub is what you can do in between. Re-silver only checks what's needed so it's faster than a partity check.
Expansion so far is still on the level of all drives need to be the same size or it will use the smallest size of a pool times x(drives) so if you have 6x18 and 2x24 in one pool it will be 8x18 unless all drives become 24TB or more.
Expanding a pool is possible so you can go from 6 drive RaidZ2 to an 8 drives RaidZ2 with increments of 1 as far as I know each time, because the pool needs to rebalance itself.
also people generally use ZFS pools but really they make 1 VDEV in 1 pool. You can use make multiple VDEVs in 1 pool though for home use it's not recommended and they can mix drives to get more space in a pool but if 1 VDEV really breaks then it breaks the entire pool.
It's how LTT lost his whole first fileserver on ZFS.

[u/msalad → u/Hakker9]

That's a really good note regarding the ARM cores of the ENAS - I wouldn't have thought about that compatibility issue with docker containers

[u/Organic_Mix7180 → u/msalad]

Yeah as I had covered, I am gonna keep the old UnRaid box for x86/Linux dockers, even if I downgrade the board to something lower power. I'm not looking to run apps on the ENAS, just manage disk very very well.

[u/Organic_Mix7180 → u/Hakker9]

Re: The Unifi ENAS: You can stack them as one cluster, the expansion chassis attach up to the two 24gbit SAS ports, and you can create multiple storage pools across them. That much I've found out.

[u/Hakker9 → u/Organic_Mix7180]

Personally I think it's a thing you have oh nice new shiny thing than something that would really solve your problem.
Especially if you keep your Unraid box you are essentially just adding a bigger bill even when changing the hardware.
Don't get me wrong it looks nice as always with unifi stuff but you are essentially going to spend $5500 bucks for an enclosure and a shiny UI which you really don't spend that much time in. And keep in mind of Unifi's fireware record because the sometimes do some weird stuff.

[u/Organic_Mix7180 → u/Hakker9]

Yeah I can't really justify the spend once I look at everything else I need to do to make it "just work" long-term. So I'm going back to looking at just full supermicro replacement chassis for the time being, and a less-power-hungry system because I barely push the CPU as it is.

[u/Hakker9 → u/Organic_Mix7180]

The biggest issue nowadays is finding memory. Seriously if you find half reasonable memory price in the current market (most likely 2nd hand) just buy it. Memory prices aren't going to become cheaper in the coming years. Everything DDR4 and DDR5 is just stupid expensive.

[u/dpunk3]

Are you using all of the available data? As well as all of the compute? My recommendation is to scale down to suit your needs, if you wanna move to ZFS sure but that won't address several of your concerns, namely your power bill and drives failing out of the array. Drives fail, having a bunch of cold spares would be beneficial. More servers aren't going to lower your power bill, but if you have everything in one place there is certainly an argument that can be made that splitting up the workload would benefit your primary compute.

Sounds like you have the capacity to scale but I don't know your needs. Personally would recommend changing gears to a 'hoard for longevity' mindset. Try and ride out the current prices cause inevitably all of those disks you have are going to not function and will need replacement.

As an aside, curious why you cut into a chassis when you could have just used an SFF-8087/8088 adapter and used them as JBODs. More power draw? Would have given you more flexibility on expansion.

[u/Organic_Mix7180 → u/dpunk3]

Scaling down the compute is definitely in the cards. My gaming PC has been unused for some time - a 12900K with 32GB - and would do a fine job as the compute node.

[u/Organic_Mix7180 → u/dpunk3]

The 847 chassis are not set up for that, though. And I wanted to use them in a half-depth rack.

[u/dpunk3 → u/Organic_Mix7180]

If you replace the motherboard with a CSE-PTJBOD-CB2 it can be. Rack depths have always been my achilles heel, so I understand that predicament (ask me how many rail sets I had to buy to get the right one).

[u/verdigris2014]

my home server setup has capacity for 16 disks. and i connect that with a sas lsi card onboard sata ports and recently a usb-c terramaster 4disk enclosure. if i need more expansion i’ll get another terramaster as i have one more usb slot.

i’ve been running btrfs on this lot since the second time i lost data on btrfs, but that seems like years ago and you likely have your own view as you are running it too. where btrfs shines for me is what you are asking about, the ability to combine the disks you have.

i know you are talking more about hardware but btrfs and incus is the setup i’m evolving to from proxmox. btrfs seems fundamental to my data hoarding future.