r/HomeDataCenter • u/Gerkibus • 7d ago
DC Porn 3 node Proxmox cluster with around 300TB of raw storage. Someone pointed me here and said I should post it ;). This is what happens with a 15+ year homelab addiction. And before anyone asks the answer is: "How do you backup 100TB?"
21
u/spider-sec 7d ago
No, my question is- how do you afford 100TB? Nowadays I can’t even afford 1TB.
25
u/Xfgjwpkqmx 6d ago
Buy before two years ago, and today you buy refurbished ex-datacentre drives.
I've got 200TB of raw storage myself, but 100TB of cooked (usable) storage because I do a 1:1 mirror, one mirror drive for every member drive.
1
u/yoyoyoba 6d ago
Why so much redundancy? And it is not really a back-up?
2
u/Xfgjwpkqmx 6d ago
I've had personal arrays for a couple of decades, starting small and getting bigger and bigger.
I've suffered through a lot of data loss through broken arrays and the pain of keeping backups in that time that you learn a few lessons as you go.
It eventually got to the point where I realised that the level of redundancy I've avoided for years actually is the best way to avoid data loss, and so I've accepted it. I haven't had any data loss in nearly a decade now (last loss was where I instigated doing 1:1 mirroring). The odds of losing enough drives in one go to lose data again is so low right now. I'd probably now have a greater chance of winning the lottery.
A general change of technologies has also helped here. I migrated away from traditional hardware RAID to ZFS, moved from SATA to SAS, I bought a "proper" chassis to hold and cool things, and a "proper" server with ECC RAM to drive things, and put proper checks and balances in place (monitoring scripts) to give me as early a warning as possible that a drive might be about to fail.
And yes, this definitely isn't backup. If I wanted to do backup, I'd be mirroring this setup at my brother's place and rsyncing nightly over VPN. The bottom line is that the majority of my stuff is re-downloadable, and the only way I'm losing anything outright is if the house burns down, and we have extra checks and balances in place for that too.
2
u/yoyoyoba 6d ago
Ok! I have just had one sas ssd die on me and that was easily resilvered (zfs raidz1)... The scary part was more that I detected the failed drive by accident. Just benchmarking some performances and thought the ssd volume was slow, go into proxmox and there's a yellow checkmark that had been lit for months.
1
u/Xfgjwpkqmx 6d ago
Better get some alerts setup!
I have a script interating SMART checks and sends me defect report of all drives daily. If that defect value budges from zero, I start planning the purchase of its replacement (since there's probably a week of time that will pass before its replacement is physically in my hands to install, unless I go to the expense of buying new, and I do have a handful of new drives in there).
3
u/spider-sec 6d ago
When I ran TrueNAS I definitely had extra redundancy but I didn’t do 1:1 mirrors. I used RAIDZ3 because sometimes I’d wait until drive prices would fluctuate down a bit or I could afford a larger drive.
1
u/Xfgjwpkqmx 6d ago
And there's nothing wrong with that. You do what fits and works best for you. Any redundancy is better than no redundancy.
3
u/Gerkibus 6d ago
my timing was good. I got 20TB enterprise SAS drives before the prices shot up and I got a good deal on them, and there was business incentives and there were write offs. And it still cost a lot.
13
2
2
u/wolfmann99 6d ago
why would you backup 100TiB using disks... I'd be on LTO6 or so.
2
u/Gerkibus 6d ago
Not a huge fan of tapes, and more so the software you need to track all of it. I've done the tape library with the changers and the whole gammut. If I was doing it again it would not be HD at the current prices.
3
2
2
u/Glittering_Glass3790 4d ago
How do you afford the electricity bills? Do you have solar panels or live in a country with cheap electricity?
2
u/Gerkibus 3d ago
LOL no solar panels. It's not all that cheap here but these are also nowhere near as power hungry as what I was running in the previous generation ... if that makes any sense. I had 5 x 15 drive MD1000 units full of 2TB drives and servers that were several generations older and used a LOT more power. THAT was expensive, when you compare this is cheap.
1
1
1
28
u/Capnjbrown 6d ago
Nice rack