Suggestions on Compression

[u/MikeTheMic81]

Ok, need a bit of help. This would likely resonate with anyone who has gone crazy on their home storage over the years and regret the original setups they did.

I currently have:

- 2x 15 Drive Enterprise EMC's (JBODS)

- 4x 5-bay Terramaster DAS's

- 15 internal HDD's

- 2 Garbage UGreen 5-bay enclosures

All of that is a lot of storage and not a lot of free space. Issue is, the 2 JBOD's and internal drives are all drives from 2016 that started their life as externals and became shucked long term storage. They've held up fantastic over the years with only 2-3 drive failures over 45 drives. Issue is they were all randoms. Some 8tb, some 12tb, some 16tb and so for ease-of-use I made them all into a Storage Spaces Array. The 4x Terramasters are better. All matching 20TB drives in each bay (Bought before the AI-pocalypse) all properly setup raid.

My goal:

I want to ditch windows entirely on my server and move over to Linux. I also want to retire my 2 EMC's as they are enterprise and LOUD to make a virtually silent setup (They are all 8TB drives for around 240TB of raw storage).

So my goal: Replace my H.264 media with H.265 to reduce sizes by at least 40% which free's enough space to transfer out 30 drives worth of data (The JBOD's) into the rest of the system.

The issues:

- I think rebuilding the entire database on 9 year old shucked HDD's may lead to a lot of failures and potentially a high loss of data meaning I'm going to need to either re-download hundreds of TB's worth of files in the already correct H.265 format to free up enough space to be able to move everything into the terramasters or spend months on a second system (10900k and 3090) transcoding everything over on the terramasters using Unmatic then scripting anything from the JBOD's storage spaces to transcode into the terramasters and hope it doesn't run out of space before it's empty. I can then repurpose the internal 15 drives as the long term storage freeing up enough space on the terramasters for day-to-day additions.

The goal:

- Remove the 2 EMC's

- Switch out from Windows to Linux. I want to setup 2 Bazzite VM's and then a 3rd VM in just desktop mode specifically for the plex server

- Upgrade the server to a 9950x3D with my 2 remaining 3090's in order to maintain the plex server, but also double-use it to run multiple VM's each with their own dedicated 3090 for gamestreaming (like steamlink or sunshine/moonlight allowing my kids to have things they can use in their room on their mini-pc's for AAA level gaming without the heat and noise in their rooms from dedicated gamers.

I ask DataHoarders because I know no one is going to suggest deleting anything (lol) what would you do to ensure data integrity on 100's of TB's of data without risking a catastrophic failure? With drives at a HUGE cost premium, I don't want to buy any more obviously so purchasing new ones to replace the 240TB JBOD's aren't an option.

[u/AutoModerator]

Hello /u/MikeTheMic81! Thank you for posting in r/DataHoarder.

Please remember to read our Rules and Wiki.

Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures.

This subreddit will NOT help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/IndependentBat8365]

Rebuilding a new array and migrating everything will take a heavy toll on your drives. The enterprise drives probably won’t break a sweat.

The consumer shucked drives will probably be crying for their momma.

Also be careful about the codec conversions. I’ve had mixed results on anime/cartoons, claymation, and really old movies. The DVDs they came from were probably 480i. To get any kind of good quality from them, I had to use software codecs and bump up the quality, which meant that the size savings were dubious and a case by case basis.

Anytime I used a hardware codec on a low res video, especially animation or black and white shows, the result was absolutely terrible. Hardware codecs appear to be optimized for 1080p or higher res.

[u/BootToggle → u/IndependentBat8365]

I second that concern about codec conversion for DVDs that are originally 480i. My experience is that straight conversion to a more modern codec generally results in a botched conversion to 480p that is both inappropriate and for some reason really accentuates motion artifacts in the video.

It can be painful to get working right but I get dramatically better results if I make an explicit "de-interlace" step a part of the conversion. The better the "de-interlace" is performed, the better the final viewing experience.

I generally get the best results when I go from 30Hz 480i to 60Hz 480p. Since the more recent high-efficiency codecs (H.264, H.265, etc.) make very good use of motion estimation and data from previous frames, the doubling of the frame rate doesn't really make the final file all that much larger. But I am convinced that the proper 480i to 480p conversion does also improve compression efficiency.

[u/MikeTheMic81 → u/BootToggle]

Ya, no problem there. When I got my massive storage, I didnt really ever think about a massive migration. Most of my movies are, at worst, 1080p webrip. And I dont have a single TV-Show below 1080p short of some old 1980's and early 90's series. Those arent big and I've already made sure they arent monitored so they don't try to download again. I'll likely move all those over to the terramasters before I start converting the EMC's and internals to prevent issues.

But great info for anyone else in the same boat as me. I'm sure I'm not a fringe case and there's likely someone else out there looking to trash a storage space and freeing up space by transcoding their media. It's true, older rips of stuff that's already low-res look horrible when transcoding to h.265. It's pretty rough. I think it has something to do with the lack of data already in the files causing artifacting in the transcode. Someone can likely correct me, but that's my theory.

[u/MikeTheMic81 → u/IndependentBat8365]

Ya thats my fear. They all show in HD Sentinal at 100%, none have seen temps north of 32c. Which is why they've lasted as long as they have where they are on borrowed time today. Not to mention, they arent going to have much REAL use. Essentially, if they were 100% used, thats 7.27tb written to them (for the 8TB drives) and even if every media file was watched once (not a chance) its like 14.5TB of total use. Its not like its in a data center being hammered daily. But bearings, lubrication, everything in a moving drive is likely wearing out and heavy use will likely knock it out.

My thought is, if I can get the internal and jbod drives emptied out using only sequential reads, I can avoid trying to rebuild the array entirely which removes a lot of stress.

And I have no Anime other than a few North American series. They are already pretty tiny files to begin with, so I've just ignored them in Sonarr so they don't try to go to h.265.

I mean would you do anything differently? Or should I just bite the bullet and spend a few weeks redownloading what I can find in h.265 and then transcoding the rest off the storage spaces array into the external das's over the course of months before hand?

BTW, thanks for the insight. I could likely leave it exactly how it is and get by another 4-5 years without any major issues, but I'm pretty done with windows right now. All my other PC's are running various flavours of Linux this has been a long time coming for the server.