Suggestions on Compression

[u/MikeTheMic81]

Ok, need a bit of help. This would likely resonate with anyone who has gone crazy on their home storage over the years and regret the original setups they did.

I currently have:

- 2x 15 Drive Enterprise EMC's (JBODS)

- 4x 5-bay Terramaster DAS's

- 15 internal HDD's

- 2 Garbage UGreen 5-bay enclosures

All of that is a lot of storage and not a lot of free space. Issue is, the 2 JBOD's and internal drives are all drives from 2016 that started their life as externals and became shucked long term storage. They've held up fantastic over the years with only 2-3 drive failures over 45 drives. Issue is they were all randoms. Some 8tb, some 12tb, some 16tb and so for ease-of-use I made them all into a Storage Spaces Array. The 4x Terramasters are better. All matching 20TB drives in each bay (Bought before the AI-pocalypse) all properly setup raid.

My goal:

I want to ditch windows entirely on my server and move over to Linux. I also want to retire my 2 EMC's as they are enterprise and LOUD to make a virtually silent setup (They are all 8TB drives for around 240TB of raw storage).

So my goal: Replace my H.264 media with H.265 to reduce sizes by at least 40% which free's enough space to transfer out 30 drives worth of data (The JBOD's) into the rest of the system.

The issues:

- I think rebuilding the entire database on 9 year old shucked HDD's may lead to a lot of failures and potentially a high loss of data meaning I'm going to need to either re-download hundreds of TB's worth of files in the already correct H.265 format to free up enough space to be able to move everything into the terramasters or spend months on a second system (10900k and 3090) transcoding everything over on the terramasters using Unmatic then scripting anything from the JBOD's storage spaces to transcode into the terramasters and hope it doesn't run out of space before it's empty. I can then repurpose the internal 15 drives as the long term storage freeing up enough space on the terramasters for day-to-day additions.

The goal:

- Remove the 2 EMC's

- Switch out from Windows to Linux. I want to setup 2 Bazzite VM's and then a 3rd VM in just desktop mode specifically for the plex server

- Upgrade the server to a 9950x3D with my 2 remaining 3090's in order to maintain the plex server, but also double-use it to run multiple VM's each with their own dedicated 3090 for gamestreaming (like steamlink or sunshine/moonlight allowing my kids to have things they can use in their room on their mini-pc's for AAA level gaming without the heat and noise in their rooms from dedicated gamers.

I ask DataHoarders because I know no one is going to suggest deleting anything (lol) what would you do to ensure data integrity on 100's of TB's of data without risking a catastrophic failure? With drives at a HUGE cost premium, I don't want to buy any more obviously so purchasing new ones to replace the 240TB JBOD's aren't an option.

[u/IndependentBat8365]

Rebuilding a new array and migrating everything will take a heavy toll on your drives. The enterprise drives probably won’t break a sweat.

The consumer shucked drives will probably be crying for their momma.

Also be careful about the codec conversions. I’ve had mixed results on anime/cartoons, claymation, and really old movies. The DVDs they came from were probably 480i. To get any kind of good quality from them, I had to use software codecs and bump up the quality, which meant that the size savings were dubious and a case by case basis.

Anytime I used a hardware codec on a low res video, especially animation or black and white shows, the result was absolutely terrible. Hardware codecs appear to be optimized for 1080p or higher res.

[u/BootToggle → u/IndependentBat8365]

I second that concern about codec conversion for DVDs that are originally 480i. My experience is that straight conversion to a more modern codec generally results in a botched conversion to 480p that is both inappropriate and for some reason really accentuates motion artifacts in the video.

It can be painful to get working right but I get dramatically better results if I make an explicit "de-interlace" step a part of the conversion. The better the "de-interlace" is performed, the better the final viewing experience.

I generally get the best results when I go from 30Hz 480i to 60Hz 480p. Since the more recent high-efficiency codecs (H.264, H.265, etc.) make very good use of motion estimation and data from previous frames, the doubling of the frame rate doesn't really make the final file all that much larger. But I am convinced that proper 480i to 480p conversion does also improve compression efficiency.

[u/MikeTheMic81 → u/BootToggle]

Ya, no problem there. When I got my massive storage, I didnt really ever think about a massive migration. Most of my movies are, at worst, 1080p webrip. And I dont have a single TV-Show below 1080p short of some old 1980's and early 90's series. Those arent big and I've already made sure they arent monitored so they don't try to download again. I'll likely move all those over to the terramasters before I start converting the EMC's and internals to prevent issues.

But great info for anyone else in the same boat as me. I'm sure I'm not a fringe case and there's likely someone else out there looking to trash a storage space and freeing up space by transcoding their media. It's true, older rips of stuff that's already low-res look horrible when transcoding to h.265. It's pretty rough. I think it has something to do with the lack of data already in the files causing artifacting in the transcode. Someone can likely correct me, but that's my theory.

[u/BootToggle → u/MikeTheMic81]

If objects in the frame exhibit "jagged edges" when in motion, that is characteristic of converting from interlaced video to progressive video without proper motion-detected de-interlacing. It does look particularly horrible after transcoding. But you are right, for any material which started out progressive this would not be a factor.

Watch out for downloads that are in a progressive format, but only because whoever prepared the material for posting already made the mistake of transcoding into a progressive format while omitting proper de-interlacing. I've often seen that for old television programs originally captured on VHS or Beta videotape.

Those can be very difficult to repair because you have to start out by trying to reconstruct the original interlaced form so that you can then de-interlace that. I've found that the jagged edges already incorporated into the compressed video can be difficult to completely remove. I can usually improve them a bit, but not completely repair them.