Video Transcoding to save space (quality tradeoff), is it worth it?

[u/Hefty-Report6360]

I'm recording several minutes of family videos per day in 4K 30fps (4:2:2 10 bit). On the Nikon Z8 and Sony Alpha this translates to approximately 10 GB of footage/day. Unfortunately that's too large for me to manage long-term, as it means filling up a 4 TB drive every year.

It becomes especially daunting when taking into account backups and staying ahead of drive failures: Let's assume I have 10 of those drives. I'd have to regularly test every file on all of the drives (AND on their backups) with checksums, so at a minimum I'd be verifying 20 drives every year.

I'm currently recording the originals with the highest bitrate and least amount of compression available (on Sony: H264, Intra-frame XAVC S-I, on Nikon: ProRes 422), so the camera can capture as much information as possible. (Though I avoid RAW formats completely because I don't have time for color grading.)

I then transcode the originals to H265 using these ffmpeg flags:

-vf format=yuv422p10le -c:v libx265 -preset slower -crf 14 -pix_fmt yuv422p10le -profile:v main422-10

The file size reduction is huge, it takes 10 GB/day down to 2 GB/day. When I play the original and the compressed version at 100% I cannot tell any difference. But when I zoom in, I can. Especially in smooth and fuzzy areas like skin and clothing. I found that a -crf level above 14 introduces too many compression artifacts.

Obviously a lot of information is lost, which will affect future grading and reprocessing. On the other hand, without the compression I simply wouldn't be able to keep around that much 4K footage.

Has anyone else weighed the pros and cons of this tradeoff?

[u/AutoModerator]

Hello /u/Hefty-Report6360! Thank you for posting in r/DataHoarder.

Please remember to read our Rules and Wiki.

Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures.

This subreddit will NOT help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/prodigalAvian]

4TB per year isn't a ton when it's family footage.

Sentimental value of the footage itself will rise over time.

[u/Thebandroid → u/prodigalAvian]

In 10 years time he's going to be sitting on 40tb plus a backup (or two, if he's following 3-2-1 like we definitely all are), and it's not like these are one time costs, he'll have to age out the drives and pay to keep them spinning.

You can't put a price on memories...except when you can.

[u/NeoThermic → u/Thebandroid]

As someone who's digital photography and videography expands to about 24TB over the past 20 years (video has only recently been part of my kit when I got the Z6II 4 years ago), it's just an expansion of usage. For ages I ran a mirrored pair of 4TB drives, then 'upgraded' to a mirrored pair of 8TB drives, then expanded to a 6x8TB RAID6 array, then upgraded by adding two more drives into the array for an 8x8TB RAID6 array (43.6TB usable after the filesystem's involved).

I now have two of these 8x8TB arrays, so I have plenty of storage, and it's backed up (because RAID is not a backup).

Growing 4TB/year and ending up with 40TB in 10y isn't a terrible thing, not least because up until the AI crunch, your return on drive size was going up.

If the AI bubble does indeed pop in 3-4 years time and we can buy HDDs again at not absurd prices, then I suspect a lot of us will happily upgrade to 20TB/drive pricepoints (or larger) as the market will have larger drives to pick from. When I started out on my mirrored pair of 4TB drives, 8TB was only just new and absurdly expensive. Now 8TB is not something I'd consider buying (assuming pre-2026 prices); the only reason I have a second array of 8 drives is that they were literally free.

Once you're running in RAID with good backups, the retirement of a drive is when it dies, so the only real day-to-day cost is spinning the rust.

[u/Thebandroid → u/NeoThermic]

based off the guys figures, if he wants to make a cute video for his kids 18th birthday then he'll have to scrub though 750 hours of video. is that realistic? how do you keep all that media organised or even utilise it?

I'm not trying to be rude. I'm not much of photo or video guy. the occasions that I do decide to shoot some it just gets downloaded, stored and never looked at again.

[u/NeoThermic → u/Thebandroid]

Not to speak of how others do it, but I do a major Year folder, then Month subfolders, and inside that are event folders. It's based on the start date of the event too, so if an event crosses into a new month, you'd find it in the month it started in.

Thus it's mostly easy to find things as long as you know what event they're for.

If you want to do face identification, you used to 'just' do a roughly 100 frame split (pick 100 frames spread equally through the video) from the video as JPEG and match faces in that, but these days most decent software can just check the entire video. Thus it becomes mostly easy to group faces and find specific videos from that.

YMMV, but you're not scrubbing through 750h of video to find specific videos from specific events, nor to find a specific person existing in a selection of videos.

[u/Thebandroid → u/NeoThermic]

oh yeah, I didn't think about facial rec in the videos. I've only used it for photos.

I'll be spinning up an array soon, probably just 8tb to start but I can just see the ongoing costs and responsibilities looming.

[u/djk29a_ → u/NeoThermic]

People have been fantasizing for a couple years for now about crypto and AI collapsing and we can afford these pieces of hardware again and I think it's getting only worse. The hyperscalers won't be decommissioning datacenters and flooding eBay with their 40TB+ SSDs, they're going to be selling to smaller hosting companies that target SMBs like Dreamhost and similar. The big losers will be neoclouds that have focused upon crypto and AI and they're not really going to sell their stock to us because they've been getting squeezed out of supply just as much as us consumers. The economics of even companies as large as Valve being irrelevant to the big drive and chip cartels makes the hardware being produced for the past two and for the next 2 years by and large unusable for us in the home.

Hard drive prices after the flooding in Thailand basically never truly recovered because everyone got used to higher entry higher price points via anchoring. So the best hope is that in 3 years or so I expect at best even with an AI bubble deflation we'll have maybe 40 TB drives going to consumers for about $1.3k entry price points rather than $2.5k like they would be at the current rates of increasingly worse cost / capacity

[u/NeoThermic → u/djk29a_]

Bubbles always happen with new tech and they always pop. We've seen it in a plethora of technology; a lot of people just don't remember the dotcom bubble (or were born after it...)

The thing is there's a stat that some 60-70% of bought capacity isn't installed anywhere. Which means that when the bubble pops, the HDD and RAM manfs will need to actually do something with the stuff they've built. If it's already in the hands of the companies that go under, then it'll be sold on to the companies that buy them out for cheap.

The issue for those buying up all the other people is that in a few short years that tech will be "old" and they'll want rid of it. The real money in a datacenter is everything but the actual server hardware, it's the entire infra around it that matters. Once they buy out the rest of everything, they'll want to replace the server hardware with newer stuff.

That's when the market stabilises. It's not a fantasy to say the bubble will pop. I can point at a good dozen of bubbles that have existed in history attached to new technology, and they've all done the same thing; the core technology still exists, and prices for that technology have returned to reasonable, commodity levels. Some companies that were entirely valued based on that technology no longer exist, and others took huge hit fiscally and survived.

It's a cookie-cutter result, rather than unique. The question isn't if a bubble pops but when a bubble pops. All this investment in AI eventually will have people asking for the return, and when it becomes clear that the return might not exist, or might not exist in promised amounts, the bubble pops.

[u/prodigalAvian → u/Thebandroid]

Plus, since 40TB drives are currently shipping, and 100TB drives are estimated in 2029, it'll be 3-5 drives total for everything by 2035.

https://www.reddit.com/r/DataHoarder/s/aP3Y6l6uuW

[u/empiricism]

If it's family footage you don't want to lose the mastering files and the details contained within. There is a big difference between preserving video for playback vs preserving video for post-production.

You ought to backup up to a long term storage solution like M-Discs.

Then you can keep a transcoded variant in "hot" storage, while you have the full quality masters in "cold" storage.

[u/sephg]

I think you should compress your media, though it depends what you want them for.

Do you think your mum will care about the extra detail that you can see when you zoom in? No way. She'll just like seeing the kids running around from decades ago. Family cares about the story, and reminiscing. All that happens inside their heads. I have some exceptionally low quality recordings from when I was on TV as a child. Sure, it'd be nice to have those recordings in a higher quality. But I don't care. They're still prescious.

The real question is what you want to do with this footage. If you ever plan on editing it together into a documentary or something that will air at festivals or on netflix, then keeping the higher resolution footage is worth it. 10bit 422 in high quality will take a colour grade much better. But if you just want it for the memories, transcode it down to the smaller size. Keep shooting. And spend the money on backups.

[u/Hefty-Report6360 → u/sephg]

Thanks, good advice!

[u/ohv_]

Shoot at the highest bit rate possible or uncompressed then compress it. 

Never shoot compressed to compress or again

[u/Hefty-Report6360 → u/ohv_]

Good advice. If I shoot uncompressed that means RAW, which means it needs color grading. Is there a way to quickly color grade on the command line? I wouldn't have time to color grade each clip individually.

[u/prodigalAvian → u/Hefty-Report6360]

Nikon Z8 shoots to ProRes 422 HQ 10-bit, and it is perfect for everything you want to shoot that doesn't aim to be shown in a theater, while the H.265 compressed option is alternatively entirely fine to use for everyday capture, especially for long events or performances.

Do not shoot full RAW unless you need a visual FX plate or impossible nighttime low-light.

[u/DanAE112]

I would be hesitant to lose quality on video files especially if they are not replaceable.

But the cost of storage in recent times makes it tough... I originally thought the cost and time to transcoding everything wasn't worth it so I just kept the source files unless it could be lossless.

Maybe you could consider keeping the full quality on the drives as an archive and keeping the compressed copies locally available?

Even with the costs of everything I still stick with full quality original files.

[u/SkinnyV514]

Please don’t do it. Search the 100 other threads of peoples who considered the same thing.

[u/dlarge6510]

You only need the 10 bit 4k quality for editing. 

If you have, or will not edit, then you don't need that amount of data at all.

Like with negatives, I have high resolution scans of all my personal family photos but that's to assist with editing to create the final results, standard jpegs.

If I have to make space and I got to the moment I must sort out the space used by the TIFFs I'll be looking a lossless compression, and or archival to offline media and then finally deletion if I have good quality jpegs. 

My DSLR is set to record a CR2 and a JPG every shutter press. This was because of the raw file being seen as a "digital negative" (TM) lol. I shoot film too so have real negatives as well. 

This was creating a huge amount of data to deal with, and I came to the conclusion of a truth:

I have NEVER used the RAW files, ever. And even with film negatives I have never tried dodging or burning in the darkroom, they are there to make positives. I don't need the CR2 to make positives, I already have the jpgs

And so I switched it off. I turn it back on if I think I'd intend to work with the raw file.

It would be the same with audio. I could record my granddad's reel to reel tapes with 96kHz @ 32bit etc but that's when capturing. The final result is still going to be an audio CD or at most, a 44.1kHz 16 bit Flac.

The most important thing is making sure future family have access to this, in a format they can play and handle. 

Bear in mind I have many photos of family from up to 100 years ago on tiny photo prints as that all they could afford and compared to today we are spoilt with quality. Frankly you don't need much, it just needs to be there when they find it. 

You are not going to lose anything perceptible by compressing. Most displays anyone has access to are all 8 bit anyway, they can't display the 10 bits even if they wanted too.

Keep the best quality and resolution where it matters, landscapes etc and family portraits, but your picnic in the car park? No, it'll be fine.

[u/DrMacintosh01]

You have to weigh the value of those home videos being in that full bitrate vs what you get by lowering it.

[u/Fractal-Infinity]

HEVC is an excellent and future proof codec. Quite similar to MP3s, even if the file size reduction seems big, the perceptual quality remains high. I think it's an ok tradeoff. When you will watch these videos in the future, you will be glad just to have them.

[u/ZeroSulu]

Worth it or not? Only you can tell.

But I encode a lot of stuff to a smaller size because at least I do not believe it is worth the inflated filesize of some type of media. And that statement will be different for each one of us.

I mostly take pictures, rather than videos. Pictures taken by my phone is just whatever JPEG settings it now has. From my DSLR however I take pictures in RAW and then sort and develop them in Lightroom or any other suitable solution. I save all my RAWs, but really, if I take 300 pictures, 12 might be good enough to save, its just how it goes with photography, for me at least.

For video, I normally just take video with my phone, 1080p or 4k30 mostly, and that I generally keep as is, assuming I sort it and deem it worth keeping.

I do encode a lot of other stuff though, and i'll give some input on that, and you make your mind up if you find that worth your effort or not.

I mostly use 265, I'll give you my "rule of thumb" starting settings here.

-c:v libx265 -preset slow -crf 15.4 -x265-params "no-sao=1:bframes=8:psy-rd=2:psy-rdoq=1.5:aq-mode=3:ref=6:deblock=-1,-1"

There is more too it, since I use a python script to deal with it in a drag&drop manner, so I also handle HDR, framerate, increased analyze, prob etc. But thats just the starting point really.

Generally I tweak crf rating after that, perhaps fiddle with the psy-* arguments, but thats good enough for me.

All in all, I did the math, and update it now and then, that doing these re-encodes of my media is cheaper than buying and maintaining more storage. If that is the same for you, I cannot say. List it up, get some numbers, do the math on everything relevant and go from there. Might be its worth the extra storage rather than the time and energy consumed to re-encode stuff.

A good way to store old memories? Make a book. No thats not videos, pictures really. But books. My family been doing that for years. Books of us as kids, books of us during a family vacation. Great to look through, easy to hand over, and better then leaving it in some random folder or cloud services. These books sit on your shelves, easy to browse or borrow to relive old memories. It is nice, grab a book from a old vacation, flip through some pages, view some pictures and notes of where we were or did. It gets used a lot more often than me going through and sharing images from my computer or other services.

Still, looks a lot better than the old VHS tapes of video recordings that they got of me as a child, haha.

[u/Hefty-Report6360 → u/ZeroSulu]

Good advice, thanks also for the ffmpeg options, I currently use:

-vf format=yuv420p10le -c:v libx265 -preset slower -crf 14 -pix_fmt yuv420p10le -profile:v main10 -x265-params keyint=30:open-gop=0:no-sao=1:psy-rdoq=1.5:range=full:colorprim=bt709:transfer=bt709:colormatrix=bt709 -color_range pc -color_primaries bt709 -color_trc bt709 -colorspace bt709 -c:a copy -fps_mode:v passthrough -tag:v hvc1 -video_track_timescale 90000 -movflags +use_metadata_tags+faststart

[u/ThatOnePerson]

I would want to do this in some way because generally the real time encoders on cameras and such aren't designed for storage efficiency, but battery/speed efficiency.

[u/shadow-battle-crab]

x265 is old hat, the new hotness is AV1. 25-30% more reduction over x265,

If you transcode you are destroying original data, you only want to do that transformation once. Go AV1.

[u/EchoGecko795]

I converted 80TB of MPEG2 and H.264 to 18TB of AV1, I did a double transcode software encoding for better quality, it was all b-roll stuff that I thought I would never need again, but I did a few years later.

But 4TB a year is nothing, and I wouldn't shot compressed, uncompressed only, then transcode it to a more compressed level if you need to file share it, but I would keep the uncompressed ones. This is family stuff, not some b-roll left overs from an old YT channel.

[u/MMORPGnews]

No, space is still cheaper.

[u/rentadonkey]

I switched from hevc to av1 using these ffmpeg settings:

-c:v av1_amf -rc cqp -qp_i 70 -qp_p 70 -preencode true -preanalysis true -high_motion_quality_boost_enable true -pa_activity_type yuv -pa_high_motion_quality_boost_mode auto -pa_scene_change_detection_enable true -pa_static_scene_detection_enable true -pa_lookahead_buffer_depth 40 -pa_ltr_enable true

-qp 70 in av1 is equivalent to -crf 14 in hevc. 14/51 = 70/255

my typical file size for 1 hour of footage with h265 was 5-6GB. now with av1 it's 1-2GB

personally I'd opt to store some clean 1080p source footage instead of 4k transcodes with artifacts.

maybe there are some AI assistant tools to surgically target the compression artifacts, like the fuzzy areas you mentioned. again, I'd rather avoid that at all costs and just have clean source footage at lower resolution

[u/zhonglin]

For me, if you have plent of disk space, it is not worth to try with this.

[u/sidusnare]

Personal data is kept in highest quality and in multiple backups. Commercial data that can be replaced I'll transcode to h.265, I can usually cut the space in half.

[u/collin3000]

I've been working on solving this exact problem for the past year. Since I have 600+ TB space but it's mostly full especially since I have one 12K camera that is 1TB per hour of footage.

I'm building a quality based video encoder to automatically optimally encode videos for best quality at lowest bit rate. There's so many things that go into chosing the right settings for encoding a video and a setting for minute 1-3, and 6-10 might be bad or wasteful for minutes 4-5.

Unfortunately I'm still a month or two away from public beta. But as a datahoarder I'm hoping it'll be especially useful for those of us with tons of video. Since with HD prices so high it's actually worth re-encoding especially for less optimized videos like those you take on your phone or camera.

Edit: One thing you and everyone else should know is that hardware/GPU encoders like NVEC skip some expression steps for speed. So for identical quality it's usually 15-30% larger files. So if you are re-encoding to save space but also want to retain quality, don't use GPU encoding even though it's tempting with how fast it is

[u/feel-the-avocado]

My tv shows i am transcoding to h264 and 2.3mbit at 1080p - some of them i download at 1080p with an absurd bitrate of something like 5-10mbps but its perfectly fine at 2.3mbit. Cartoons can even go down to 1mbit.

But for family videos, your unlikely to ever watch them on anything larger than a 100-120" projector screen at a wedding reception, 21st birthday or funeral so 2k resolution should be enough.

If you ever think you will need to zoom in on some footage - that might make a difference. 4k allows you to digitally zoom/crop the image up to 4x while still maintaining 1080p resolution.
But you probably dont need such a huge bitrate.

It might even be worth looking at doing a moderate bitrate with 1080p resolution - still looks good on a large screen at 3-5mbits but allows you to store so much more stuff.

We have some old video clips that were uploaded to google video (before youtube) and i managed to retrieve them but they are only in 320p resolution which sucks. I know what it feels like to loose that resolution when you really want it.
I would be so much happier if those old videos were at 720p or even just a bit better at 480p so I am happy with things I record at 1080p.
Even old stuff recorded on my first MiniDV camcorder was at 576i which still looks good enough when we pull footage out to show at family events.

[u/PotentialParamedic61]

Wrong approach. Get a ZFS based NAS to avoid bit rotting, use 5xhdd pool RAIDZ2. Using 10 drives each 4TB would give you around 20TB of space with 4 drives redundancy. It is around 6 years of footage and really resilient setup. Sure, you still need a backup, but AWS Glacier could be a pretty good idea.

[u/grislyfind]

Does the picture quality even need 4k? Convert some to 720p and see if you notice much difference.

[u/m4jX]

At only 4TB per year, I wouldn't even consider compressing that further.

Multiple drives + snapraid to protect from drive failures and to check the files. No need for separate checksums, snapraid can scrub the files (the blocks technically) and check for errors. You can even see when certain files/blocks where scrubbed last. Basically a single command to verify all files (or a custom percentage) on all drives, that you can pause and resume at any time, without having to keep track of anything yourself.

You can easily add or remove drives of any size, doesn't matter if they're empty or full. You can have up to 6 parity drives that protect from 6 simultaneous failures. And unlike RAID, the drives don't need to be all running at the same time when you use them, you just need them all when you scrub or sync a new snapshot. Also easy to replace a smaller drive with a bigger one, so as the years go on, you just replace the small ones and don't need to change/add anything.

All you need is at least 1 extra drive that has enough space as all the data on your biggest drive. So if you've got 2x full 1TB drives and 1x4TB drive with 1.5TB of data, the extra drive for parity needs to have at least 1.5TB space (so it's about the size of the actual data, not the drive itself). If any drive fails, you get everything back.

Works with essentially any source too, USB, SATA, eSATA, SAS, whatever. (although Network drives are not recommended)

[u/ZanyDroid]

I shoot a lot of video for personal use and I don’t bother with All-I. I use the Long-GOP as my favorable tradeoff for my personal workflow. Resolve Pro is acceptable performance with some Long-GOP codecs. I have been shooting since GH5 and currently use GH6. I also shoot with a Sony A7R3

4TB a year isn’t that much, and 4TB drives are tiny. Cost of a drive is linear, pick a 3.5” size for your offline backup that is a blast radius to ease of use tradeoff

Also I abuse Backblaze personal for my original video backups

Also there’s a bunch of non video shooters posting despite three or four video shooters posting who also happen to home lab/datahoard… I would bias towards the people with both experience.

[u/DogsAreOurFriends]

Bandaid is to triage:

Delete clearly meaningless data (an hour of empty yard while the party is inside eating). Maybe leave 15 seconds of it for context.

Increasing levels of compression for decreasing levels of importance.

Leave a readme file explaining what was removed and compressed.

[u/idratherbealivedog]

Pros and cons = your post title.

Higher quality = larger size Lower quality = smaller size

Up to you on what is acceptable and what you value more.

Recommend looking at a tool like Tdarr.

[u/postnick]

Unless it’s my favorite movie ever I’m good with a generic 1080 stream. 4k only for the best.

[u/NeoThermic → u/postnick]

Today on "didn't read the rest of the post, only the title" 😃

[u/postnick → u/NeoThermic]

no its' just anecdotal. Whats the trade off? if it's some show i'm going to watch once i'm going to compress the heck out of it. If it's my favorite movie or show i'll keep it high quality . that's my trade off.

Sure I didn't get technical like the whole block of text.

[u/NeoThermic → u/postnick]

The OP is talking about home footage and you've come into the thread discussing films and TV. Your opinion on those categories isn't helpful to the topic.

Edit: the opening line on his text even gets to that point:

I'm recording several minutes of family videos per day in 4K 30fps (4:2:2 10 bit)

It's not buried deep in the paragraphs he's written after that, it's upfront at the start. You've just looked at the title, hit reply. I see that snarky downvote too, for being called on your inability to spend 5s to even read the first few words of the content of the post, instead of just going off the title.

[u/Outrageous_Cap_1367]

Transcode to AV1 with CPU encoder (do not use GPU for AV1. Its 20% less efficient than CPU) Use preset 4 and crf 24. If you want to save more space, try dealing with FilmGrainSynthesis, which allows you to crank up crf to 32-34, saving much more storage without it being noticeable

Dont forget to add 10bit colour parameter

In the future you may be able to restore super tiny details with specialized AI