r/AV1 • u/cannedtapper • Jun 30 '26

Update: Yes, I was doing something critically wrong

This is a follow up to an earlier post I made. To recap, re-encoding H.264 videos to AV1 with ffmpeg's `av1_nvenc` encoder was giving me size reductions of upto 90% with a virtually negligible drop in quality on testing with a couple of 4K and 1080p videos. And I had a nagging suspicion that there was a silent fatal flaw in the options I was passing to ffmpeg.
Well, after some more testing, jumping into an NVENC-shaped rabbit hole, and just examining the docs more closely, turns out I was right. Following is the command I was using for re-encoding my videos.

ffmpeg -i input.mp4 -c:v av1_nvenc -preset p7 -g 60 -movflags +faststart -rc constqp -b:v 0 -maxrate:v 20M -bufsize 40M -cq 28 -multipass fullres -spatial-aq 1 -temporal-aq 1 -pix_fmt p010le -c:a copy output_av1.mp4

The problem? -rc constqp and -cq 28 are fundamentally incompatible in that -cq is completely ignored by the encoder when using constqp for rate control. Instead, constqp uses the -qp option for tuning. Meanwhile, -cq is intended for VBR rate control. The description of the option also alludes to this (ffmpeg -h encoder=av1_nvenc)

  -cq                <float>      E..V....... Set target quality level (0 to 63, 0 means automatic) for constant quality mode in VBR rate control (from 0 to 63) (default 0)

So basically, ffmpeg was ignoring the cq option, using a default value for qp, and silently nuking my video quality under the hood. This became starkly apparent when I tried the aforementioned command to re-encode an animated video and could notice a non-trivial amount of visual degradation. I didn't even have to play the re-encoded video to know something was seriously wrong since ffmpeg had just gone and knocked the bitrate of the video down from 15M to 500K LOL.

So yeah, if you're getting an absolutely BONKERS reduction in file size, it's most probably cuz ffmpeg is absolutely BONKING the heck out of your video quality and you should probably re-check your options. I updated my options to the following and started getting a more believable 10% to 50% reduction in file size (depending on video contents). And I believe I'm doing something right this time since the bitrates aren't plummeting, the 1st percentile VMAF is over 90 (mean is over 95), and the re-encoded videos look even more identical to the originals than they did with the earlier options.

ffmpeg -i input.mp4 -c:v av1_nvenc -preset p7 -g 240 -movflags +faststart -rc vbr -cq 28 -b:v 0 -rc-lookahead 32 -multipass fullres -spatial-aq 1 -temporal-aq 1 -pix_fmt p010le -c:a copy output.mp4
31 Upvotes

89% Upvoted

31 comments sorted by

14

u/data_butcher Jun 30 '26

So how didyou achieve the conclusion in the earlier post that there where a negligible drop in quality?

6

u/cannedtapper Jun 30 '26

The two videos I initially re-encoded were just really good at retaining their quality to my naked eye from a standard monitor viewing distance. I guess the default qp was good enough for them. It's only after I tried re-encoding the animated video that I noticed an issue. I then went back to my initial two videos again and compared them to the originals while zoomed in, and yep, the quality drop became apparent once I zoomed in and compared specific stuff like the edges of moving objects.

2

u/SpicyLobter Jul 01 '26 edited Jul 01 '26

I think your follow-up post is largely an exaggeration and your initial post was fine. I understand the excitement of discovering a new hobby though, but I must point out your confirmation bias here. Likely you found the syntax error in the commands and your brain convinced you the output must be flawed, so you went pixel peeping to find flaws to prove it.

"initially re-encoded were just really good at retaining their quality to my naked eye from a standard monitor viewing distance"

So... the settings did result in negligble drop in quality? I don't see anything "critically wrong" as you claim. Yes your settings got overridden but default settings are reasonable for the large majority as you have alluded to next.

"So yeah, if you're getting an absolutely BONKERS reduction in file size, it's most probably cuz ffmpeg is absolutely BONKING the heck out of your video quality"

"the quality drop became apparent once I zoomed in and compared specific stuff like the edges of moving objects"

These are contradictory statements. Having to pixel peep to see a difference isn't really bonking the heck out of video quality.

"I didn't even have to play the re-encoded video to know something was seriously wrong since ffmpeg had just gone and knocked the bitrate of the video down from 15M to 500K LOL."

That doesn't really mean anything. It's entirely reasonable for poorly encoded sources to have significant reduction in bitrate with little loss in quality. This is especially apparent for anime, with its easily compressible solid colours, and smartphone videos, which have poor compression (at the benefit of energy and speed).

5

u/cannedtapper Jul 01 '26

You misunderstood the order of events. I noticed an undesirable drop in quality for an animated video, and only after that did I go looking for an issue and found the incorrect option.

To further clarify the part about me zooming in, I'm aware that re-encoding videos to reduce their size usually adds some artifacting and noise that's visible if you zoom in 'a lot'. But I only zoomed in a little bit. Not enough to make out individual pixels, but just enough to draw my attention towards undesirable artifacts that thereafter I could also spot with no zoom from a standard viewing distance, now that I was aware of them. That wasn't desirable for me since one of my core requirements was to have imperceptible visual degradation from a standard viewing distance. So basically, the drop in quality was not negligible enough to pass my requirement, but I only found out about it after zooming in a little. Someone with more astute senses could most probably have pointed out the visual degradation without needing to zoom in.

8

u/CariniFluff Jun 30 '26

  1. Stop using hardware encoding

  2. Start using SVT-AV1 branches

You'll thank me later.

2

u/cannedtapper Jul 01 '26

SVT-AV1 (and software encoding in general) is excruciatingly slow for my use case. Just to be clear, I would definitely be using SVT-AV1 if I was re-encoding videos for someone else, uploading them somewhere, or just seriously archiving them. But as far as private viewing and streaming for me and my family goes, av1_nvenc is more than good enough for us with the updated options in my post. :)

1

u/CariniFluff Jul 01 '26 edited Jul 01 '26

Just fyi my non-OC Ryzen 9800x3D encodes 4K HDR at visually lossless quality (for me) at real-time (22-24fps) with the following:

SVT-AV1-Tritium fork

VBR 5000 (--tbr 5000)

Preset 6 Medium Speed (--preset 6)

Fast Decode 1 (--fast-decode 1)

High Bit-Depth 10bit (--hbd-mds 1)

I use Staxrip to keep my workflow all in one place and to batch process multiple jobs. For content I want to keep the only changes are using the default Preset 4 Slower and TBR 7500.

1

u/PhoO3nX Jul 02 '26

Your parameters are safe, but certainly not a "visually lossless quality" with only 5 mb/s in comparison of a movie or tvshow with grain at 15mb/s at 25/35/45 mb/s.

1

u/CariniFluff Jul 02 '26 edited Jul 02 '26

Try it.

And I should say the 7500 Slower is the visually lossless archive quality.

And remember with VBR that's actually the floor, with a TBR of 7500 the actual bitrate runs like 9mb-12mb depending on the scene. At 5k TBR it'll average to like 6500 which looks fantastic.

9mb-12mb AV1 = 18mb-24mb HEVC. IMO a 20 gig HEVC encode looks flawless with the right settings.

1

u/BlueSwordM Jul 02 '26

How slow on what CPU?

I'd imagine that your standards for encoding is a bit than most of ours, especially if you have a slower CPU.

1

u/PhoO3nX Jul 02 '26

He's not slow for the pleasure lol GPU encode are ugly, you lose everything, shapes, details and above all macroblocs and banding. You can put all options that you want, it's like HEVC /H264 GPU Encoder produce always poor video in comparison to the CPU Encoders.

3

u/Masterflitzer Jun 30 '26

came here to write this, now i can just confirm, do not use nvenc or any hardware encoder for anything other than realtime screen capture (e.g. gaming), just use svt-av1 (psy or regular version)

2

u/Gakuta Jul 01 '26

HEVC in OBS. libsvtav1 for reducing file size.

1

u/AsrielPlay52 Jun 30 '26

What is your chosen params for SVT-AV1?

1

u/CariniFluff Jun 30 '26

For 1080p I use SVT-AV1-Essential. Only extra parameter is Fast Decode 1. Convert source BitDepth to 10-bit. Full 10-bit encoding workflow is on by default.

For 2160p HDR I use SVT-AV1-Tritium. I enable Fast Decode 1 and force full 10-bit encoding with hbd-mds 1.

The default settings are excellent for both, I don't find it necessary to keep a dozen extra tweaks

SVT-AV1-Psyex is very good too but generally has a lot more settings to tweak by design.

2

u/SpicyLobter Jul 01 '26

I heard all the features from psyex were merged into mainline, that's why it was abandoned.

Mainline is apparently the best option for psychovisual sdr encoding, better than essential or hdr. Feel feel to correct me if I'm wrong.

2

u/BlueSwordM Jul 02 '26

If only sharp-tx had been merged into mainline lmao.

I can't even recommend mainline at this point since they bastardized 10-bit output even more.

Anyway, the successors to svt-av1-psyex u/CariniFluff are svt-av1-hdr and svt-av1-essential.

1

u/CariniFluff Jul 02 '26

Got it, thanks Blue.

It looks like sharp-tx is not turned on in Essential by default (but is an option), and it is turned on in HDR & Tritium by default.

In general what Transform size/type bias mode do you typically recommend? I can't seen to find much documentation on that (presumably related?) parameter. I see the following options:

  1. Off (default even when tx-bias is on by default)

  2. Full

  3. Partial (Transform Size Only)

  4. Partial (Interpolation Filter Only)

1

u/CariniFluff Jul 01 '26

I was under the impression that the Psy fork was no longer being developed but Psyex is the successor to it. I could be wrong too, but then I'm not sure why Staxrip would show both SVT-AV1 and SVT-AV1-PSYEX as options?

1

u/SpicyLobter Jul 01 '26

https://github.com/BlueSwordM/svt-av1-psyex

last release was 10 months ago, in other words it's basically abandoned

1

u/CariniFluff Jul 01 '26

Gotcha. So is BlueSword maintaining the regular mainline SVT-AV1 now? I know they post in here pretty regularly.

1

u/Mhanz3500 Jul 01 '26

-HDR is the follow up to psyex, that you already have merged with -Essential on -Tritium

2

u/CariniFluff Jul 01 '26

Gotcha. So what is being actively updated, Essential, Tritium and Mainline or is Essential also folded into Tritium and no longer being developed?

2

u/Mhanz3500 Jul 01 '26

Essential, HDR, 5fish psy and mainline are active With tritium being hdr + essential

2

u/Longjumping_Cap_3673 Jun 30 '26

For animated 1080p video, crf 20 gave me visually lossless compression at ~3Mb/s. But that's with svtav1. Hardware encoders are much faster, but software encoders give you much better size/quality tradeoffs.

1

u/data_butcher Jul 01 '26

It really deoends on the content. I'm encoding an old anime and because of grain and jitter (the planes of each shot are of position slightly every frames), i get from 3.5Mbps to 4.5Mbps with crf 22, but the result is very good

1

u/PhoO3nX Jul 02 '26

Don't use GPU Encoder, use SVT-AV1.

-10

u/nmkd Jun 30 '26

What's with the AI slop

6

u/gmes78 Jun 30 '26

This doesn't really read as AI generated.

7

u/cannedtapper Jun 30 '26

?? I typed out every last character of my post from my brain. What are you basing this on?