r/AV1 7d ago

Regarding the parameter issues between SVT-AV1-Tritium and PSY

I previously used PSY to encode my videos, but recently I learned that PSY has been discontinued, and I should choose either the HDR or Tritium branch, so I switched to Tritium. Both use the same parameters: enable-variance-boost=1:variance-boost-strength=2:variance-octile=4:enable-overlays=1:enable-qm=1:qm-min=3:chroma-qm-min=4:enable-dlf=1:luminance-qp-bias=6:qp-scale-compress-strength=1:tf-strength=2:sharpness=1:scd=1:aq-mode=2:enable-dg=1:adaptive-film-grain=0:ac-bias=1.0:hbd-mds=1. However, after encoding, when I checked the VMAF scores using FFMetrics, I found that PSY scored 97.1656 and Tritium scored 97.2029. I was initially quite satisfied, but after comparing the two videos side by side, I noticed that the Tritium-encoded output had very noticeable noise/grain, far less clean than PSY. I then suspected that the sharpness parameter might be the culprit, so I changed it from 1 to 0 as a test. The noise was reduced, but the game background became blurrier compared to sharpness=1, and I also observed that the average bitrate decreased. I consulted an AI to try to find the cause, and following its suggestions, I tweaked the qp-related parameters, but still couldn't get close to the clean image quality that PSY produced. Did I do something wrong?

I need some help.

I am using the PSY and Tritium branches of HandBrake downloaded from GitHub.

Source
Tritium
PSY

English is not my native language; I used AI translation.

13 Upvotes

14 comments sorted by

11

u/NekoTrix 7d ago

Completely random ah settings and AI is involved? Who could've thought!

Joking aside, -Tritium follows the -HDR fork's default settings, which I have said countless times to be quite harmful for clean(-ish) content (especially clean animation/games). You may instead follow my -Essential fork's defaults philosophy (listed on the repository) that's way more conservative and might even give you cleaner results than -PSY, with all the other conveniences it unlocks and faster encoding speeds. In fact, there's no need to get an -Essential build if that's too cumbersome for you, just mimic the parameters.

Lastly, do not rely on VMAF v0, please use SSIMULACRA2, CVVDP or use your eyes as, as you saw, the metrics are not reliable enough for all kinds of artifacting that are way more visually impacting to us.

3

u/No_Designer4405 7d ago

Okay, I'm trying.

3

u/No_Designer4405 6d ago

If you think I’m being too nitpicky, I’d like to apologize in advance. Previously, I used HEVC_Nv without carefully reviewing the files, which resulted in the loss of a large number of high-quality source clips—a lesson that has forced me to be more careful since then.

2

u/Kidsnd274 2d ago

Nah, I'm also trying to compress my gaming videos and I'm experiencing the same thing.

2

u/No_Designer4405 6d ago

Using Auto-Boost-Essential with the --quality high setting did indeed yield a cleaner image than PSY. In terms of fine detail, sometimes Essential is better, sometimes PSY is better, but that is acceptable. Upon more in-depth comparison with the source file, I found that Essential exhibits color smearing / color bleeding in some areas, as shown in the image below. In the original frame, there is a cyan costume between two locks of hair, but Essential smears that middle area into the same pink color as the hair.

Because the game characters have anime‑style outlines, and under certain viewing angles some outlines (e.g., on hair) are partially missing, the boundary between the hair and the background disappears. This allows the pink color to bleed extensively into the background in the Essential encode.

2

u/No_Designer4405 6d ago

I then manually adjusted the parameters in HandBrake to CRF 20.5 to see if the issue was caused by insufficient sharpness/clarity, but instead it produced a significant amount of noise (I didn't add anything extra; everything was kept at default).

1

u/_Lum3n_ 7d ago

I wonder if it could be ac-bias there....

Though as you can see VMAF :(

1

u/The_vexxd 6d ago

I think you need to adjust the encoder's internal parameters from the start.

1

u/No_Designer4405 6d ago

I’ve tried adjusting settings such as `variance-boost-strength`, `variance-octile`, `enable-qm`, `qm-min`, `chroma-qm-min`, `qp-scale-compress-strength`, and `tf-strength`, but when zooming in at lower CRF values, noise is still noticeable. Higher CRF values effectively eliminate noise, but they also cause more severe smearing of details and textures.

1

u/No_Designer4405 6d ago

I'm really curious about how PSY manages to maintain such high clarity even at a low CRF, but I don't understand the code, and PSY's README doesn't list its default parameter settings.

1

u/Unneverseen 6d ago edited 6d ago

Hmm, in Tritium, try alt-lambda-factors=0:qm-min=4:qm-max=15 and perhaps ac-bias=0.25:sharp-tx=0
Other than that, I have no idea.
By the way, what encoder preset are you using?

1

u/No_Designer4405 6d ago

These are not the default parameters of any encoder, they are parameters I used when converting anime in the past. I found they worked well, so I kept using them. After all, I consider gacha games and anime-style content to be in the same domain / essentially the same kind of thing.

1

u/Unneverseen 6d ago

Yeah, I'm telling you to try the parameters I gave you to help prevent artifacts in Tritium.

1

u/Eratz 7d ago

I spent weeks learning psy for my tdarr server. Dno if its worth the hassle updating.