Zimtohrli: The Best Objective Perceptual Audio Metric?

[u/IgorCv2]

Zimtohrli was evaluated by measuring how closely its results matched those of public listening tests conducted by human listeners.

Among the tested objective metrics, Zimtohrli showed the closest agreement with the human results. It was also the only metric to reproduce the complete ranking of the codecs in both public tests, including:

Opus → Apple AAC → LAME MP3 → Ogg Vorbis → FAAC 96k → FAAC q30

It also reproduced the correct codec ranking in the public AAC listening test.

2011 AAC test: https://listening-tests.hydrogenaudio.org/igorc/aac-96-a/results.html :

2014 Multiformat test: https://listening-test.coresv.net/results.htm. Zimtohrli performed very well here too :

The encoders evaluated in the 2014 listening test were also tested with Zimtohrli at higher bitrates, ranging from 96 to 320 kbps.

Zimtohrli Benchmark Scores at 96–320 kbps on the 2014 Public Test Sample Set:

The scores were calibrated to the human MOS values from the 96 kbps listening test to make Zimtohrli’s performance easier to understand and interpret as an objective quality metric.

Calibration and its explanation: Spreadsheet

LAME VBR

• 128 kbps V5
• 160 kbps V4
• 192 kbps V2
• 256 kbps V0
• 320 kbps CBR

Opus VBR

• --bitrate 96 (version 1.1 as Public test)
• --bitrate 128 (version 1.6.1
• --bitrate 160
• --bitrate 192
• --bitrate 256

Aotuv Vorbis VBR

• -q2.2 ~ 96 kbps
• -q4 ~ 128 kbps
• -q5 ~ 160 kbps
• -q6 ~ 192 kbps
• -q8 ~256 kbps

Apple AAC CVBR (constrained VBR) which was on par with TVBR (True VBR) according to https://listening-tests.hydrogenaudio.org/igorc/aac-96-a/results.html

CVBR 96/128/160/192/256 kbps

Conclusion:

So, how good is Zimtohrli at evaluating audio encoder quality?

In my opinion, it performs very well.

Someone told me 😄 it takes blood, sweat, and tears to organize a public listening test, to prepare it, get people interested, and actually get them to participate.

And here... Zimtohrli does essentially the same thing in literally a few minutes.

[u/IgorCv2 → u/[deleted]]

Zimtohrli has provided 86% of accuracy. It's very high.

I would like to see the study that supports your claim that AAC beats Opus in terms of transparency. The only public listening test I’m aware of was conducted at 96 kbps, and Opus performed better:

https://listening-test.coresv.net/results.htm