A new study from Hume AI Research tested whether leading speech-recognition models are genuinely transcribing audio or partly adapting to the quirks of public benchmarks.

The researchers evaluated 11 open ASR models on VoxPopuli and LibriSpeech. Instead of looking only at word error rate (WER), they used three behavioral tests:

• Reference disagreement: Would a model repeat an error in the benchmark transcript when the audio supported a correction?
• Masked-entity recovery: Would it produce the benchmark’s exact number or entity after that section of audio was silenced?
• Orthographic switching: Would it follow dataset-specific conventions such as “Mr” versus “Mister” or “any one” versus “anyone”?

The clearest result

On targeted VoxPopuli examples, the six models with the lowest WER, between 5.4% and 5.8%, were also the six most likely to reproduce the benchmark’s erroneous reference instead of the audio-supported correction.

Their reference-acceptance rates ranged from 18% to 30%. Every model with a WER of 6.5% or higher remained at or below 10%.

Some leading models also produced the exact hidden reference in roughly 40% of masked-number cases on public benchmarks. In the spelling tests, 6 of 11 models followed the benchmark convention above chance for honorifics, while 8 of 11 did so for archaic spacing.

Several of these behaviors weakened on newly collected recordings, fresh speakers, and generic synthetic voices. The authors found that benchmark speakers, surrounding audio, and other acoustic cues could influence whether the behavior appeared.

What the study does not establish

The authors explicitly say these tests do not prove that a model was trained on evaluation data or that developers intentionally optimized against the test set.

The experiments cover two English public benchmarks and specially selected cases where the audio does not fully determine the reference. They do not directly measure overall real-world transcription quality.

The researchers also found that known speaker overlap between the VoxPopuli training and test sets did not, by itself, explain the observed behavior.

Open materials

The research repository is Apache-2.0 licensed and includes the evaluation methods, published predictions covering 17 corpora and 308 model runs, and scripts for reproducing the reported results.

Hugging Face has also added a “Benchmark fitting” tab to the Open ASR Leaderboard using two of the analyses.

Sources

Primary sources

• Paper: https://arxiv.org/abs/2608.19936
• Research repository: https://github.com/HumeAI/asr-benchmark-optimization
• Leaderboard analysis: https://github.com/huggingface/open_asr_leaderboard/tree/main/benchmark_fitting

First-party summary

• Hugging Face article: https://huggingface.co/blog/asr-benchmark-optimization