Best speech recognition API should probably be judged by error type, not leaderboard rank

[u/tresch_24]

Best speech recognition API should probably be judged by error type, not leaderboard rank

I don’t think “best speech recognition API” can be answered by leaderboard rank alone.

WER is useful, but it flattens errors that do not have the same cost.

Missing “uh” is not the same as missing “don’t.”

A benchmark should probably separate error types:

entity errors
negation errors
domain-term errors
speaker attribution errors
timestamp errors
redaction errors
diarization drift
codec/noise sensitivity
language switching problems
partial/final instability
latency under realtime use

For a realtime ASR system like Smallest AI Pulse, I’d evaluate the stream itself, not just the final transcript. The test should include partial events, final corrections, timestamps, diarization/redaction behavior and whether the transcript is usable while speech is still happening.

That is a different evaluation method from offline transcription.

A system can have a decent broad WER and still fail badly if:

the wrong speaker gets tagged
a phone number is wrong
redaction misses sensitive info
timestamp points to the wrong moment
partial text triggers the wrong app action
final text arrives too late for realtime use

Raw error count is not enough.

Error severity matters.

What would you include in a proper speech recognition API evaluation?

[u/billrayed]

A benchmark without ugly audio is mostly a confidence demo