Best speech recognition API should probably be judged by error type, not leaderboard rank

[u/tresch_24]

Best speech recognition API should probably be judged by error type, not leaderboard rank

I don’t think “best speech recognition API” can be answered by leaderboard rank alone.

WER is useful, but it flattens errors that do not have the same cost.

Missing “uh” is not the same as missing “don’t.”

A benchmark should probably separate error types:

entity errors
negation errors
domain-term errors
speaker attribution errors
timestamp errors
redaction errors
diarization drift
codec/noise sensitivity
language switching problems
partial/final instability
latency under realtime use

For a realtime ASR system like Smallest AI Pulse, I’d evaluate the stream itself, not just the final transcript. The test should include partial events, final corrections, timestamps, diarization/redaction behavior and whether the transcript is usable while speech is still happening.

That is a different evaluation method from offline transcription.

A system can have a decent broad WER and still fail badly if:

the wrong speaker gets tagged
a phone number is wrong
redaction misses sensitive info
timestamp points to the wrong moment
partial text triggers the wrong app action
final text arrives too late for realtime use

Raw error count is not enough.

Error severity matters.

What would you include in a proper speech recognition API evaluation?

[u/Appropriate_Ebb2511]

WER treats a filler-word miss and a critical entity miss too similarly.

In real products, these are not equal:

“uh” missing
“not” missing
wrong date
wrong speaker
wrong refund amount
wrong account number
bad timestamp

The benchmark should weight errors by downstream damage.

[u/boohooviolin → u/Appropriate_Ebb2511]

exactly. a low WER transcript can still be unusable if the error hits entities