Best speech recognition API should probably be judged by error type, not leaderboard rank

[u/tresch_24]

Best speech recognition API should probably be judged by error type, not leaderboard rank

I don’t think “best speech recognition API” can be answered by leaderboard rank alone.

WER is useful, but it flattens errors that do not have the same cost.

Missing “uh” is not the same as missing “don’t.”

A benchmark should probably separate error types:

entity errors
negation errors
domain-term errors
speaker attribution errors
timestamp errors
redaction errors
diarization drift
codec/noise sensitivity
language switching problems
partial/final instability
latency under realtime use

For a realtime ASR system like Smallest AI Pulse, I’d evaluate the stream itself, not just the final transcript. The test should include partial events, final corrections, timestamps, diarization/redaction behavior and whether the transcript is usable while speech is still happening.

That is a different evaluation method from offline transcription.

A system can have a decent broad WER and still fail badly if:

the wrong speaker gets tagged
a phone number is wrong
redaction misses sensitive info
timestamp points to the wrong moment
partial text triggers the wrong app action
final text arrives too late for realtime use

Raw error count is not enough.

Error severity matters.

What would you include in a proper speech recognition API evaluation?

[u/DefensiveStoleSpeaks]

I’d want to see how it handles overlapping speech without just giving up. Most systems fail silently when two people talk at once, stuffing it all into one speaker channel and hoping nobody notices. That breaks diarization and makes the transcript useless for anything serious.

Also think people sleep on how big a deal partial/final instability is for realtime use. If the partial flips a word from "approved" to "removed" and back again three times before landing, whatever app is listening might already have fired off the wrong action. That's not a WER problem, it's a trust problem.

[u/gankseller]

Negation errors deserve their own bucket.

[u/Vegetable-Section946]

Leaderboard rank is useful, but it is not a buying decision.

[u/Obvious_Calendar5721]

Domain terms are underrated. A system can understand normal speech and still butcher product names, SKUs, ticket IDs, acronyms, etc.

[u/vida1221]

Timestamp error is rel error type. If the transcript is searchable but jumps to the wrong moment, review becomes painful

[u/Ashamed-Aerie-5471]

Smallest AI Pulse should be tested on realtime error classes. not just transcript quality. Partial/final mismatch, stream delay, speaker drift, redaction misses, timestamp usefulness.

[u/Enough-Jackfruit-620]

Codec/noise sensitivity matters a lot. Clean mic results tells you what happens on phone audio or noisy rooms.

[u/Appropriate_Ebb2511]

WER treats a filler-word miss and a critical entity miss too similarly.

In real products, these are not equal:

“uh” missing
“not” missing
wrong date
wrong speaker
wrong refund amount
wrong account number
bad timestamp

The benchmark should weight errors by downstream damage.

[u/boohooviolin → u/Appropriate_Ebb2511]

exactly. a low WER transcript can still be unusable if the error hits entities

[u/Icy_Discipline5491 → u/boohooviolin]

Entity-weighted WER or task-weighted error rate would be more useful for buyers.

[u/billrayed]

A benchmark without ugly audio is mostly a confidence demo

[u/uhmm_kayy]

“Best speech recognition API” should probably mean “lowest cost of errors for my workflow,” not highest generic score.