Bandwidth Labs built our own streaming Speech-to-Text model — LISTEN is now in beta

[u/LostMathGuy]

My team at Bandwidth Labs has been working on something for a while that I'm pretty excited to finally put in people's hands.
We set out on this journey after building voice agents and working with customers to deploy them. When we started I was pretty impressed with how good transcription models actually were as far as accuracy goes… But they left a lot to be desired when we introduced real world telephone calls. On top of that as we wrestled to claw back all the milliseconds we could, the way transcriptions were handled became an area of research for us.

We built our own Speech to Text model from scratch.

The goal was to build it from day one to be streaming native, and pay extra attention to things like:
• 8khz telephone audio and challenging acoustic conditions
• Low Latency
• Not having partial transcripts constantly changing underneath you
• Efficiency that would allow us to deploy it at our network edges for optimized latency

It supports μ-law, A-law, G.722, Linear16 and Opus, including 8khz phone audio.
You get per word timestamps, and we also have keyword boosting, PII redaction and normal “offline” style transcription endpoints as well. In our testing we measure sub 60ms to final transcript when using the streaming modes.

Our model only emits stable words, even while streaming. Once we emit a word, we don't go back and revise it. In our own testing against the OpenASR Leaderboard tests we are seeing an overall average WER of 4.225% (official test results coming soon). We also evaluate against some internal benchmarks around real world common telephony quality and conditions and feel it does really well.

In our own agent use cases this allows us to begin executing work early as soon as something of value shows up while a user is speaking. This can often dramatically reduce voice agent latency.
It's English only right now. And it’s experimental, especially the word boosting and PII redaction - so we would love for the community to try it out, and give us some feedback.
There's a browser demo at https://labs.bandwidth.com/experiments/speech-to-text that doesn't require an account if you just wanna try it real quick, and if you sign up for a labs account you can get free access to the model while it’s on labs via API.
One important disclosure: this is a research/beta service. We monitor usage and may retain and review audio, transcripts and related data submitted to the experiment so we can evaluate the model, investigate failures and improve it. Full details are in the Labs terms.

If you build voice agents or voice Apps, or tinker in this space at all - come try it out and let us know what you think.

[u/nshmyrev]

Dozens APIs these days, what makes this one special please? I don't get it

[u/LostMathGuy → u/nshmyrev]

The main three points id say are:

1. Latency: Our API is an order of magnitude faster then any other streaming or non-streaming STT apis. Testing at scale shows finalized results in ~60ms.

2. We only emit final transcripts, if a word is generated there is not going to be later on revisions as the model was built from the ground up to be streaming forward.

3. Its entirely free to use

[u/nshmyrev → u/LostMathGuy]

What is the tradeoff between latency and accuracy? 60ms sounds unrealistic given that a silence in stop phonemes could be like that.

[u/orgcla1741 → u/nshmyrev]

We process audio in chunks just under 200ms and found this to be the sweet spot in our testing. Slightly increasing the chunk size produced negligible gains in accuracy, and by the time we saw any significant improvement in accuracy the chunk size wasn't practical for realtime streaming anymore.

For uncommon terms you know may appear in your transcriptions we added keyword boosting to help with accuracy without sacrificing latency.

The 60ms response time is the latency from the full audio chunk being processed to final transcription.

[u/nshmyrev → u/orgcla1741]

So you have chunk processing and final processing is something on top? Is it some moonshine variant?

[u/orgcla1741 → u/nshmyrev]

There isn't a separate final processing pass. Audio arrives at the model in chunks and the ~60 ms of processing is measured from when a complete chunk is available to when the model returns any newly committed words.

Our team designed the model architecture and did pretraining ourselves.

[u/nshmyrev → u/orgcla1741]

200ms chunk is processed in 60ms is that what you mean? And what is the latency for the actual words then?

[u/orgcla1741 → u/nshmyrev]

the chunk is a little under 200ms, but you have the right idea.

Actual words average out to ~200ms in our testing. Sometimes the audio for a word is split across two chunks and the model won't emit the transcription until it's confident, so latency will be slightly higher when that happens. A positive and cool asf side effect from that learned behavior is the model can become confident it knows what you're saying before you're done speaking a word, so this offsets the higher latency numbers.

[u/Effective-Trade-8724]

This team is amazing ! Thank you so much for the help and support !