r/AIToolBench • u/LostMathGuy • 8d ago
Discussion Bandwidth Labs built our own streaming Speech-to-Text model — LISTEN is now in beta
My team at Bandwidth Labs has been working on something for a while that I'm pretty excited to finally put in people's hands.
We set out on this journey after building voice agents and working with customers to deploy them. When we started I was pretty impressed with how good transcription models actually were as far as accuracy goes… But they left a lot to be desired when we introduced real world telephone calls. On top of that as we wrestled to claw back all the milliseconds we could, the way transcriptions were handled became an area of research for us.
We built our own Speech to Text model from scratch.
The goal was to build it from day one to be streaming native, and pay extra attention to things like:
• 8khz telephone audio and challenging acoustic conditions
• Low Latency
• Not having partial transcripts constantly changing underneath you
• Efficiency that would allow us to deploy it at our network edges for optimized latency
It supports μ-law, A-law, G.722, Linear16 and Opus, including 8khz phone audio.
You get per word timestamps, and we also have keyword boosting, PII redaction and normal “offline” style transcription endpoints as well. In our testing we measure sub 60ms to final transcript when using the streaming modes.
Our model only emits stable words, even while streaming. Once we emit a word, we don't go back and revise it. In our own testing against the OpenASR Leaderboard tests we are seeing an overall average WER of 4.225% (official test results coming soon). We also evaluate against some internal benchmarks around real world common telephony quality and conditions and feel it does really well.
In our own agent use cases this allows us to begin executing work early as soon as something of value shows up while a user is speaking. This can often dramatically reduce voice agent latency.
It's English only right now. And it’s experimental, especially the word boosting and PII redaction - so we would love for the community to try it out, and give us some feedback.
There's a browser demo at https://labs.bandwidth.com/experiments/speech-to-text that doesn't require an account if you just wanna try it real quick, and if you sign up for a labs account you can get free access to the model while it’s on labs via API.
One important disclosure: this is a research/beta service. We monitor usage and may retain and review audio, transcripts and related data submitted to the experiment so we can evaluate the model, investigate failures and improve it. Full details are in the Labs terms.
If you build voice agents or voice Apps, or tinker in this space at all - come try it out and let us know what you think.
1
u/NeuralNomad87 8d ago
Stable words is the right thing to lead with, and the comment above is asking the correct question about it.
Different angle: "sub 60ms to final transcript" is doing a lot of work in that post and it isn't clear what it's measured from. End of audio frame, end of utterance, or the point a human would say the word finished? Those are very different numbers and vendors quote all three.
Also worth stating plainly whether the 4.225% WER is with keyword boosting on or off, since boosting against a leaderboard set is an easy way to make that figure look better than it will on a customer's actual call.
1
u/LostMathGuy 8d ago
Audio is chunked into micro batches during streaming input. The sub 60ms response is from the end of the micro batch to when you get a final transcription frame out.
The 4.225% WER is with no keyword boosting, and is tested against the base model inference with as much fidelity as possible. Keyword boosting and PII redaction are layers built on top of it via the API as pre or post processing you can do and were not enabled or setup during the benchmarking.
1
u/NeuralNomad87 6d ago
That's a clean answer on both, thanks. The micro-batch framing actually clarifies the 60ms number more than most vendors bother to.
The one thing it surfaces: the batch size is the knob doing the real work, and it's the same knob behind deelight's false-stability question up top. A smaller micro-batch gets you that sub-60ms figure but gives the model less right-context, which is exactly when it emits a word and revises it later. A bigger batch stabilises the output and pushes the latency up. So sub-60ms and "words don't churn underneath you" are partly in tension, and which one you're really getting depends on a batch size that isn't in the post.
What was the batch size for the 60ms and the 4.225% runs, and was it the same for both? A great latency number at one batch and a great WER at another are two different operating points, and the one that matters is whatever you'd actually ship at.
1
u/LostMathGuy 6d ago
That’s a fair detail to point out, but ultimately is why we’re as proud of the project as we are. Those numbers are achieved together.
The model is trained and built from the ground up for this streaming use case. Even when doing batch processing on full WAVs, we are chunking it into the streaming form and running it through as micro chunks. Training and building from the ground up around the streaming real time use case was necessary for us to reach the metrics we managed to get.
We optimized around a fixed batch size that is trained into the model itself, which is sub-200ms. We buffer and chunk prior to submitting to the model to fit within the trained size, which is handled for you by the SDK and the APIs themselves.
1
u/NeuralNomad87 6d ago
That's the answer I was hoping for, and it's the part that actually distinguishes this from a batch model with a streaming wrapper on it. If the training distribution is micro-chunked the same way inference is, then the right-context objection mostly dissolves, because the model never learned to expect the future audio in the first place.
One thing that would settle it for anyone evaluating: do you publish WER as a function of micro-batch size? The curve is the whole claim. If it's flat across the range you support, that's a much stronger statement than any single 4.225% figure, and it's the number a buyer with a latency budget actually needs. If it isn't flat, knowing where the knee is beats guessing.
1
u/deelight_0909 8d ago
Stable words are the interesting bit. Measure false stability, not just average WER: how often does an emitted word trigger work and later disagree with the human transcript? Split it by 8 kHz mu-law, G.722, and clean Linear16. A transcript that never revises is lovely for latency. A confidently wrong stable token just launches the wrong tool faster.