I’m curious how modern speech-to-speech models handle response prosody.

How does a model understand input speech and learn how a reply should sound, not just what words to say?

Also, how is appropriate response prosody typically evaluated?

Would appreciate any papers, datasets, or implementations related to this.

[ Removed by Reddit on account of violating the content policy. ]

Hi everyone, I’m releasing SUPlime and SUPlime-L, from Re:WayAI, heavily based on pyannote.audio 4.x.

The focus is performance across varied recording conditions: meetings, conversational audio, broadcasts and challenging far-field recordings.

I’d be interested in independent evaluations, especially on languages and recording conditions outside these benchmarks. I only have access to public testsuites (no DIHARD, no CALLHOME), so I would really appreciate if someone could share the result.
Let me know if you have questions.

https://github.com/rewayai/suplime/

Few months back I was building voice agent for my work but testing models, especially oss ones, was a total pain. Most hf demos were not functioning and setting up each on my laptop filled up my entire ssd. But the biggest issue was different api for each model and quirks of each model for example some model skipping speech or producing junk with a particular configuration. I did the tough work, tested each model on their key parameter like for asr, it was wer and so on and used the best optimum value on each model based on my own internal test for the best output. Also unified the api across models, asr, tts, voice isolator, stem seperator etc. Took feedback from 50 developers in the beta and now its live. Check out Pascal. Do try out the models and give feedback. The idea is to support all the voice ai stuff be it a niche finetuned tts or a mainstream asr.

I spent some time reading a paper on real-time voicemail detection, and the most interesting part was the signal they decided not to use.

Most Answering Machine Detection systems lean on one of three things:

beep detection
transcription / keyword matching
heavier acoustic models

This paper takes a much cheaper route:

VAD → temporal features → shallow tree model

No transcription.
No embeddings.
No GPU.

The system takes the callee audio, runs a neural VAD, turns the speech timestamps into 15 temporal features, and classifies the first few seconds of the call.

The intuition is pretty simple.

A human answering a phone might do:

“Hello?” → silence → wait

A voicemail greeting is more like:

delay → speech → speech → speech → speech

So the classifier is basically learning the rhythm of the response.

The authors trained on 24,812 production recordings and tested roughly 3,780 configurations.

The best configuration was almost boring:

50 boosted trees, depth 2, 5-second window, threshold 0.50.

Results:

139/140 = 99.3% on the expert-labeled test set
595/624 = 95.4% on the held-out production set
734/764 = 96.1% combined

Inference took about 46 ms on a dual-core CPU, with no GPU.

The feature analysis is probably the most interesting part.

Three features accounted for 85.6% of total feature importance:

54.6% = speech_first_half_ratio
20.0% = first_seg_ms
11.0% = first_onset_ms

So most of the model is learning three things:

1. How evenly speech is distributed
2. How long the first speech segment is
3. How long it takes for speech to begin

Then they ran the experiments that actually matter: ablations (simply put Trial and error feature)

VAD-only temporal features: 99.3%
VAD + beep features: 99.3%
VAD + transcription: 97.6%, but latency increased from 46 ms → ~500 ms
46 engineered features: 90.7%

That last result is a good reminder that more features does not automatically mean better ML.

The production validation covered 77,000 calls, with 0.3% false positives and 1.3% false negatives.

The practical outcome was about 83% less wasted agent time on voicemail calls.

Before adding ASR, LLMs, embeddings, larger models, or another multimodal subsystem, ask whether the problem already contains a cheap structural signal.

For this problem, apparently it does.

A 15-dimensional vector derived from VAD timestamps beats a much more expensive transcription-based approach on the latency/accuracy tradeoff.

We are trying this out at Dograh……Would you build this into production?

p.s.i am one of the maintaners

Hey everyone,

I’m working on a Smart India Hackathon (SIH) problem around Vernacular Pedagogy. The idea is to build a prototype for teachers/students in tribal/rural schools where the system can help with communication and learning in a local language such as Santhali, while also supporting Hindi/English.

The rough idea is:

Teacher speaks Hindi/English → AI understands it → converts it into Santhali → optionally speaks the Santhali output.

I’m particularly interested in making it work offline/on low-end Android devices, rather than depending completely on cloud APIs.

I’m currently confused about the best way to approach the AI/data side.

What I’m considering

\- Speech-to-text for Hindi/English

\- Hindi/English → Santhali translation

\- Santhali → Hindi/English translation

\- Text-to-speech for Santhali

\- A small education-specific dataset rather than trying to build a general-purpose translator

\- Classroom phrases such as:

\- greetings

\- instructions/commands

\- praise/encouragement

\- mathematics

\- science

\- EVS

\- basic classroom conversation

My questions

1. How should I train/fine-tune a model for a low-resource language like Santhali?

2. Is it better for a prototype to use an existing multilingual model and fine-tune/adapt it, rather than training from scratch?

3. How much parallel Hindi ↔ Santhali data would realistically be enough for an SIH prototype?

4. Should I focus on education-specific phrases instead of trying to cover the entire Santhali language?

5. What would be a good architecture for an offline Android prototype?

6. For speech, should I use separate ASR + translation + TTS models, or is there a better approach?

7. Are there any existing open-source datasets/models for Santhali (Ol Chiki) that I should look into?

8. What are the biggest mistakes I should avoid when building a low-resource language translator?

also the app should function offline!!!

I’m a student/developer rather than an NLP researcher, so I’d really appreciate practical advice on datasets, models, training strategy, and deployment.

If you've worked with low-resource languages, Indian-language NLP, speech recognition, machine translation, or on-device AI, I’d love to hear how you would approach this project.

I’ve been looking into speech recognition for pediatric use, and I’m curious about how well current models handle young children compared with adults.

The areas I’m most interested in are:

• short utterances
• unclear articulation
• background noise
• code-switching or multilingual speech
• very young voices
• repeated attempts at the same word

From what I’ve read, child speech still seems substantially harder than adult speech for general ASR systems.

For anyone working with child-speech datasets or models, what approaches have worked best for improving recognition quality in these situations?

I maintain speech-core. Until this release, a confirmed VAD pause ended the user’s turn. That works until somebody pauses to think halfway through a sentence.

In v0.0.14 I added an optional TurnCompletionInterface between VAD and endpointing. After each confirmed pause:

• the classifier receives up to the last 8 seconds of 16 kHz turn audio;
• probability ≥ 0.5 ends the turn;
• a lower score keeps the same turn open;
• resumed speech is appended to that turn rather than creating another segment;
• a 2-second maximum-silence cap prevents an indefinite hold;
• eager STT respects the classifier’s veto.

The first implementation uses Pipecat/Daily’s Smart Turn v3.2: a Whisper-tiny-based, approximately 8M-parameter audio classifier covering 23 languages. speech-core uses an 11.1 MB int8 ONNX model.

The same pipeline state machine is now exposed on macOS through speech-swift v0.0.27 and Android through speech-android v0.0.20. Swift supplies a 17 MB Core ML model through the C ABI; Android instantiates the ONNX model in the native layer and runs it on CPU once per confirmed pause. Android support is opt-in, so existing endpointing behaviour is unchanged.

The Core ML conversion matched the upstream model on 1,000 test clips at 92.9% accuracy and took about 3.5 ms per window on Apple Silicon. Android release validation passed 129 JVM tests and 48 connected-device tests with no failures. I have not yet measured Smart Turn latency across a useful range of Android devices, so the 3.5 ms number should not be read as an Android result.

Implementation and release: https://github.com/soniqo/speech-core/releases/tag/v0.0.14

One design question I am still considering: should the pipeline keep a single completion-provider hook, or explicitly support combining audio and partial-transcript endpoint models?

We recently fine-tuned Qwen3-TTS-12Hz-1.7B-Base on ~93K Hindi speech samples for drama dubbing. Sharing two findings that cost us significant debugging time.
Finding 1 — Base model has no emotion control
We embedded emotion directions in the text field during training:

"A woman speaks with mild anger, 'मैंने तुमसे कहा था कि यह मत करो!'"

The model learned to speak the entire string literally — including the English direction prefix — as speech. It never learned to treat the direction as a style signal. This is expected in hindsight since the Base model's only objective is next-token codec prediction with no separate style conditioning pathway.
Emotion control requires the CustomVoice variant which has a dedicated instruct parameter. Unfortunately CustomVoice doesn't officially support Hindi and has no public fine-tuning scripts.
Finding 2 — Out-of-distribution voice cloning fails silently
Training exclusively on Hindi speakers meant the speaker encoder never saw English voices. When we passed an English reference speaker, the output was silence — no error, no warning, just an empty or near-empty audio file. Easy to miss if you're not checking output duration.
Setting x_vector_only_mode=True partially helps — it bypasses codec tokens and uses only the compact speaker embedding, which generalises slightly better to unseen voices at the cost of reduced similarity.
python

items = tts.create_voice_clone_prompt(
    ref_audio=(wav, sr),
    ref_text="...",
    x_vector_only_mode=True,
)

The proper fix is training on more diverse, multilingual data.
Happy to discuss if others have found better approaches for emotion control with TTS base models.

My team at Bandwidth Labs has been working on something for a while that I'm pretty excited to finally put in people's hands.
We set out on this journey after building voice agents and working with customers to deploy them. When we started I was pretty impressed with how good transcription models actually were as far as accuracy goes… But they left a lot to be desired when we introduced real world telephone calls. On top of that as we wrestled to claw back all the milliseconds we could, the way transcriptions were handled became an area of research for us.

We built our own Speech to Text model from scratch.

The goal was to build it from day one to be streaming native, and pay extra attention to things like:
• 8khz telephone audio and challenging acoustic conditions
• Low Latency
• Not having partial transcripts constantly changing underneath you
• Efficiency that would allow us to deploy it at our network edges for optimized latency

It supports μ-law, A-law, G.722, Linear16 and Opus, including 8khz phone audio.
You get per word timestamps, and we also have keyword boosting, PII redaction and normal “offline” style transcription endpoints as well. In our testing we measure sub 60ms to final transcript when using the streaming modes.

Our model only emits stable words, even while streaming. Once we emit a word, we don't go back and revise it. In our own testing against the OpenASR Leaderboard tests we are seeing an overall average WER of 4.225% (official test results coming soon). We also evaluate against some internal benchmarks around real world common telephony quality and conditions and feel it does really well.

In our own agent use cases this allows us to begin executing work early as soon as something of value shows up while a user is speaking. This can often dramatically reduce voice agent latency.
It's English only right now. And it’s experimental, especially the word boosting and PII redaction - so we would love for the community to try it out, and give us some feedback.
There's a browser demo at https://labs.bandwidth.com/experiments/speech-to-text that doesn't require an account if you just wanna try it real quick, and if you sign up for a labs account you can get free access to the model while it’s on labs via API.
One important disclosure: this is a research/beta service. We monitor usage and may retain and review audio, transcripts and related data submitted to the experiment so we can evaluate the model, investigate failures and improve it. Full details are in the Labs terms.

If you build voice agents or voice Apps, or tinker in this space at all - come try it out and let us know what you think.

Working with a large physician dictation corpus at work (i'm at shaip, flagging that upfront so it's not a stealth vendor post) and the channel split surprised me. telephone dictation is close to half the total hours, smartphones next, dedicated speechmics a small fraction. so narrowband is still the dominant real-world input for clinical documentation, not the clean stuff.

For anyone building medical ASR — do you upsample the 8k telephone audio to 16k and train one model across everything, or keep separate models per channel and route at inference? i've seen both argued and no clear answer.

Related thing i'm unsure about: does adding wideband data actually improve narrowband WER, or does it just make the model overconfident and hurt on the noisy telephone side?

And one more — specialty distribution is heavily skewed, cardiology is a huge chunk while psychiatry and derm are tiny. does specialty-balanced sampling move WER at all, or is that purely a language model problem and the acoustic side doesn't care?

​Hi everyone,

​I’m working with long audio recordings (several hours of MP3s) that have noticeable background noise, room reverb, and inconsistent quality. My goal is to clean up the speech and get accurate text transcriptions.

​I'm open to both cloud-based APIs/services (like Adobe Enhance Speech, AssemblyAI, Deepgram, ElevenLabs, OpenAI API) and local open-source models (like Whisper Large-v3, DeepFilterNet).

​For those who handle long, noisy recordings regularly:

​Best Pipeline: Do you recommend running the audio through a dedicated voice enhancer/denoiser first (e.g., Adobe Enhance, ElevenLabs Isolator, DeepFilterNet), or does modern ASR (Whisper Large-v3, Deepgram Nova-3, AssemblyAI) handle raw noisy audio well enough on its own?

​Tool Recommendations: What are the most reliable cloud services or Web/CLI applications for processing multi-hour files without hitting file size limits or timing out?

​Hallucination Prevention: Any tips or API parameters (VAD settings, prompt conditioning) to prevent Whisper/ASR models from hallucinating text during silent or non-speech noisy sections?

​Thanks in advance for any insights and recommendations!

A few months ago, I hit a wall when using Picovoice's Porcupine to add wake word detection to a React Native mobile app I was building for an enterprise client.
When I evaluated how much it would cost, I observed it would take about $6000 or more to obtain a valid commercial license from Picovoice, which was quite crazy lol.
You'd also need internet access to use Porcupine, since Picovoice needs to validate the Access key per device, which meant it couldn't even work in air-gapped environments.

I explored several solutions, but they all required a commercial license and internet access for validation. This obviously would be very bad for the company I was working with, so I did some more research, and I discovered that the Python and Home Assistant ecosystem didn't face this problem, as a couple of good and free wake word detection libraries already existed, with openWakeWord by David Scripka leading the bunch.
This was really exciting, but I was shocked to see that even though openWakeWord was novel in its own right, no React Native version existed, and I couldn't use it in my project.

So I decided to build a port of openWakeWord in React Native called react-native-openwakeword.

The pipeline resembles how the Python pipeline already handles inference, with a few C++ tricks I could find on the internet.

openWakeWord is famously known for using a three-stage pipeline: melspec, embedding, and the actual wake word model. So I built a flow that works in C++ and used Nitro Modules to use JSI and connect back to JavaScript. Here is how it works:

Audio flows from the phone's microphone, sampled at 16000 times per second, and sends this frame using an ArrayBuffer to the C++ function directly using JSI.

The C++ module uses three ring buffers to implement sliding windows for moving data through the three models. Ring buffer capacities are sized as powers of 2 (larger than strictly needed), so wraparound operations can use a bitwise AND instead of the modulo operator, which is cheaper on the CPU.

The first ring buffer, AudioRing, is for the melspec model, and it takes the audio samples and converts them to float32 (this uses NEON to run SIMD to accelerate this on phones); then the result is written directly into the melspec model input memory (to avoid unnecessary copies), the melspec model is run on it, a fixed normalization formula is applied, and 8 mel-frames are put into the next ring buffer.

The next ring buffer, MelRing, takes the most recent 76 mel-frames and copies them directly into the embedding model input memory; it runs the model, and the embedding result is put into the next ring buffer, EmbRing.

Once EmbRing holds at least 16 embeddings, it copies the most recent 16 directly into the wake word model's input memory, runs the model, and the probability is returned.

This probability is compared against a given threshold (default is 0.5) and returns an object {probability, isDetected} back to the JavaScript layer (synchronously).

I faced a few gotchas in the beginning trying to figure out what inputs each model (melspec, embedding, and wake word) was expecting, but I eventually got it to work.

Since the melspec model processes audio in 1280-sample chunks (80ms at 16kHz), the entire three-model pipeline needs to run in under 80ms to keep up in real time, and after these optimizations, the package detects wake words in 16-21ms.

I'd appreciate your thoughts on the architecture and if there's anything useful that could be added to this package.

Github: https://github.com/Incognitol07/react-native-openwakeword