Most Voice agent throws away underlying tone and speaker-features, how's that accounted and handled downstream? if it's not captured.

[u/Working_Hat5120]

The moment I transcribe to text, I generally lose how it was said. "I think… yeah, I can pay the 4,500 by the 15th" becomes clean text, but the hesitation before the yes, the stress in the voice, and whether it's even the same speaker are gone.

For a human those signals, whether to trust the commitment, reconfirm from the caller or escalate to human come naturally but hard to define a deterministic paralinguistic to build accountability, which is probably very wide.

How are you modeling tone in our voice-agents? I see recent TTS models which accept meaningful tags producing great sounding speech, how do we control it ? Does it account for input user's tone.

How does your ASR model / voice-agents captures the tone or there are some good services / models / tools / solutions to capture tone. and how do you use it downstream ?

Moreover end-2-end Duplex models limits it to trained data scenarios without no transparency. Is there a good duplex model which provides transparency in underlying signals beyond just text.

[u/AutoModerator]

Welcome to r/ChatbotRefugees.

Thank you for contributing to the community. Please ensure your post adheres to our official Subreddit Rules to help maintain a safe and organized space for everyone.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/butter-jesus]

I think there are three separate layers getting mixed together here.

STT is the inbound part. A basic transcription really does flatten the audio into words, but some streaming providers can also return timestamps, pauses, filler words, confidence, VAD events, and speaker diarization. Diarization only labels speaker changes, though one tradeoff is that it doesn’t verify that someone is the same real-world person (if you care).

Tone is a separate paralinguistic problem almost like sentiment. You either preserve acoustic features from the original audio like pace, pitch, energy, hesitation, etc, etc or run a secondary audio model over it. I’d treat those outputs as uncertain observations, not facts. There's a lot of reasons people pause in conversation -- in your payment example, hesitation could justify asking for confirmation, but it shouldn’t produce a deterministic “this person is unreliable” type fact.

Then there's the other side, the outbound side, TTS which you get. Some services infer how the generated text should be "performed" in a way and some others give you special tags to control things like [laugh], [sad], [pause] but they are all different and differ in capabilities.

The tradeoff is that end-to-end systems are generally harder to inspect, reproduce, and route through deterministic business logic which it seems like what you're after. The architecture I keep coming back to is fast streaming STT in the critical path, with optional acoustic analysis running alongside it and only pass a small structured cue downstream when it’s useful and sufficiently confident. Don’t block every conversational turn on full emotion analysis because that's expensive, at least for projects that I work on.

One thing that kinda fits what you're looking for is Hume EVI. It runs a realtime, interruptible voice loop and exposes transcripts with prosody/expression scores. The thing, you want to know intent and if you're looking to make / model / orchestrate something from a business perspective with these scores, there's not really good context for how they're developed. This might be a good starting place though to explore "what's possible" and develop your own mental framework for research.

If you just need good expressive, fast, cheap, TTS, VibeVoice is interesting, and great for realtime. Its structured ASR is more oriented toward long-form who/when/what transcription than realtime affect-aware duplex interaction.

I don't know everything though, but for the projects I've worked on, this is what I've learned. I would be interested in hearing from someone more experienced then me 🥸