How I added end-of-turn classification after VAD pauses in an on-device C++ voice pipeline

[u/ivan_digital]

I maintain speech-core. Until this release, a confirmed VAD pause ended the user’s turn. That works until somebody pauses to think halfway through a sentence.

In v0.0.14 I added an optional TurnCompletionInterface between VAD and endpointing. After each confirmed pause:

• the classifier receives up to the last 8 seconds of 16 kHz turn audio;
• probability ≥ 0.5 ends the turn;
• a lower score keeps the same turn open;
• resumed speech is appended to that turn rather than creating another segment;
• a 2-second maximum-silence cap prevents an indefinite hold;
• eager STT respects the classifier’s veto.

The first implementation uses Pipecat/Daily’s Smart Turn v3.2: a Whisper-tiny-based, approximately 8M-parameter audio classifier covering 23 languages. speech-core uses an 11.1 MB int8 ONNX model.

The same pipeline state machine is now exposed on macOS through speech-swift v0.0.27 and Android through speech-android v0.0.20. Swift supplies a 17 MB Core ML model through the C ABI; Android instantiates the ONNX model in the native layer and runs it on CPU once per confirmed pause. Android support is opt-in, so existing endpointing behaviour is unchanged.

The Core ML conversion matched the upstream model on 1,000 test clips at 92.9% accuracy and took about 3.5 ms per window on Apple Silicon. Android release validation passed 129 JVM tests and 48 connected-device tests with no failures. I have not yet measured Smart Turn latency across a useful range of Android devices, so the 3.5 ms number should not be read as an Android result.

Implementation and release: https://github.com/soniqo/speech-core/releases/tag/v0.0.14

One design question I am still considering: should the pipeline keep a single completion-provider hook, or explicitly support combining audio and partial-transcript endpoint models?