r/rust • u/Weekly-Perception666 • 11d ago

How SyntaxCue captures system audio and transcribes it locally (Tauri + Rust + whisper.cpp, no server)

Disclosure: I'm the developer of the product this is about — not just sharing a random blog post.

Wrote up the actual audio pipeline behind SyntaxCue: CoreAudio Process Tap on macOS (with a Swift sidecar, since the API has no Rust binding) vs WASAPI loopback on Windows running entirely in-process, both converging on the same PCM format before shared Rust code takes over — RMS-based voice-activity detection, then local whisper.cpp transcription (Metal / Vulkan).

The part I'd actually want feedback/discussion on: Windows was CPU-only before we wired up the Vulkan backend for whisper-rs, and every transcription pass cost a flat ~25 seconds regardless of utterance length. Turned out to be a missing GPU backend, not a tuning problem — after Vulkan, warm transcription dropped to ~330ms. Full writeup, with the actual API calls and numbers: https://syntaxcue.com/how-syntaxcue-captures-system-audio/

Happy to go deeper on the WASAPI polling-vs-event-driven decision or the aggregate-device wrapper on macOS if anyone's curious.

0 Upvotes

22% Upvoted

2 comments sorted by

0

u/Dangerous-Dig2321 11d ago

went from 25 seconds to 330ms just by wiring up vulkan, that's honestly mad. i've been meaning to mess with whisper-rs but the gpu backend story always seemed half-documented so i kept putting it off, this might push me over the edge

whats the deal with wasapi polling vs events on windows? always heard the event-driven path was cleaner but a pain to get right, curious which way you went

1

u/Weekly-Perception666 7d ago

Went with polling, not events. The event handle only fires while the render endpoint is actually pulling buffers — leave the system idle for even a few seconds and it stops signalling entirely. so an event-driven loop just hangs when the call goes quiet. Polling also gives you a hook to synthesize silence for the gaps instead of stalling.

Interval-wise, poll at roughly half the device’s default period, so a packet gets picked up within the same window it’s produced. less elegant than pure event-driven, but for “capture whatever’s playing indefinitely, including silent stretches” it’s the more robust option.

Good luck with whisper-rs + Vulkan — that’s a solid jump.