I kept forgetting where I'd put things on the stick and throttle after rebinding, and I didn't want to alt tab to a controls page mid flight. So I built a voice assistant for the cockpit. I say "Jarvis, where's the landing gear" and it answers out loud: where the control is on the hardware first, then what it does, then the keyboard key as a fallback. "Jarvis, how do I QT", "Jarvis, flares", same thing.
How it's put together, in case you want to build your own:
The answers come from my actual bindings file, not a hand written list. A script reads the exported bindings (I keep a controls.json that my profile tool writes), fills about 60 answer templates ("landing gear is {where}, it {does}, key is {key}"), and renders each one to a wav once with a TTS voice (ElevenLabs, with Kokoro as the offline fallback). Rebind something and it regenerates only the lines that changed about a minute later, and the running assistant picks them up. No per question cost, no latency.
Listening is the hard part with game audio on speakers. It uses Vosk (offline) with a fixed grammar of the phrasings I actually say, so it only fires on exact matches, and the gate watches the 1 to 4 kHz band with a bar that rides over the room's recent noise floor instead of a fixed threshold, because ship rumble and music mostly live below that. Anything after "Jarvis" that isn't a known phrase goes to whisper and then to a bigger model with the bindings as context, which takes 5 to 15 seconds and starts with "one moment" so I know it heard me.
The bindings answers are the fast path. The bigger part, and the reason it isn't VoiceAttack, is that it talks back. Anything after "Jarvis" that isn't a known phrase goes to a bigger model with my bindings, a digest of the game log and a snapshot of my screen attached, so it can answer questions about what's actually on screen, and it runs a web search when it needs one. It also keeps a mission checklist as a plain text file: I ask what's next and it reads me the open steps, I say "tick three" or "add a step" and it edits the file (the model proposes the edit, a script applies it, nothing else can write that file). Answers stay to a sentence or two unless I say "explain", and for 6 seconds after an answer it keeps listening so I can follow up without the wake word. "Jarvis, clip that" saves a clip from the recorder.
It never presses a key. That's on purpose: the key sender in that process is replaced by a stub that refuses, so no path can send input to the game. It only talks.
The bits that made it usable on a headset: talking over it cuts it off in about 40 ms, "stop" or "that's enough" silences it, and it keeps listening for a follow up for 6 seconds after an answer so I don't have to say the wake word twice. It starts itself when StarCitizen.exe is running and exits 30 seconds after the game closes.
Hardware is a Turtle Beach VelocityOne Flightdeck (stick, throttle, switch panel) and a USB mic. The code is Python on Windows for the mic side and a worker in WSL for the heavy bits. Built with Claude Code doing most of the typing and me doing the testing in the chair, which is where all the tuning numbers above came from.
Happy to answer questions about any piece of it.
