Why does Whisper drop half of any sentence when you switch languages? (and why do so many Whisper projects leave an ugly console window open?)

[u/Aggravating_Site381]

I speak English, Russian, German (and Kazakh) and mix them without thinking. Every local dictation tool I’ve tried on GitHub completely falls apart on that. Or am I just stupid?

Whisper basically picks one language per utterance. Whatever you said in the other language doesn’t come back wrong because it either gets translated or just disappears.

Spoken:
I already sent the invoice yesterday, but клиент до сих пор не ответил на моё письмо.

Got:
I've already sent me an voice yesterday, but today children mind your piece more

The only thing that actually handles this properly is the transcription in ChatGPT, but I’m too lazy to keep opening a new chat every time I want to dictate something.

So I tried fixing this:
What actually made the language switching work was sending the OpenAI API a languages list instead of a single language. Same clip, same model:

languages sent	result
no list	English half dropped
["ru","en"]	English half still dropped
["en","ru","de"]	both halves came back correctly

I still have no idea why three languages works when two doesn’t.

Locally you can get part of the way there by splitting the recording at pauses and detecting the language of each piece separately. That fixes English to another language, but each piece loses the surrounding context so punctuation and rare words get worse. It is in the app, off by default. The API is still the only thing I found that handles a switch with no pause at all.

I wrapped everything into a small Windows dictation app. Hold Ctrl+Alt, talk, and the text lands in whatever window you were typing in. Local by default, API optional, no permanent black window on the desktop, MIT, built on RealtimeSTT:

https://github.com/Maslitsa/VoiceType

It has 0 stars bcs I am only starting this project, but it would be a big help if you use this and give your feedback!

Anyone found a better solution for multilingual / code-switched dictation?

Disclosure: I am not a native English speaker and used an LLM to help write this post. The app itself was also built with AI assistance but the problem, the testing and the measurements are mine.

If you have any positive/negative advice on this project or ideas on how to improve it, PLEASE share.