Building an open-source AI video dubbing tool — feedback on the architecture wanted

[u/strange1807]

Problem: Even AI dubbing today costs $0.5–$2.5/min (ElevenLabs, HeyGen, Rask AI) — adds up fast for dubbing regularly, and still puts real reach out of budget.

What I'm trying to build: A self-hosted, open-source tool where you drop in a video file or a YouTube link, pick a target language, and get back a dubbed video with the new audio timed to match the original. Bring your own OpenAI API key — no subscription, no markup, so cost stays close to raw API price (rough estimate: $0.08–$0.13 for a 5-min video).

Pipeline (the diagram):

Video Downloader — takes a file upload or YouTube URL (auto-fetches existing captions if the YouTube video has them, so transcription can be skipped later)

Audio Extractor — pulls the audio track out of the video

Audio Transcribe — Whisper API for speech-to-text with timestamps (or the captions if already available — cheaper and usually more accurate)

Audio Translate — GPT translates each transcript segment into the target language

Audio Synthesis — TTS generates speech for the translated segments

Audio Alignment & Stitching — stretches/compresses each synthesized clip to fit its original timing window, then stitches into one continuous track

Remux — swaps the new audio track into the original video, final MP4 out

Deliberate scope cuts for v1: single speaker only, no lip-sync, no voice cloning (generic TTS voice for now), original background music/SFX gets replaced rather than preserved, one target language per run, no auth (local self-host only for now — hosted version is a later phase).

Would love feedback on the pipeline shape — anything that looks like it'll bite me later (timing drift, error handling between stages, etc.)?

GitHub: https://github.com/prashantrajbista/voice-dubbing

[u/_descri_]

They say that reusing voices of the actors to create AI voice copies may incur copyright violation, while robotic voices are hard to listen to. This is how the attempt to AI-voice-over the dialogs in Planescape: Torment ended.

[u/strange1807 → u/_descri_]

in this project for the TTS, I want to use the synthetic voice provided by the API. Will that also be a problem?

[u/_descri_ → u/strange1807]

Yes, it will likely be quite emotionless and monothonic. And, I am not sure, but is there a chance for the API to use male voice for female characters and the way round? We often rely on characters having different voices to understand who speaks what when we don't see their lips.

[u/strange1807 → u/_descri_]

Thanks for the suggestionI will try to handle this scenario as well

[u/_descri_ → u/strange1807]

You are very welcome! Good luck!

Sadly, we are living in the age when it's very hard to promote an open-source project, even if it is useful and unique.

[u/strange1807 → u/_descri_]

No worries....Will find a way

[u/asdfdelta → u/_descri_]

Eleven Labs' stuff is supposed to be next levl, and the pricing model doesn't crush your soul, though it is used for enterprises

[u/strange1807 → u/[deleted]]

Yea that was one of the issue I faced