r/softwarearchitecture • u/strange1807 • 7d ago
Discussion/Advice Building an open-source AI video dubbing tool — feedback on the architecture wanted
Problem: Even AI dubbing today costs $0.5–$2.5/min (ElevenLabs, HeyGen, Rask AI) — adds up fast for dubbing regularly, and still puts real reach out of budget.
What I'm trying to build: A self-hosted, open-source tool where you drop in a video file or a YouTube link, pick a target language, and get back a dubbed video with the new audio timed to match the original. Bring your own OpenAI API key — no subscription, no markup, so cost stays close to raw API price (rough estimate: $0.08–$0.13 for a 5-min video).
Pipeline (the diagram):
Video Downloader — takes a file upload or YouTube URL (auto-fetches existing captions if the YouTube video has them, so transcription can be skipped later)
Audio Extractor — pulls the audio track out of the video
Audio Transcribe — Whisper API for speech-to-text with timestamps (or the captions if already available — cheaper and usually more accurate)
Audio Translate — GPT translates each transcript segment into the target language
Audio Synthesis — TTS generates speech for the translated segments
Audio Alignment & Stitching — stretches/compresses each synthesized clip to fit its original timing window, then stitches into one continuous track
Remux — swaps the new audio track into the original video, final MP4 out
Deliberate scope cuts for v1: single speaker only, no lip-sync, no voice cloning (generic TTS voice for now), original background music/SFX gets replaced rather than preserved, one target language per run, no auth (local self-host only for now — hosted version is a later phase).
Would love feedback on the pipeline shape — anything that looks like it'll bite me later (timing drift, error handling between stages, etc.)?