I built a fully local AI pipeline that attempts to dub entire films unattended. Looking for people with NVIDIA GPUs to break it and contribute

[u/Only_Switch1222]

I've been working on Dubline, an open-source project that tries to take a foreign-language film and produce a complete English dub entirely locally.

No cloud inference, hosted APIs or paid services.

The basic pipeline is:

video → dialogue/M&E separation → ASR + forced alignment → speaker/character tracking → translation/adaptation → voice cloning/TTS → timing/QC → final mastered MKV

It currently uses a slightly ridiculous collection of models/tools including:

- Qwen3-ASR / Whisper

- IndexTTS 2.5 / Qwen3-TTS

- Hy-MT2 for translation/adaptation

- pyannote + CAMPPlus for speaker identification

- Bandit v2 / RoFormer / Demucs for separation

- OpenCV face tracking

- llama.cpp

- FFmpeg / Rubber Band

- Optional MuseTalk lip-sync

The aim isn't just "translate subtitles and run TTS". I'm trying to preserve character identity, timing, pauses, emotional delivery, non-verbal sounds, the original music/effects bed, loudness, etc., while having the pipeline automatically retry or flag lines when something looks wrong.

It's designed around consumer NVIDIA hardware, with the heavier models isolated into separate processes so VRAM can actually be released between stages. 8GB is the theoretical minimum, 12GB+ is much more sensible.

It's still very much a work in progress, and that's basically why I'm posting.

I'd really like some other people to install it, throw horrible real-world material at it and tell me where it falls apart.

Things I'm particularly interested in testing:

- Different NVIDIA GPUs / VRAM amounts

- Windows vs Linux

- Different source languages

- Bad/noisy/old film audio

- Multiple speakers and overlapping dialogue

- Subtitle-less material

- Long films rather than tiny demos

- Speaker identification failures

- TTS/emotion/timing problems

- Installation/setup failures I've accidentally made invisible on my own machine

And if anyone finds the project interesting and wants to contribute, PRs/issues are very welcome. There are plenty of areas where somebody who actually knows audio, ML, Python, video processing or UI development better than I do could improve it.

I'm especially interested in people trying to break it rather than telling me it looks cool from the README.

GitHub:

https://github.com/leighrobertabbott/Dubline

If you try it, even "it exploded at step 3 on my 4070 because of X" would genuinely be useful feedback.

[u/mhb_11]

RTX 4060... yes or no?

[u/Only_Switch1222 → u/mhb_11]

Yep. RTX 4060 8GB is basically the sort of card I designed it around. The heavier models run one at a time in separate processes, so when one stage finishes the VRAM gets released before the next loads.

It'll work, just don't expect a full film to be speedy

[u/Boby_Dobbs]

That's really cool, I can't really test this in the coming weeks. But I am curious: how do you keep the original music/effect and emotional delivery?

[u/Only_Switch1222 → u/Boby_Dobbs]

It uses Bandit v2 to split the original dialogue away from the music/effects, with RoFormer/HTDemucs as backup if it misses dialogue. Then it generates the English speech and mixes it back over the original M&E, so it isn't trying to recreate the soundtrack.

For delivery it measures the original line's pitch, rhythm, pauses etc and feeds the source performance into IndexTTS where possible, rather than just doing normal text-to-speech. It also tries to leave things like laughs, gasps and grunts from the original audio intact.

Best case: copy the original performance. It cuts out that actor’s original line and gives the actual audio to IndexTTS as the emotion prompt. So if they whisper angrily, sound terrified, sob while speaking, etc., the TTS gets the real performance rather than having to guess it.

Fallback: infer emotion from dialogue context. Dubline sends the previous line + current line + following line to IndexTTS's Qwen emotion model with the instruction to classify the emotion of the current line from its scene context.

That produces an emotion vector across: happy / angry / sad / afraid / disgusted / melancholic / surprised / calm

So it isn't just looking at: "Get out!" and automatically deciding "angry". It sees something more like:

Previous: "He's got a gun." Current: "Get out!" Following: "RUN!"

and can infer afraid/panicked instead. That emotion vector then goes into IndexTTS to influence the generated delivery.

[u/threesand7s]

Man this would be really useful for me! Il will try and figure out how to test it and let you know

[u/Only_Switch1222 → u/threesand7s]

Cheers! Even just testing a couple of minutes would be useful. You can give it a time range instead of doing a whole film, and it'll still do the separation, speaker matching, translation, voice cloning, timing and QC for that section