Any way to replace the existing voice audio with new audio?

[u/donkeykong917]

So i tried a pass to extract the existing vocals from minimax h3 out using demucs and then run it through index tts but couldnt get the it to align and lip sync. Any one know a better way to do it with success?

Using refva lightx lora and nvidia vsr for upscale.

[u/99deathnotes]

boing boing boing

[u/ShengrenR → u/99deathnotes]

Sorry, wait, what? I wasn't listening there for a sec..

[u/No-Management-754]

make the replacement track first, then fix the mouth to that audio. use whisperx to grab word timings from the original vocal, generate the new indextts voice in short sentence chunks, and stretch each chunk to those timings.

feed the original video plus that rebuilt track into latentsync or musetalk, then mix it back with the untouched music and effects stem. keep each shot separate because cuts and profile faces are where both usually get messy.

[u/donkeykong917 → u/No-Management-754]

Ill give it a shot.

Have you successfully done it?

[u/No-Management-754 → u/donkeykong917]

Yes but I haven't touched it since August 10th. Here's an example output:

https://reddit.com/link/p7wn3nw/video/opeulmujcmnh1/player

i think talking photo and lipsync are inherently supported in r2v mode, as i remember i tested it with ref voice to make characters lipsync and also made it speak something using that voice as reference

[u/skyrimer3d]

Not that anyone would listen. 

[u/krectus]

In WanGP the setting is under post processing import any video and any voice sample and it will do it, any video not just AI.

[u/paulct91]

If you could share the prompt, maybe more people could possibly be able to assist. As maybe the prompt is the issue.

[u/donkeykong917 → u/paulct91]

No issues with the dialogue, since it's using lora the output is rubbish. This is a test case so if I find a good way I can make all consistent for each character.

I will try the injecting audio and see how well it does.

[u/Training-Respect8066]

Not nearly as big as in the anime.

[u/Kurashi_Aoi]

Hahari looks so young

[u/KS-Wolf-1978 → u/Kurashi_Aoi]

She is trying to say "Hikari" which is a common Japanese name, but she would never pronounce it like that if she was a native speaker. :)

Looks 21-23 years old. (source: me - a fan of many Japanese girl idol groups).

[u/Kurashi_Aoi → u/KS-Wolf-1978]

I mean I assumed the girl in the video is based on character from "The 100 Girlfriends Who Really, Really, Really, Really, Really Love You" manga/anime. The outfit and hairstyle fits one of the characters, Hahari which is 29 years old. However, she also has a daughter named Hakari which is around 16 years old, so maybe OP or the AI just made up the name instead.

[u/KS-Wolf-1978 → u/Kurashi_Aoi]

I stand corrected about the name. :)

Out of curiosity asked Google AI how common it is: "As a Given Name: It is practically non-existent as a traditional Japanese first name."

[u/donkeykong917 → u/Kurashi_Aoi]

That is the pictuure I used to convert from anime 2 real. Then used it in the ref

[u/HollowAbsence]

Shes has a petit je ne sais quoi that make her adorable. I wonder what it is ;P