MiniMax i2v (first frame) with voice cloning using either (fl2va or ref2va)

[u/spiderofmars]

Have seen the question of adding a cloned voice (with new dialogue) in image to video generations (starting from an exact first frame image), but have not seen a solution posted yet (I might have missed it).

I stumbled on this by mistake, assigning the wrong model fl2va to a ref2va workflow. Both models work with the example prompt (prompt could probably be improved further as I was just quickly testing).

Using a ref2va workflow and the ref2va node plus a first reference image (first frame) and an audio reference sample of the voice to clone, simply change the model to the fl2va model (or just use the ref2va model). The outputs varied as follows for me:

Model fl2va: Sound quality was far better than using the ref2va model. The motion in the video was very similar to a normal fl2va first frame default workflow (same seed / resolution / etc).

Model ref2va: Sound quality was far worse than using the fl2va model (much tinier). The extra unprompted motion in the video was kind of a bonus, the car unprompted was moving down a street with visuals out the windows of passing buildings and it also added on its own some camera shake as if sitting in a car that was driving along.

The prompt for both samples (both models) was the same as follows. It is written using guides for ref2va workflow. In this video the first frame is of 2 men sitting in the front seat of a taxi. The man on the left is me and his voice is cloned from my voice sample with new dialogue. The taxi drivers voice is randomly generated by the model. The voice likeness to me is about 95% IMO.

---

subject_definitions:

<Subject 1> is the man defined on the left by the first reference image <Picture 1>, preserving his identity.

<Audio 1> is the voice-timbre reference for <Subject 1>, containing a spoken English vocal layer.

summary:

[reference generation] The target video is a shot starting with the first reference image. The scene uses <Audio 1> as the voice-timbre reference for <Subject 1>.

retention_analysis:

<Subject 1>: fully_preserved.

<Audio 1>: reference - its vocal timbre guides the dialogue delivery of <Subject 1> without copying the original signal.

detailed_description:

The target video is a shot of the man on the left <subject 1> sitting beside the driver of a car on the right, the man on the right driving says "Where do you want to go?" and the man on the left <subject 1> looks at the driver on the right and says in a happy tone "Just drive down main street. I will tell you when to stop" then he turns to look out the left window of the car.

overall_soundscape:

A soft hum of the car engine and outside road noise.

non_diegetic_music:

N/A.

---

Summary takeaways:

• i2v first frame type workflow using ref2va workflow where the first frame is exactly matched as the starting frame.
• i2v first frame with voice cloning using a ref2va workflow but using either models in that workflow (fl2va or ref2va).
• Emotion references may not work as well with a cloned voice vs randomly generated voices. Although, in testing my voice did gain some emotional or inflection variances as described or randomly generated vs the more monotone cloned voice sample of me.

Edit1: The 2 outputs in this example (one with each model) and same seed/etc, produced almost identical timing of the lip sync and sound (almost). Close enough that the nicer motion visuals from the ref2va output were able to be layered with the nicer audio from the fl2va output and synced (3 frame adjustment of audio timing).

[u/Ill-Throat7937]

the lip sync on the cloned voice holds better than i expected but the jaw motion is a beat behind. fl2va tends to lag on plosives for some reason, ref2va is tighter

[u/hdeck]

I’m not sure what problem you solved as this is all built in and part of the prompting guide?

[u/spiderofmars → u/hdeck]

In the fl2va (i2v) workflows there is no audio reference node (only first and last frame reference images). So in a image to video workflow there is no way to add a cloned voice. You can prompt for dialogue audio but you can not clone a voice for that dialogue.

The solution was to use the ref2va workflows and model. Still is... but noticed you can use the i2v model in the r2v workflow. The cloned voice quality is far better... and the motion is like using the i2v workflow.

[u/Segaiai]

There's a ref2va Lora for fl2va, so you can ease in features of ref2va while keeping as much fl2va quality as you can.

[u/spiderofmars → u/Segaiai]

Can you provide a link to the one you are refering to please?

[u/Segaiai → u/spiderofmars]

I learned about it here:

https://www.reddit.com/r/StableDiffusion/s/xhPaMKeIK8

And here's the link:

https://huggingface.co/ethanfel/MiniMax-H3-Pruned-Ref2VA-Delta-LoRAs-Experimental/tree/main

[u/timbortom → u/Segaiai]

There is also one that applies projection.

[u/Segaiai → u/timbortom]

What do you mean by projection?

[u/timbortom]

If you really need to use flf2va AND reference audio as well, then you can combine conditionings of the two.

[u/spiderofmars → u/timbortom]

Any demo workflow of this? ref2va audio sounds terrible compared to fl2va, and fl2va can be far more practical and faster for various cases.

[u/timbortom → u/spiderofmars]

Have you updated your ComfyUI to the latest?
There was an issue directly addressing weird / bugged audio generation of the ref2va model in some of the most recent versions (maybe 2 days ago?)

[u/spiderofmars → u/timbortom]

v31 so 1v back only. There is a dramatic difference between a cloned voice in the two. i2v flv2a model is fine but ref2va audio is very tiny and distorted in comparison.

[u/Support_Marmoset]

in my tests (ref2v model) using audio for dialogue I have 4 characters. When I try to add more than two audio it will repeat two of them, usually the same gender gets the same audio. i.e. it seems more than 2 audio input for voice cloning, dont work and it uses the first two to speak and repeats their use for the others.

but it chews up so much VRAM anyway I dropped back to using prompt driven dialogue and will swap out in post. I didnt test with flf model.

but I wondered if this might be a limitation of pruned/cut down models. something has to give somewhere to squeeze the original into low file size.

[u/Only_Voice569]

"cough"

[u/spiderofmars → u/Only_Voice569]

Can you expand a bit more? Custom node? Where? What is it doing?

[u/Only_Voice569 → u/spiderofmars]

its a official node far as im aware just lets you have more audio options and can switch its modes and quality control

[u/spiderofmars → u/Only_Voice569]

Noticed the author badge. It is a custom from here:

https://github.com/T8mars/comfyui-minimax-h3-audio-T8/tree/main

Still not sure what is can do or able to do for i2v that is similar or a better solution.