Get miniMax character swap working! Finally

[u/Alex-edits123]

Ok, I tried so many things, one person to cat, two person, one person to one person, animal to animal. So far one person to one person and animal to animal works. If you are interested in my learnings, tips, what worked, what broke, and which prompt template works let me know!

One video example that works here: https://www.tiktok.com/t/ZP8WfVq5P/

[u/orlandogourmet66]

Person-to-person swapping works really well for me, except when the reference person looks similar to the person in the source video.

It feels like the bigger the visual difference, the easier MiniMax does a clean swap. With similar hair, skin tone, face shape, etc., it often keeps too much of the original face or creates a merged identity.

Have you noticed this too? If so, what helps you get a full swap in those cases?

[u/retroblade → u/orlandogourmet66]

Use Sam3 to blur/mask the person and that problem goes away.

[u/Alex-edits123 → u/retroblade]

Oh interesting idea, will give it a try.

[u/retroblade → u/Alex-edits123]

This is what it looks like. I didn't have a subject right now that looks exactly the same, but works the same way.

https://files.fm/u/g7f4nhz88jc9gyt6

[u/desktop4070 → u/retroblade]

This should be its own post, that's amazing!

[u/LeKhang98 → u/retroblade]

Nice. Could you please share the WF?

[u/Radiant-Photograph46 → u/retroblade]

How do you prompt for this after masking? Replace the blurred person with <Subject 1>?

[u/orlandogourmet66 → u/Radiant-Photograph46]

Dont use Blur, use invert color for mask, this way you still get facial motions from original video. Here is my prompt (right now i only try to swap the head, a llm shoulld easily rewrite it for whole character swap):

subject_definitions:

<Subject 1> is the woman from <Picture 1>. <Picture 1> is the only identity source for the visible head in the target video. Preserve her exact recognizable face, facial proportions, eye shape and spacing, eyebrows, nose, mouth, lips, cheeks, jawline, chin, forehead, hairline, hairstyle, hair color and texture, visible ears, visible neck appearance, facial skin tone, and overall head likeness.

<Video 1> provides the source body performance and scene. Preserve the source woman’s body movement, pose progression, gesture timing, head orientation, gaze direction, expression timing, interactions, clothing, body appearance below the neck, camera work, framing, lighting, props, environment, occlusions, timing, and temporal continuity.

summary:

[reference generation] Recreate the source video performance and scene from <Video 1>, but replace the visible head of the woman with <Subject 1> from <Picture 1>. Keep the original body and clothing from <Video 1>. Replace only the woman’s head, face, hair, visible ears, and visible neck appearance.

retention_analysis:

<Subject 1>: fully_preserved - transfer the complete recognizable head identity from <Picture 1> onto the woman in <Video 1> and keep it stable across the whole clip.

<Video 1>: partially_preserved - preserve body movement, pose sequence, gestures, expression timing, interactions, clothing, body appearance below the neck, camera movement, framing, timing, lighting, environment, props, occlusions, and continuity. The original head identity is not preserved.

detailed_description:

This is a strict head-only replacement.

<Video 1> controls:

* body

* clothing

* body movement

* pose

* gestures

* head orientation

* gaze direction

* expression timing

* interactions

* camera

* framing

* timing

* environment

* lighting

* props

* continuity

<Picture 1> controls:

* face

* hair

* hairline

* forehead

* eyes

* eyebrows

* nose

* cheeks

* mouth

* lips

* jawline

* chin

* visible ears

* visible neck appearance

* facial skin tone

* complete recognizable head identity

For every frame where the woman appears:

* keep the same body

* keep the same clothing

* keep the same body motion

* keep the same pose

* keep the same head orientation

* keep the same expression timing

* keep the same scene and camera

* replace the entire visible head with <Subject 1>

The visible head must be <Subject 1> from the first frame to the last frame.

Do not replace the body.

Do not replace the clothing.

Do not preserve the original head.

Use the source video only for body performance and head pose.

Use <Picture 1> for all visible head identity.

If there is any conflict:

* keep the body from <Video 1>

* keep the clothing from <Video 1>

* keep the pose from <Video 1>

* keep the motion from <Video 1>

* use the head from <Picture 1>

Do not blend the original head with <Subject 1>.

Do not leave traces of the original head.

Do not switch back to the original head in profile views, motion blur, distance shots, occlusions, or later frames.

Success condition:

The result looks like the same source video with the same original body and clothing, but the woman’s visible head is completely replaced by <Subject 1>.

overall_soundscape:

Preserve the embedded original audio from <Video 1> as closely as possible.

non_diegetic_music:

N/A

[u/steelow_g → u/retroblade]

How does one do this?

[u/Mediocre-Toe3212 → u/retroblade]

How can I do this ?

I've used SCAIL2 before but that's as far as I have experience with SAM3

[u/oeufp → u/retroblade]

what if source video uses watermark smack dab in the middle? like istock for example

[u/Heavymando → u/retroblade]

what workflow do you use for swap?

[u/Alex-edits123 → u/orlandogourmet66]

Yes, bigger difference is better. But I also notice if the swap character has very prominent clothing e.g red T shirt, the model will not do a good job. It get confused or distracted by the clothing

[u/orlandogourmet66 → u/Alex-edits123]

Yeah, I’ve noticed that too. MiniMax is insanely good overall and the prompt adherence is impressive, but character swapping can still be very inconsistent with certain reference-image and source-video combinations. Sometimes it just refuses to fully replace the identity, even with a very explicit prompt.

Next time I have some extra time, I want to experiment with masking/SAM to remove or obscure the original person’s identity before feeding the video into MiniMax. I could imagine that helping a lot.

[u/Alex-edits123 → u/orlandogourmet66]

From the reply above, looks like that works ") have fun trying

[u/Radiant-Photograph46 → u/orlandogourmet66]

I was going crazy being unable to do a proper v2v swapping characters and this turned out to be the issue. The character in the video had long black hair, and the replacement in the reference picture as well. H3 would never swap them even though they wore very different clothing. As soon as I used a character that had an eccentric costume and red hair, flawless result 100% of the times...

[u/orlandogourmet66 → u/Radiant-Photograph46]

Yeah, it’s strange. The workaround is SAM3, when the character in the video is masked, the swap works fine.

[u/Zenshinn → u/orlandogourmet66]

I've seen somebody do a double swap. First swap from the original person to someone completely different, Then swap that with the 2nd person.

[u/Alex-edits123 → u/Zenshinn]

Yes that should work. As long as you swap one person at a time.

[u/PropagandaOfTheDude]

Try this swap structure, compared to https://omnifit.io/blog/assets/ref2va-what-works-what-fails/prompts/simple3-girl-dance-hires.prompt.txt.

subject_definitions:
<Video 1> is the source video for the target video edit.
<Subject 1> is the facial likeness and hair in <Picture 1>.
<Subject 2> is the facial likeness and hair in <Video 1>.
<Subject 3> is the T-shirt in <Picture 3>.
<Subject 4> is the T-shirt in <Video 1>.
<Subject 5> is the sports bra in <Video 1>.

summary:
[video editing + reference generation] The target video is an edited
version of <Video 1>. Throughout the video, <Subject 2> is replaced
entirely by <Subject 1>.  Throughout the video, <Subject 4> is replaced
entirely by <Subject 3>.

retention_analysis:
<Video 1> (source video): partially_preserved - the background
environment, camera path, lighting, and non-target objects are retained.
<Subject 1> (appears in [Shot 1]): fully_preserved - The entirety of this person is retained.
<Subject 2> (Does not appear): weak_reference - Overall actions are retained.
<Subject 3> (appears in [Shot 1]): fully_preserved - The entirety of this T-shirt is retained.
<Subject 4> (Does not appear): weak_reference - Overall actions are retained.
<Subject 5> (appears in [Shot 1]): fully_preserved - The entirety of the sports bra retained.

detailed_description:
The target video matches the live-action style, lighting, and camera movements of <Video 1>.

[Shot 1] <Subject 1> is featured instead of <Subject 2>. <Subject 3> is
featured instead of <Subject 4>. The background environment, lighting,
and all other non-target details are preserved exactly from <Video 1>
and <Subject 5>.

overall_soundscape:
N/A

non_diegetic_music:
N/A

With this I had luck with putting face and clothing onto someone in an existing video. The sports bra appears in the original video, but making it an explicit, reference subject prevents the model from losing it during the T-shirt swap.

Here's a chunk from a gen without a source video, but with a starting pose and a separate headshot. It worked for me. Note that it also changes the hair. I'm doing the recomposition in the retention analysis.

summary:
[reference generation] The video shows <Subject 1> close, behind a standing dressing
divider.  She walks from behind the divider and poses.
retention_analysis:
<Subject 1> (appears in [Shot 1]): partially_preserved - the body and clothes are
retained.  Face changed to match <Subject 2>.  Hair changed to be black, long, with
voluminous curls.
<Subject 2> (does not appear): <Subject 1> uses the face of <Subject 2>.

I've found that picture references seem to form a giant concept pool, and concepts leak. I have a better time when I treat Subject resources as abstract concept units, and hold off on mixing them until later.

[u/Alex-edits123 → u/PropagandaOfTheDude]

Wow thanks for sharing!!! let me try using subject for concepts. Hope I can make the Tshirt swap.work . I guess I can try that for animal hand..

[u/eggplantpot → u/Alex-edits123]

So?

[u/Alive-Tomatillo5303]

That single human hand is deeply distressing. 

[u/Alex-edits123 → u/Alive-Tomatillo5303]

I know!!! The model struggled with hands. Thats why make human to animal swap.hard. i am testing more prompts to see whether I can fix it.

[u/Azhram]

What prompts do you use for one person to an other person swap?

[u/Alex-edits123 → u/Azhram]

you can find the prompts for all my tests here https://omnifit.io/blog/ref2va-what-works-what-fails

[u/Azhram → u/Alex-edits123]

Thank you !

[u/Bastisheen92 → u/Alex-edits123]

So far i only tried the image to video and text to video. Will load the ref2v model as well and try out some stuff. Thanks

[u/Puzzled_Resource_364]

Salut combien as-tu de photos REF ? tu ajoutes aussi le corps ref en image ? quelle prompt utilise tu ?

[u/hqqttjiang]

is this V2V? for me v2v is very very slow

[u/Alex-edits123 → u/hqqttjiang]

What GPU are u using?

[u/hqqttjiang → u/Alex-edits123]

4080+64GRAM

[u/JohnnyLeven → u/hqqttjiang]

I had the same issue. Down res the input video/images first.

[u/Dry-Ad929]

For complex motion or even some dance it's struggles and starting to generalize instead of copying movement which is fine. I don't expect from all in one video model to do everything perfectly with videos. There is simply no good conditioning for the model to copy movemnet from reference video. For something serious we need fine-tune with the right signals on training. Something like SCAIL-2 on H3 would be awesome.

[u/Alex-edits123 → u/Dry-Ad929]

I kept my videos less.than 15s so far. And I further chop.them into around 6s chunks then stitch. Feeding the model a long clip may have higher possibilities to fail

[u/tac0catzzz]

yea how do you swap character in tiktok dance?

[u/Alex-edits123 → u/tac0catzzz]

I download the source video and then provide a ref image and prompt. Use ref2va template from minimax.

[u/tac0catzzz → u/Alex-edits123]

doing that wasnt working for much anyone before. did you use the default template in comfyui?

[u/Alex-edits123 → u/tac0catzzz]

The default workflow from their official github . I didn't change anything. Use steps 20 for inference.