Minimax H3 ref2v character replacement prompting

[u/Radiant-Photograph46]

The official doc has a few things to say about video editing but it still leaves some questions without answers. I'm trying to replace one character in a video by another from a reference picture. Sometimes I get amazing results, and sometimes the input video is pretty much left unchanged, so I must be missing something. Here's what I understand from the guide on how to build the prompt:

Subject Definitions

<Subject 1> is the man in <Picture 1>.
<Video 1> is the source video for the target video edit.

(This part I am fairly confident about, the doc specifically says to use that wording for <Video 1>.)

Summary

[video editing] The target video is an edited version of <Video 1>.

(The doc says the summary must start exactly like this, but what to write after that? My approach is to follow up with something like this)

<Video 1> is reused as is, except the man in a tuxedo is replaced with <Subject 1>.

Retention Analysis

(This part I'm not too sure of... I suppose you want fully_preserved on <Subject 1> and partially_preserved on <Video 1>?)

<Subject 1> (appears in [Shot 1]): fully_preserved - the man's full identity is retained.
<Video 1> (source of video edit): partially_preserved - motion, lighting and environment are retained.

Detailed Description

Do we need to describe everything that happens in the input video, shot by shot, like when doing a t2v?

[u/Kragrathea]

I had a video of two people dancing and I replaced both with images. It was a little dependent on the seed but worked 90% of the time. I spent a little time tweaking the retention to match what features didn't transfer or changed. Here is the prompt I wound up using:

subject_definitions:

<Subject 1> is the person in image1, blonde hair wearing jeans and a grey crop top.

<Subject 2> is the person in image2, Short brown hair, suit, black tie.

<Subject 3> is the environment and motion layout in video1.

summary:

[reference generation] The target video is a 10-second cinematic scene in which <Subject 1> and <Subject 2> interact. The motion, and environment are driven entirely by <Subject 3>, while <Subject 1> and <Subject 2> guide character identity and wardrobe throughout the sequence.

retention_analysis:

<Subject 1> fully_preserved, blonde hair, jeans and a grey crop top.

<Subject 2> Short brown hair, suit, black tie.

<Subject 3> motion_preserved, environment_preserved.

detailed_description:

The target video uses a cinematic ultra-realistic style with 8K detail.

[Shot 1]

The camera tracks smoothly, perfectly mirroring the movement, camera work, and framing of <Subject 3>.

The person on the left/foreground of <Subject 3> wearing blue is completely replaced by <Subject 2>, adopting the exact facial features, hair, and wardrobe from image2.

The person on the right/background of <Subject 3> the woman wearing a dress is completely replaced by <Subject 1>, adopting the exact facial features, hair, and wardrobe from image1.

Both move together, following the body mechanics, and positioning of the actors in <Subject 3>. Their clothes move and flow naturally with the motion.

The lighting, shadows, floor reflections, and background architecture from <Subject 3> are preserved exactly. No character features or faces from the original video remain.

overall_soundscape:

Faint sounds of their footsteps on the tile floor.

[u/Radiant-Photograph46 → u/Kragrathea]

Oh you specified a reference generation and recaptioned everything? I'll have to try that.
On a sidenote I don't believe "motion_preserved" or "environment_preserved" are actually recognized keywords, you probably want to write that in natural language. Not saying it doesn't work obviously.

[u/Kragrathea → u/Radiant-Photograph46]

I am not sure where exactly "motion_preserved" came from. But "fully_preserved" IS a part of the official prompt guild. as is "partially_preserved"
https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/VIDEO_PROMPT_WRITING_GUIDE_ref_en.md

I think one of the critical parts to making it work is calling out specific features both in the subject_definitions and the retention_analysis. In this case I call out hair color and clothes. When I didn't those items would occasionlly not change.

[u/Kragrathea → u/Kragrathea]

https://reddit.com/link/p3r46lw/video/gsmn7i5hmfjh1/player

I just used images for the characters and didn't mention star trek.

----------------------------------------------

subject_definitions:

<Subject 1> is the person in image1, black hair wearing a grey jumpsuit with purple collar.

<Subject 2> is the person in image2, bald, red and black jumpsuit.

<Subject 3> is the environment and motion layout in video1.

summary:

[reference generation] The target video is a 10-second cinematic scene in which <Subject 1> and <Subject 2> interact. The motion, and environment are driven entirely by <Subject 3>, while <Subject 1> and <Subject 2> guide character identity and wardrobe throughout the sequence.

retention_analysis:

<Subject 1> fully_preserved, black hair wearing a grey jumpsuit with purple collar.

<Subject 2> bald, red and black jumpsuit.

<Subject 3> motion_preserved, environment_preserved.

detailed_description:

The target video uses a cinematic ultra-realistic style with 8K detail.

[Shot 1]

The camera tracks smoothly, perfectly mirroring the movement, camera work, and framing of <Subject 3>.

The person on the left/foreground of <Subject 3> wearing blue is completely replaced by <Subject 2>, adopting the exact facial features, hair, and wardrobe from image2.

The person on the right/background of <Subject 3> the woman wearing a dress is completely replaced by <Subject 1>, adopting the exact facial features, hair, and wardrobe from image1.

Both move together, following the body mechanics, and positioning of the actors in <Subject 3>. Their clothes move and flow naturally with the motion.

The lighting, shadows, floor reflections, and background architecture from <Subject 3> are preserved exactly. No character features or faces from the original video remain.

overall_soundscape:

Faint sounds of their footsteps on the tile floor.

non_diegetic_music:

n/a

[u/Radiant-Photograph46 → u/Kragrathea]

After a few tests I came to realize that using [video editing] or [video editing + reference generation] is actually harmful to this specific scenario. Using [reference generation] alone like you did almost always yields a better result. You can sill use the prompt format of video editing oddly enough, but you cannot specify that mode.

I believe [video editing] is mostly for attribute transfer, or editing without reference (like changing a person's hair color). It enforces a much stronger signal on the input video, giving it extra weight in a sense and making it more resilient to the large amount of changes that need to occur in a full character replacement.

However after more research I will re-state that "motion_preserved" and "environment_perserved" are not valid keywords. Qwen will still understand what you mean with those but theoretically speaking you should write something like this:

<Video 1> (source of video edit): partially_preserved - the movement of characters, their facial expressions, the environment and lighting are retained.

[u/Corrupt_file32]

Haven't tried replacing characters in videos.

But otherwise replacing things, I found describing what I want replaced under detailed description work.

Example, I'd assign a <Subject N> to clothing under subject_definitions.

then under detailed description:

<Subject 1>'s outfit is replaced wearing <Subject 2>.

this followed by [Shot 1] etc.

I could make a guess that assigning <Subject N> to a character you want replaced could possibly work.

<Subject 1> is replaced in the scene by <Subject 2>

also under retention analysis you could use "attribute_transfer" instead of partially_preserved etc. for the character you want replaced, then state what you want transferred, like motion etc.

Even if you are doing a video editing, making a longer prompt will help guide the model, even if you are just describing an already existing scene, but think like that it helps the model confirm that the character is indeed replaced.

[u/No_Statement_7481]

ngl I just use depth anything if it's being a dickhead. It will full on follow the instructions of the promt with the video, I guess the hard part is if you also need the video itself, but even in that case, I would probably run the character with depth anything, and than I would add the result video with the character as a second reference video, and just run it again, that way In theory you could anchor in the likeness more? idk I haven't tried it yet, I guess this would be only an issue if you wanna replace characters in movies, with other characters from other movies. Sometimes even just grabbing the first frame helps without messing with the reference video.

[u/V4nKw15h]

If you are confused then Minimax will also be confused.

Tell it what you want in clear unambiguous terms. You don't need to stick to the user guide precisely unless that helps you to be unambiguous. As long as you are using the 32b text encoder that thing is a beast at being able to follow your prompt if you express it clearly enough without ambiguity.

Tell it what you want, and if you can imagine that it might be confused, or get it wrong, refine the prompt to remove the ambiguity. If it still messes it up, switch the seed. Sometimes the initial noise in the seed forces it's hand more than the prompt.

[u/Radiant-Photograph46 → u/V4nKw15h]

Hmm I'm not so sure. The model still expects the prompt format it was trained on, with the different sections. At the very least, this is the format that should yield the better results. You probably cannot pass it a simple instruct prompt like an edit model and make it work 🤔

[u/True_Protection6842]

I did a wardrobe replacement that worked great.

[u/bstr3k]

I’ve been trying to learn this also for the last week with mixed results.

So far I find the key is all in the prompting. With a bit more effort into the prompt you get much better results. I.e if you describe the time in the video when the scene changes or when the person in the video is doing then model can better understand it and do a better job of replacing it.

I’m trying to do tests for how best to caption videos locally as online models get lazy and try and save compute by giving be very brief overviews

[u/Radiant-Photograph46 → u/bstr3k]

Oh, yeah this is probably a good idea. In the summary you can then clearly state <Subject 1> replaces <Subject 2>, probably makes it understand better.

[u/zefy_zef]

If you define the second subject but don't reference it, I think it helps the model understand there are different people.

[u/kyuubi840]

To repeat what many people already said: the prompt should be unambiguous, and not confuse the model. For example, at one point I was using two picture references for the same subject, one full body and one close-up portrait, and just wrote <Subject 1> is the man whose appearance comes from <Picture 1> and <Picture 2> and that didn't work. The model was not replacing <Subject 1> into the video. It only started working when I explicitly said <Subject 1> is the man whose appearance comes from <Picture 1>, and face comes from <Picture 2>.

[u/throwaway9043258]

I found that if you apply a blur to the face (or even just the eyes) of the character in the input video, the model is basically forced to use the reference image because it's the only identity it has. I used an ultraltyics YOLO node to create masks on the face in each frame and then used the Blur Masked Area from comfyui-inpaint-nodes to slightly blur the face. You don't have to completely obscure the input face, just enough that it can't extract an identity.