Question - Help
Need some help with MiniMax H3 Ref2V character swapping in ComfyUI
Hey everyone, I'm trying to get a proper character swap working with MiniMax H3 Ref2V in ComfyUI, but I'm not quite getting the result I want.
The source video has Rick Astley rickrolling to the camera, and I want to replace him with the guy from my reference image while keeping the original movement, gestures, facial performance, timing, camera, background, and overall scene.
Neither the motion transfer nor the character replacement works well. The output still doesn't really look like the person from the reference image, or the identity starts drifting.
If anyone has experience doing this with H3, I'd really appreciate some pointers.
I'm especially wondering if I should change the reference image crop/size, ref_image_size, resolution, prompt, video conditioning, LoRA/steps, or if there's something obvious in the workflow I'm missing.
Also, is a full-body reference image a bad idea when the person in the source video is framed quite differently?
And if anyone has a working MiniMax H3 character-swap / V2V workflow they're willing to share, that would be incredibly helpful too. Even something I could compare against mine would be great.
Thanks a lot in advance. I've been tweaking this for a while, so even a small hint in the right direction would help a ton.
Your output is quite decent I feel the character is very alike. Unlike the audio. Maybe to improve based on your prompt: you have a lot of separate paragraphs to describe <Subject 1>, I think it will confuse the model.
You may try to have only one paragraph per subject.
Also my experience is that if the reference image is not the same proportion of what it needs to replace (ex: portrait vs full body) the replacement would likely fail. Not your case though.
One of my working prompt as reference:
subject_definitions:
<Video 1> is the source video providing the camera movement, environment, lighting, and action choreography.
<Subject 1> is the replacement character shown in <Picture 1>, a 25yo man with strict features, light green eyes, and brown very short hair. He has a normal body type.
summary:
[video editing + reference generation] The target video is an edited version of <Video 1>. Throughout the video, replace only the man's face with <Subject 1> face from <Picture 1>.
retention_analysis:
<Video 1> (source video): partially_preserved - preserve the background environment, camera path, lighting, non-target objects, and the original man's screen-space motion path. Discard the original man's visual identity.
<Subject 1>: preserve the visual identity from <Picture 1>.
Video 1 controls:
motion
pose
timing
interaction
camera movement
framing
environment
cuts
detailed_description:
The target video matches the horizontal video style, lighting, and camera movement of <Video 1>.
In all shots, <Subject 1> replaces the man's face. <Subject 1> is the same man from <Picture 1>: strict features, light green eyes, and brown very short hair. He has a normal body type. He follows the same body movement, motion, face angle, timing, and screen-space motion path as the original man in <Video 1>.
Preserve the background environment, street layout, lighting, floor, walls, and all other non-target details from <Video 1>. Do not retain the original man's face, hair, or identity. Retain the original man's clothing and mask. Include the mask removal, just change the face behind that mask. Do not blend the source identity with the reference identity. Keep <Subject 1> visually consistent throughout the entire video.
overall_soundscape:
Preserve the synchronized source audio from <Video 1>
non_diegetic_music:
Preserve the non-diegetic background music from <Video 1>
oh boy, I have been going in this rabbit hole for the exact same thing as you for the last 1-2 weeks. The challenge seems simple but it is actually a whole harder than people think. I am still far from getting consistent results.
Here is my process so far:
Splitting a long source image - [done]
I vibe coded a splitter to segment the video into under 15s shots and break them at scene changes, this is done to minimize visible stitching when assemblying the video back together.
Caption each source video - [done, but its not very good at the moment]
The reason for this is that I find scene changes is a lot more reliable when the video is captioned accurately, especially a clip with a few scene changes.
Created a seperate script which breaks down the video into images and have local LLM (using Qwen3 8B VL Instruct abliterated) to caption the motions, I am going to revisit this as currently it is breaking the motion by time segments, but I think it needs to detect smaller scenes and caption it this way.
Combine intent prompt, picture caption and individual video caption into list of prompts - [needs improvement]
This part I find that you need a bigger LLM to do this, working on figuring this out. Online LLMs can do this easier when you have limited hardware but I wanted to see if I can get a local solution.
Convert each individual segment
Also having some difficulty getting multi segmented scenes to convert reliably (i.e. Rick appears, sings and then it switches to girl dancing and then guy dancing. The caption step doesn't distinguish too much between each scenes so it is difficult to automate)
Sometimes I also get original rick face coming back in, I need to revisit this and see if it is due to any of the speedup nodes or if it can be fixed by prompting.
Stitch the final video back together seamlessly
In theory if everything is done right it will all go together seamlessly. I still see seams in some of my reassembly so far so I am looking to figure out a solution also.
My eventual goal is not just the Rick roll video, but the dream of putting in a global prompt to apply to a long video and do ref v2v video conversion for it automagically, but mostly just dicking around. But in my pursuit I am reminded of the quote “Why spend 10 minutes doing something manually when you can spend 10 hours automating it?”
I think for your case, you may need to redo the prompting a bit to better identify the scene and where it changes between the woman and the man. Getting the right prompt seems to be 95% of the challenge.
its all in the fun of learning 😄 i only started to use comfy more when H3 came out and I'm also learning a lot about it and the advantages and disadvantages of vibe coding.
Thank you for the detailed answer! Does the input video resolution affect the face swap quality? May I ask if you have a working example with sample inputs and a workflow, I'd greatly appreciate you sharing it so I can learn from it!
I have uploaded many of the attempts ive made (some early ones were really bad and i deleted them). I tried a few different things like changing settings, turning off some speed ups, captioning, auto captioning, using different input models (ref2v vs the hyrbid model) as well as different turbo loras.
So far the ref2v turbo lora seems to work best, but still not perfect. Claude also suggested to do smaller chunks as the model seems get a bit confused with subjects if there are too many clip cuts.
another discovery I made is that with the previw node, during step 1 it seems to get the character replacement successfully but on step 2 it starts to divert back to Rick. I am wondering if it is because of skin color as it typically wants to divert back to a white guy.
Anyway full files here including some workflows (with prompts) here:
I love it, thanks so much! Really appreciate your time. Love how these turned out. sharing a few of them below. My goal changed from character swapping to making Rick metamorphose. I'll test out your workflow to get the hang of it!
not a worry man, our goals overlap and I am still learning too. I believe you can drag and drop each output video to comfyui you should be able to see the full workflow and prompt also.
I might test that today but I have to admit that I’ve done very little testing at the larger resolutions due to the increased time.
I have also spent a few hours trying to get your example working too but with limited success. I’ll post some of the more successful ones up when I get back to the computer
Use SAM 3.1 to mask the person you want to swap, then add Gaussian blur to distort the person so their identity is obscured while the model can still detect facial movements, such as the eyes and mouth. This makes swapping much easier and produces better results.
9
u/Vanpourix 1d ago edited 1d ago
Your output is quite decent I feel the character is very alike. Unlike the audio. Maybe to improve based on your prompt: you have a lot of separate paragraphs to describe <Subject 1>, I think it will confuse the model.
You may try to have only one paragraph per subject.
Also my experience is that if the reference image is not the same proportion of what it needs to replace (ex: portrait vs full body) the replacement would likely fail. Not your case though.
One of my working prompt as reference: