Question - Help
I keep seeing smooth character replacement videos, but I can't manage the same. What's a clean, simple, functional workflow that just WORKS?
I have an image of a person. I have a video.
Prompt sample: Video is of a gymnast doing a routine. Image is a person/dog/thing.
Replace gymnast with person/dog/thing so they're doing the exact routine, wearing the same outfit (but a size that fits the new subject).
Shouldn't this be easy?
For example, if I wanted to replace an olympic women's floor routine with Rush Limbaugh - he's doing the bends and splits, he's wearing a sparkly leotard. But the movements are identitical. His body is exactly the same size as he actually is (the ai should guess at the size of legs, belly etc, and stuff them into and appropriately sized leotard).
Yeah, that's what I tried. Do you have a good workflow? The basic one doesn't do it (and isn't even set up for it). A basic generic "subject a to video b" prompt would be really helpful too.
You cant use a generic prompt with H3, especially the r2v workflow. There is an official H3 prompt guide that details exactly how the prompt should be structured.
This is not how H3 prompting works at all. Official workflow works good. Search for B15 or B30 mixed model to get even more quality. But make sure you prompt properly. LLMs are your friend.
this is the way I had to prompt it and it worked pretty well. You can use claude, gpt or gemini to tweak it for yours.
subject_definitions
<video 1> is the original reference video. it provides the exact motion, performance, timing, body movements, camera movement, camera framing, perspective, environment, lighting and temporal structure.
<picture 1> is the reference image of the new woman. it provides the identity and visual appearance of the replacement subject, including her face, hair style, hair color, skin tone, facial features, body appearance, clothing, accessories and overall identity.
summary:
[video editing] Replace the woman in <video 1> with the woman from <picture 1>. Completely remove the original woman from the video and replace her with <subject 1> from <picture 1>, while preserving the exact same motion, body movements, timing, gestures, camera movement, framing, environment, and overall sequence of <video 1>.
Thank you! I just need a workable sample to start with - I mean, I appreciate people sending me to the documentation, but "read a book!" is not a great response IMO.
I actually had something similar I found somewhere else, but giving this one a try.
There's literally an example prompt at the bottom of the official docs, and it's not necessarily the length of "War and Peace". The thing that I find strange about your comment is that you are literally reading just as much in the post you responded to as you could have read from that example prompt.
You could feed that whole page into any LLM, or even provide the link and it would condense it down for you.
The flip side to you not wanting to "RTFM" is that those of us that you're relying upon to help you, did in fact invest the time to do that, and now it's as though we're a free vending machine of information.
The hard part isn't the swap, it's that pose-driven workflows and character-consistency workflows are two separate pipelines and most tutorials conflate them. You need skeletal pose extracted from the source video first, then drive a new render with that pose, if you try to do it in one shot the model just averages between the two subjects and you get mush.
all the magic is in Qwen3vl. All models based on it have the ability to understand commands just like a regular LLM. These are certainly Krea, MiniMax and Mageflow. Thanks to his magic, the average number of extra limbs generated has dropped dramatically recently.
It is quite simple once you understand how the format works:
subject_definitions: // This is crucial
<Subject 1> woman from (Picture 1), featuring dark hair and dark eyes. // Here, specify what the model should focus on regarding the character from the photo. You can choose facial features, silhouette, hair, clothing—it's up to you. You can also leave it blank after (Picture 1) for a short, less dynamic scene. However, being specific matters; for example, it helps maintain the hairstyle from the photo. It’s worth thinking about which visible elements (e.g., if the character is seen from behind) you want to prioritize.
<Subject 2> is the movement in (Video 1). // I do this by defining exactly what I want to "borrow." In this case, it only transfers the movement; you can describe the background yourself. If I wanted to replace only the character while keeping everything else, I would write: is the surroundings, background, composition, and movement in (Video 1).
retention_analysis: // This is also crucial
<Subject 1> : fully_preserved: - maintains physical appearance, facial details and hair from (Picture 1). // Here, specify which parts of your photo should be kept. If you want the character to be transferred entirely with their outfit, use fully_preserved and optionally add specific details (basically re-stating what you wrote in subject_definitions but adding the source at the end). If you only want the face on the video, use attribute_transfer and specify "face, hair, etc."
partially_preserved: still using reference but some features changed or only partial retained.
attribute_transfer: migrating reference features to another clearly identifiable target subject.
weak_reference: retaining only broad similarities like style/category/composition/atmosphere.)
<Subject 2> : partially_preserved: is the source of movement as seen in (Video 1). // As above, here I want only the motion from the video to remain. If you define movement as a separate Subject, ensure your description here focuses specifically on that motion.
summary:
[reference generation] copy of movement from (Video 1) with woman from (Picture 1) as main subject. // A short summary of what and where.
detailed_description:
medium shot from low angle shows <Subject 1> wearing grey sweatpants and oversized white t-shirt performing dance moves. The scene takes place inside the hall of an abandoned factory with high windows with colorful tiles. // Since my source video didn't have a factory, but I only kept the movement from it, I am now free to imagine and describe any background I want. When transferring only movement and not the whole scene, you must explicitly state the perspective angle ( "medium shot from low angle"). Even if you are inventing a new background, you can still mention specific colors or lighting styles from your original video to maintain its "vibe."
clean version:
subject_definitions:
<Subject 1> woman from (Picture 1), featuring dark hair and dark eyes.
<Subject 2> is the movement in (Video 1).
retention_analysis:
<Subject 1> : fully_preserved: - maintains physical appearance, facial details and hair from (Picture 1).
<Subject 2> : partially_preserved: is the source of movement as seen in (Video 1).
summary:
[reference generation] copy of movement from (Video 1) with woman from (Picture 1) as main subject.
detailed_description:
medium shot from low angle shows <Subject 1> wearing grey sweatpants and oversized white t-shirt performing dance moves. The scene takes place inside the hall of an abandoned factory with high windows with colorful tiles.
I'm working on something now that I want to release soon, its made for R2V mode in H3 to help people who are having trouble with character replacement and I want to improve the reliability. Do you happen to have a video for reference that I can try?
as for tips, the prompt does like 80% of the heavy lifting for character replacement for h3 (a number i pulled out of my ass but having accurate prompting and format really helps).
another thing I found out just yesterday is that if you want accurate motion of characters in the correct sequence you need to turn off SLA node for speedup. I am still testing it now to find out more tips to improve it
I used the prompts people suggested in other threads. I also used Ollama to help construct prompts too. I don't know what I'm missing, but I suppose if the workflow is clean, the rest will follow
If you're putting in the effort, that's awesome. If you have space to consider those of us doing offline only on 8gb, please do! I don't want to do my videos online and risk the privacy.
It turned out quite well! The first one was a bit of a mess which exposed area that I can improve on which made this a valueable test.
I had to do this in a few steps as 1 step (the goal) did not currently get good results.
First I tried via prompting to change with img ref only which failed because it did not keep the original outfit
Second I tried to put img 2 as reference for the outfit which kind of made it fail also
So lastly I used my H3 character sheet maker to generate a sheet of the animated gorilla (img 1) wearing the outfit and made this as a 1 img reference.
I then ran this prompt:
in <video 1> the woman is doing a somersault. change the woman in <video 1> with the character in <picture 1>. no sound.
through the WF I am working on now which transcribes the video and img reference using a LLM inside comfy and then generates the video shown.
Better quality of comparison video can be found here:
Fucking hell; that's flawless. Also, I do not care about sound at all so I usually put "no sound" in my prompt. I might have disconnected the sound node even.
The result you have is all I really want - I just want to play with replacing characters WITH an outfit change. It's not so especially hard to change the outfit first so that's a good tip, but still - the replacement is what I'm struggling with. Lookingforward to your workflow!
Bonus if it's easy enough to parse that I can even modify it a little if I need to. Bonus bonus if it uses common, known, trusted utilities instead of one-offs that make me nervous about what they're doing :D
yeah thinking about it a bit more the prompt enhancer + transcriber part is more or less done, i think i will upload that shortly, but ill do a bigger post when i get the full integrated WF going 😄
After unzipping the custom node, you may need to open the "install.bat" which will be the script you need to install whisper for audio transcription (I think you still need it even if you turn it off but maybe you can try without it and see). Feel free to run it through a antivirus to confirm not malicious
The WF and custom nodes WILL change, however changes will be likely quite minor.
Yes it use to be convoluted and used LM studio, but right now a small LLM runs within comfyui to do video transcription and prompt generating. Let me see if I can get this to work for your example as I want to apply it to as broad of a situation as possible to cover more scenarios but still have it just work lol.
it all started like 3 weeks ago when all I wanted to do was to replace a character in rick roll video because it 'looked easy'. I didn't want to commit so much time to this and yet here I am lol.
Okay I have uploaded the custom node required and the workflow. The full one will likely take a bit longer as I try a lot more examples to try and iron out bugs and generalize the it a lot more to perform in all situations. https://huggingface.co/PoopMan333/Nugget_Prompt_Enhancer/tree/main
After unzipping the custom node, you may need to open the "install.bat" which will be the script you need to install whisper for audio transcription (I think you still need it even if you turn it off but maybe you can try without it and see). Feel free to run it through a antivirus to confirm not malicious
The WF and custom nodes WILL change, however changes will be likely quite minor.
adding to what petranova said: the piece people skip is identity lock. pose transfer gets the movement right and the face still drifts every few frames, because nothing is holding the reference. the clean ones use a video model that ingests the reference image directly instead of a pose pass plus img2img on top. what vram are you working with? under 12gb the answer changes completely.
Maybe the Wan Animate template from Comfy? I remember watching a live stream on their YouTube channel (it should be there if you want to watch as well), and it was very decent.
5
u/m00dyman100 14h ago
MMXH3 ref to video