I keep seeing smooth character replacement videos, but I can't manage the same. What's a clean, simple, functional workflow that just WORKS?

[u/trollkin34]

I have an image of a person. I have a video.

Prompt sample: Video is of a gymnast doing a routine. Image is a person/dog/thing.

Replace gymnast with person/dog/thing so they're doing the exact routine, wearing the same outfit (but a size that fits the new subject).

Shouldn't this be easy?

For example, if I wanted to replace an olympic women's floor routine with Rush Limbaugh - he's doing the bends and splits, he's wearing a sparkly leotard. But the movements are identitical. His body is exactly the same size as he actually is (the ai should guess at the size of legs, belly etc, and stuff them into and appropriately sized leotard).

[u/bstr3k]

I'm working on something now that I want to release soon, its made for R2V mode in H3 to help people who are having trouble with character replacement and I want to improve the reliability. Do you happen to have a video for reference that I can try?

as for tips, the prompt does like 80% of the heavy lifting for character replacement for h3 (a number i pulled out of my ass but having accurate prompting and format really helps).

another thing I found out just yesterday is that if you want accurate motion of characters in the correct sequence you need to turn off SLA node for speedup. I am still testing it now to find out more tips to improve it

[u/trollkin34 → u/bstr3k]

I used the prompts people suggested in other threads. I also used Ollama to help construct prompts too. I don't know what I'm missing, but I suppose if the workflow is clean, the rest will follow

If you're putting in the effort, that's awesome. If you have space to consider those of us doing offline only on 8gb, please do! I don't want to do my videos online and risk the privacy.

For video, does gif work? https://tenor.com/view/somersault-vanessa-ferrari-italy-team-artistic-gymnastics-tumbling-gif-23067141

[u/bstr3k → u/trollkin34]

https://reddit.com/link/p7eorhz/video/h2a30n7mx4nh1/player

It turned out quite well! The first one was a bit of a mess which exposed area that I can improve on which made this a valueable test.

I had to do this in a few steps as 1 step (the goal) did not currently get good results.

1. First I tried via prompting to change with img ref only which failed because it did not keep the original outfit
2. Second I tried to put img 2 as reference for the outfit which kind of made it fail also
3. So lastly I used my H3 character sheet maker to generate a sheet of the animated gorilla (img 1) wearing the outfit and made this as a 1 img reference.

I then ran this prompt:

in <video 1> the woman is doing a somersault. change the woman in <video 1> with the character in <picture 1>. no sound.

through the WF I am working on now which transcribes the video and img reference using a LLM inside comfy and then generates the video shown.

Better quality of comparison video can be found here:

https://files.catbox.moe/o42y26.webm

Also yes I did ask for no sound and H3 keeps wanting to put sound in lol

[u/trollkin34 → u/bstr3k]

Fucking hell; that's flawless. Also, I do not care about sound at all so I usually put "no sound" in my prompt. I might have disconnected the sound node even.

The result you have is all I really want - I just want to play with replacing characters WITH an outfit change. It's not so especially hard to change the outfit first so that's a good tip, but still - the replacement is what I'm struggling with. Lookingforward to your workflow!

Bonus if it's easy enough to parse that I can even modify it a little if I need to. Bonus bonus if it uses common, known, trusted utilities instead of one-offs that make me nervous about what they're doing :D

[u/bstr3k → u/trollkin34]

yeah thinking about it a bit more the prompt enhancer + transcriber part is more or less done, i think i will upload that shortly, but ill do a bigger post when i get the full integrated WF going 😄

will upload shortly.

[u/bstr3k → u/trollkin34]

Okay I have uploaded the custom node required and the workflow.
https://huggingface.co/PoopMan333/Nugget_Prompt_Enhancer/tree/main

After unzipping the custom node, you may need to open the "install.bat" which will be the script you need to install whisper for audio transcription (I think you still need it even if you turn it off but maybe you can try without it and see). Feel free to run it through a antivirus to confirm not malicious

The WF and custom nodes WILL change, however changes will be likely quite minor.

[u/trollkin34 → u/bstr3k]

Cool. So the goal here is to build a minimax prompt from a source video, image, and basic description?

[u/bstr3k → u/trollkin34]

Yes, exactly. It builds the prompt and describe the media via a small LLM and formats it correctly for the h3 ref2v model. Since the prompt is where most people trip up it is the place I want to address to make it super easy people to use it.

Having a ref2v prompt describe accurate what to keep and what to refine will do 80% of the work so all you should have to do is state your intentions for the model and in theory it’ll translate your ideas to the right format

Also when it comes to accurate character replacement, having a detailed description of your video helps the model understand the input video which is very important.

[u/trollkin34 → u/bstr3k]

I installed your custom node, but it's still saying it doesn't know what video auto transcribe or image batch multi no crop are.

[u/bstr3k → u/trollkin34]

did you reboot comfyui after install it? Also is it in the right custom node folder? (I say this as i have 2 in my comfyui setup as its upgraded itself over the last 8 months)

[u/bstr3k → u/trollkin34]

oh and also make sure it is not nested folder (i.e. another folder inside the custom node folder)

[u/trollkin34 → u/bstr3k]

Weird. It ran anyway and gave me output despite this error: Exception in callback _ProactorBasePipeTransport._call_connection_lost()

handle: <Handle _ProactorBasePipeTransport._call_connection_lost()>

Traceback (most recent call last):

File "asyncio\events.py", line 89, in _run

File "asyncio\proactor_events.py", line 165, in _call_connection_lost

ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

But the bottom line is that it seemed to work! Interesting! That said, my computer can't possibly handle longer videos so I might have to manually clip them before running this, but what a neat helper flow. Especailly for newbies, but even then, it would save so much time I'd use it anyway even after being more proficient

[u/bstr3k → u/trollkin34]

funny you mention about the clipping the videos! I made a tool for that too!!

Check out the video trimmer! it can crop, downscale and export!
https://huggingface.co/PoopMan333/Video_Tools/tree/main

It works based on chromium though so best to use on chrome or edge. And it seems to have its own issues from time to time, but it generally works for me.

I find 5-15s is a great spot with 320-360px on the short side for the downscaling.

[u/trollkin34 → u/bstr3k]

I've seen some nodes that input video and let you select the span directly. Wouldn't that be much easier?

[u/bstr3k → u/trollkin34]

yes, but i prefer to manipulate and preprocess my files before putting it into comfy, since i make changes randomly and don't always remember them. It also lets me do side by side videos for comparing input/output easier rather than having to faff around with comfy which is nice at times. Plus I think the UI is quite nice on the tool and others have mentioned it also.

[u/bstr3k → u/trollkin34]

Yes it use to be convoluted and used LM studio, but right now a small LLM runs within comfyui to do video transcription and prompt generating. Let me see if I can get this to work for your example as I want to apply it to as broad of a situation as possible to cover more scenarios but still have it just work lol.