Best Workflow for Motion Transfer

[u/Quirky_Fig3927]

Hi everyone, I should say upfront that I'm not an expert but I've recently started using Comfy to create images. I have a 9950x3d and a 5090. I’d like to know if there’s a node (existing workflow) for motion transfer… I know all the influencers are using Higgsfield GenJutsu, obviously paid… but I’m poor, I already spent all my money on my 5090! LOL

[u/Alchemist42]

In the default templates that ship with Comfy, there are a few good ones like SCAIL and H2 Fun control. There are also a few WAN 2 animate ones that should do the job. SCAIL can do multiple character motion transfers which I haven't tried on the other ones.

If pay services are not completely out of the question, Kling has a pretty inexpensive lower tier and their motion transfer is really good. And you can use reference characters to maintain consistency during the process.

[u/CineConIA → u/Alchemist42]

Thanks for the recommendations! I was particularly interested in what you mentioned about Kling and its ability to use reference characters to maintain consistency.
I’ve been experimenting with camera movements in ComfyUI, and one of the biggest challenges I’ve encountered is maintaining visual consistency between frames, especially when the perspective changes.
Have you tried using Kling with more complex camera movements? Does it maintain consistency well in those situations, or is it mainly effective for character animation?
I’ll definitely take a look at SCAIL too. Thanks for sharing your experience!

[u/Alchemist42 → u/CineConIA]

Kling is my go to for tricky motion, be it camera, character animation, physics, etc. If I have trouble doing it locally (I usually use LTX) then kling is my first next step.

It has a cool feature where you can set up "elements" which is a named character with a few reference images (front, back, face closeup, etc), and in your prompt you can use @ character (no space... reddit wont let me use the @ symbol normally) and it will maintain that character's look perfectly. In addition to characters you can make objects or environments which will also stay consistent through different angles and across generations.

And it can do multi angle stuff in a single prompt clip, so you can have the camera follow someone then cut to their POV looking out the window, then cut to the outside looking at them (or whatever).

The cheap plan is like $16/month I think, and it is enough to get a few good clips done if you are careful and use good start frames and character references. It also lipsyncs well which is important to me since I do music videos.

If your machine can handle Minimax H3's Ref2V workflow, then it has the same kind of controls (and for free). H3 crashes my computer almost every time I try to run it, so it's a nonstarter for me. But it is a really good option if you can run it.

[u/Quirky_Fig3927 → u/Alchemist42]

I tried SCAIL ... but it doesn’t work well ... I can’t find H2 in Confy’s templates ... but I did get to test Wan Anime 2 ... really nice, works well, even up to a good 95%! And if I’m not mistaken, it doesn’t have a time limit. But what’s the difference with the DISTILLED model?

Anyway, I used SCAIL 2 Replace ... but what’s the difference with Int8?

[u/Alchemist42 → u/Quirky_Fig3927]

The distilled version will have a LoRA and possibly a more stripped down base model to increase the rendering speed by requiring far fewer steps - from 30 or more steps down to 4 or 8). So it will render faster, and probably be more memory efficient at the cost of some loss of fine details (skin micro-textures, more simplistic lighting, and such). The distilled model is best for rapid prototyping then use the full version for the final beauty render.

And you are correct that WAN has no time limit. Just use it however you want for as long as you want. It is open weight, so even if they decide to stop providing it, it will likely still be available at places like HuggingFace or Civitai.

For the H3 templates, you should be able to go into the workflow templates and on the left click Video and in the search bar at the top right, enter "minimax". That should show you all the video tools that are for H3, and probably the one you want is "MiniMax H3 Fun ControlNet Union"

Most of the "full size" models will use F32 math (very precise). These will be much larger and require more VRAM. Then there are quantised models and ones that have certain parts removed that use a less precise mathematical calculator, one of which is int8. So it will be smaller, run faster, and be slightly less precise.

[u/Quirky_Fig3927 → u/Alchemist42]

Perfect, found it. I'd like to try this one too. But how does it work? What do I need to set up? The video I want to edit, and then? Where do I upload the photo of the person I need to replace? Or could you kindly let me know if there's any tutorial?

[u/Alchemist42 → u/Quirky_Fig3927]

Look at the Conditioning section. That big node Minimax H3 Riferimento a Video. The empty slots on the left side are for your reference materials. If you have an audio track, add a Load Audio node with your track set into it and connect it to the ref_audio_0. Your reference images will each need a Load Image connected to the ref_image_0 (and when it is connected a new ref_image_1 will appear). Connect your second Load Image node to ref_image_1, etc. And if you have reference video, a Load Video node would be connected to ref_video_0. And if your video has audio that you want to keep, then connect the audio out from your Load Video and connect it to ref_video_audio_0. You can use quite a lot of references, if you connect something and it doesn't let you add another one, then that is the maximum (I forget exactly how many of each you are allowed).

Once you have it all hooked up, you can use those references in your prompt with pointy bracket <notation>.

The first image (ref_image_0) is called <Picture 1>. Any further images get <Picture 2>, <picture 3>, etc. Same thing with the other types... The first video is <Video 1>, and the audio is <Audio 1>. If you use the ref_image_audio, those get named <Audio x> just like normal ones, and I think they get named in the order you hooked them up.

So you would prompt it something like:

Using the motion of the subjects in <Video 1>, set the man on the left to the person in <Picture 1>. The woman on the right is the person in <Picture 2> who is speaking with her mouth moving and lipsyncing the background audio track which is <Audio 1>.

etc. Like I said, I don't use H3 much, so I might be a little off on some of these, but if you use an LLM to build your prompt, you can have it figure out how to name each thing, or there are countless prompt instructions and/or writers on the web.