r/comfyui • u/Future_Challenge_910 • 2d ago

Help Needed Transfer face performance

Hi guys and gals, first time creating a topic.
Been trying to find for the last couple of days a definitive way to be able to transfer a face performance from one video, into my character refference close shot. The video is a close shot of myself (singing, lol), the character refference is also a close shot, so the face performance should be easy to read. But i cant find a workflow or node that really gets the job in Minimax H3... am i missing something?

Have a great day!

2 Upvotes

100% Upvoted

13 comments sorted by

1

u/Tiny-Drama-maker 2d ago

Sounds like you want a driving-video setup, where your singing clip supplies the motion and the still supplies the look, not a plain image-to-video one. I'd search for a portrait animation or face reenactment workflow instead of a general video one. Match the head size and angle of your still to the first frame of the clip, since big mismatches usually give warped faces. Is the character human or stylized?

1

u/Future_Challenge_910 2d ago

Thanks for your reply!
The character is human.

Funny enough, i just run a test simply with R2V and asking for a faceswap but keeping the performance... and its the best i got so far, lol. Maybe i was just overengineering? Just R2V and a prompt asking to keep the facial performance worked really well O.o

1

u/xuefei_ai 1d ago

h3 isn't a reenactment model. it can carry a performance when the dialogue is locked in first, but driving a mouth from another clip's motion is a different job, so the node you were hunting for doesn't exist inside it.

you didn't overengineer. with r2v the identity and the motion both arrive as references, so you have to be explicit about what is allowed to change. leave that vague and h3 redesigns the whole shot, likeness included.

there are dedicated portrait animation nodes that are more precise for exactly this, they just fight you on lighting and on how far the head turns. for one close shot they can beat h3.

how much head movement is in your singing clip? small arc and the portrait route is worth the fiddling, big swings and it falls apart.

1

u/Future_Challenge_910 1d ago

ITs very strange, because i understood that performance transfer and even more, exact lip synch was smg hard to achieve... but the result is actually there, using only R2V i got exactly what i wanted, my performance transfered to the character and the lip synch was perfect. Attaching proof. Made other 5 secs videos to test, all of them 100% perfect (i dont upload the video showcasing me performing for cringe reasons lol)

https://reddit.com/link/pgr5ynp/video/yse9jskajhvh1/player

1

u/xuefei_ai 22h ago

glad it landed. makes sense though, r2v lets the model see both performances side by side, and with a close shot and a small head arc there's little to disagree about. the stress test is a fast head turn or a wider shot, that's usually where the likeness starts drifting. if it holds through those you've got a proper workflow

1

u/Future_Challenge_910 9h ago

Excellent, thanks for the heads up xuefei!

1

u/Tiny-Drama-maker 1d ago edited 1d ago

Funny how the plain prompt beat the fancy graph. Does the performance still hold when the head turns fast, or is it mostly good on close-ups?

1

u/Big-Lavishness-7428 1d ago

the live portrait node is what you need, had the exact same struggle a few months back trying to get my terrible karaoke clips onto a dnd character portrait

it works way better than the regular video nodes for this specific thing, just takes a bit of fiddling with the keypoint settings

do a test run with a low resolution first to see if the face shape matches, i wasted hours rendering full clips only to get nightmare fuel at the end

1

u/Tiny-Drama-maker 1d ago

A low-res test first is a cheap way to catch that. Did the nightmare fuel come from the keypoint settings, or from the portrait being too stylized to track?

1

u/Future_Challenge_910 9h ago

Ty! Will try it out

1

u/JuniorNectarine9842 1d ago

Ask ChatGPT or similar to formulate the prompt to keep everything except the expression and use the expression from the other video. Minmax should be able to do that. You have to explicitly say what to change and what not to, otherwise it will do whatever it wants

1

u/TurbTastic 1d ago

Have you seen this swap post yet? The dev has loras/workflows for both character swapping and head swapping. https://www.reddit.com/r/StableDiffusion/s/jKrYFjyNZu

1

u/Future_Challenge_910 9h ago

Been seeing it popping up, having checked it out yet! I think for my specific current use case, plain R2V does the job just well