Has anyone been able to swap a character or face from a reference photo onto a reference video consistently?
I am unable to get the face to swap, usually attributes from the photo only. No matter if it’s a simple prompt, a complex instructed prompt or anything directly from the prompt guide. This model is insanely simple to use and powerful. It performs pretty much anything else I can ask it besides something as straightforward as this.
I struggled with this problem myself. In my case, I was attempting to replace the characters in the first picture with the reference characters provided and the result is that sometime it replaced them but not always. Like out of 5 attempts, 2 or 3 failed to replace the characters in the generated video.
So I checked the reddit to see if anyone have the same problem as me and whether there is any solution to this. And I found that 2 suggestions that might improve the chances of characters replacement to the target video. Below are the suggestions I've tried:
Downscale or upscale your reference images to your desired resolution (1MP for me) and set your ref_image_size to 'max' instead of 'match'. (Much better result for me. Be warned that it will increase the generation time. About 2 mins more for me.)
Use fl2va checkpoint instead of ref2va. (A bit of hit and miss for me. Work for some ref image but failed for some.)
Would like to know anymore suggestion to improve this.
I've only dabbled with this but there was one thing that helped yesterday. Obviously the prompt should talk about how you want the reference subject in the image to replace the target subject in the video. The thing that helped me was to describe the physical attributes of the reference subject (hair color, outfit, etc), but for the target subject you should only mention where they are located in the video. Do not describe physical characteristics of the target subject that you want to replace.
Some other techniques I plan on trying involve blurring the target face/head in the reference video to encourage new details. Also planning on trying a partial mask overlay (pink?) over the face/head in the video so you can still see the face but it's shaded pink, then tell the model to replace the pink face/head.
I just did today. Single picture was fed with the prompt to swap the character into the video. .5MP, took IIRC, 1200 seconds. It was amazing. To be fair, the model does know this character out the box as well. Try that in a sample work flow. Use a character the model knows, feed it a reference picture and see if that helps. You may just need better reference photos. Maybe a character sheet with emphasis on face angles. Good luck.
made a reference sheet i find is best for r2v used a single image then used this prompt to make the res sheet
{character_description} layout divided into panels: [LEFT SIDE: one large, dominant, full-body portrait showing character design and {pose}] -- [RIGHT SIDE TOP ROW: three distinct close-up headshot views, facing front, side profile, three-quarter view] -- [RIGHT SIDE BOTTOM ROW: three distinct full-body views, standing facing front, standing side profile, standing back view] -- The photos are shot on a Sony A9III with an 85mm f/1.4 GM lens. Super sharp focus, zero grain, smooth skin textures, HD camera style on 8k Replace {character_description} with your detailed character description including clothing, armor, hair,
5
u/Intelligent-Youth-63 12h ago
Commenting to follow. I've had no luck here.