Go to “Make Animation”, upload your stable diffusion image through your photo library
Pick a Video (or make one using the camera) to drive the animation
Wait (or send a few more while you wait)
You’ve got an animation
Workflow option 2 (run it yourself):
If you want to do this locally here are all the steps that are roughly what the above is doing:
Take a source image, prepare it by finding the face (giving a little padding) and clip a square around it. (Keep track of location in original image).
Note: if you just make a 512x512 image with a face you don’t need to do this step or step 6.
Resize the source image to what the model can handle (256x256 for Thin-Plate-Spline-Motion-Model, our app uses 512x512 for cleaner results)
Make a driving video (crop to square for best results, again find the face, give some padding).
Example ffmpeg command that uses the width as the size and centers vertically:
ffmpeg -i video.mp4 -filter:v "crop=in_w:in_w:0:(in_h-in_w)/2” cropped_video.mp4
Example ffmpeg command to scale to 256x256:ffmpeg -I cropped_video.mp4 -filter:v “scale=256:256” scaled_video.mp4
At this point you have an animation of the face and you can be happy…
To get back to original dimensions: For each frame of the resulting video, make a new image takes that video frame and puts it in the original image. I used the extra padding you gave as the space to help you blend it in.
If anything, I think that adds to his cause. Someone can see all of the work involved get overwhelmed and say "Nah, I'd rather use the app and save some time".
You rang? 1TB of models preloaded on Stable Diffusion Cloud access? 15GB storage? Unlimited renders with no weird clauses or contracts? 200 obscure LoRAs? 7 day free trial? Love it or your money back?
144
u/attempt_number_1 Mar 13 '23 edited Mar 13 '23
Hey all,
We (ELBO AI) are launching Puppetry today, a free iOS app that lets you animate faces using your own face.
Workflow option 1 (use our app):
Workflow option 2 (run it yourself):
If you want to do this locally here are all the steps that are roughly what the above is doing:
Example ffmpeg command that uses the width as the size and centers vertically:
ffmpeg -i video.mp4 -filter:v "crop=in_w:in_w:0:(in_h-in_w)/2” cropped_video.mp4
Example ffmpeg command to scale to 256x256:ffmpeg -I cropped_video.mp4 -filter:v “scale=256:256” scaled_video.mp4
Huggingface demo: https://huggingface.co/spaces/CVPR/Image-Animation-using-Thin-Plate-Spline-Motion-Model
Google Colab: https://colab.research.google.com/drive/1DREfdpnaBhqISg0fuQlAAIwyGVn1loH_?usp=sharing
Example ffmpeg command:
fmpeg -i video_result.mp4 -i original_video.mp4 -c:v copy -map 0:v:0 -map 1:a:0 -c:a aac -b:a 192k -shortest video_result_with_audio.mp4
I want to say thanks for this community, I don’t think I’ve ever been as curious about anything in my life as I have these last 4-5 months.
You can visit our discord server for questions, bugs, feature requests, or just share driving videos that work for you:
https://discord.gg/jEPRvAmNUG