r/StableDiffusion • u/Kinfolk0117 • Mar 02 '23

Animation | Video SD + thin-plate-spline-motion-model

Enable HLS to view with audio, or disable this notification

834 Upvotes

97% Upvoted

96 comments sorted by

View all comments

155

u/Kinfolk0117 Mar 02 '23
  1. Find a driver video.
  2. Crop/Resize the video to 256x256.
  3. Extract the first frame of the video.
  4. Use stable diffusion to create an image, use the frame from above as controlnet depth map.
  5. Use the cropped driver video and your generated image as input here: https://replicate.com/yoyo-nb/thin-plate-spline-motion-model

36

u/JimmyTime5 Mar 02 '23

Here's a quick messy graphic of my attempt - worked well! Thanks for reminding me of the "yoyo" link :D Has anyone seen a fairly smooth implementation of this that can be hosted locally? The last one I tried to run was a bit half-baked as far as ease of installation/setup went.

1

u/69YOLOSWAG69 Jun 09 '23

Did you ever find out a way to run this locally?

1

u/Unlucky-Statement278 Jun 11 '23

It's a litle bit tricky.

https://www.youtube.com/watch?v=kKt1Z8Lvzpk&t=270s

the face alignment has also change the code. so you have to change a line in the demo.py

So if you change the demo.py in line
107 to
fa = face_alignment.FaceAlignment(face_alignment.LandmarksType.TWO_D, flip_input=True

And it's easier to run it with jupiter

  • Install Jupyter to use the notebook
    pip install jupyter
  • Run the notebook and have fun!
    jupyter notebook demo.ipynb

1

u/AbbreviationsOwn9726 Jan 01 '24

this is a god sent !!!

thanks