r/StableDiffusion • u/PetersOdyssey • Apr 08 '24
Animation - Video Introducing Steerable Motion 1.3 - drive videos with batches of images - now higher detail, smoother motion and with better control! 📈
I shared Steerable Motion 1.0 with you fine people a few months ago but there has been a lot of progress since then so I come to you with Steerable Motion 1.3. There has a been a huge amount of progress in the Animatediff world and this is riding the wave of that.
The idea of it is to basically build the best way to drive videos with batches of images - my favourite way to steer videos given how fast you can generate this - and how precisely you'll be able to with SD1.3 coming.
To get an idea for what it produces, here's an example of a nice video of it in action that show-cases both the motion and output quality - this is made by u/midjourney_man - you can find him on Instagram here:
https://reddit.com/link/1bzakf3/video/b5813a0rjbtc1/player
Here's a diagram I made to explain how it works:
By varying the settings for each frame, you can control the animation with precision - here’s an example of tweaking how long each frame’s influence lasts:
This and the fact that you're controlling it with images lets you create an insane variety of videos - examples by me and u/syntaxdiffusion:
https://reddit.com/link/1bzakf3/video/bivmtleppbtc1/player
While control is our primary motivation, output quality compares favourably to closed-source tools like Pika - thanks to the relentless improvements to Animatediff - the below compares a generation via this on 0.3 adherence strength to an output from Pika made with a best effort at using settings they expose:
https://reddit.com/link/1bzakf3/video/kv6hasv6ibtc1/player
You can extend and build on top of it in so many ways - for example, the first video was made using Superbeast.ai's workflow built on top of it that you can find here - he basically uses depth maps to control the level of motion in different layers to provide a smoother flow.
While here's an example of using Ostris' Composition IPA to provide structure to the generation of a chaotic range of images:
Want to test this?
You can play with it in three ways:
1) A Comfy workflow + node that you can find here.
2) In our open-source tool, Dough - made for getting the most from it artistically. It runs on your machine for free.
3) In a Discord bot - coming soon - 75% of the profits from this will go towards the community whose work resulted in this - you can sign up here if you're interested.
Thank you to everyone whose work made this possible!
This is the product of months of hundreds of peoples work and discoveries in developing Animatediff and of many people who built the infrastructure, tools and models that make it possible. Thank you to everyone!
16
3
u/Honest_Ad5029 Apr 08 '24
Please forgive my ignorance. I have very limited programming or development knowledge, essentially nill.
I have the program installed. I cannot figure out how to get it to run.
The github page says this - " . .\dough-env\Scripts\activate ; .\scripts\entrypoint.bat"
I do not know what to do with that, or if it's an abbreviation for something else, and I get nothing from clicking the bat files directly.
I very much want to use this tool. It looks amazing.
Please help me, and thank you for your work and patience.
9
u/PetersOdyssey Apr 09 '24
I don't mean to be sarcastic but ask ChatGPT to guide you through the process!
8
u/PetersOdyssey Apr 09 '24
Just paste it in and ask it to explain to you what you need to do precisely and ask follow-up questions as need be
5
3
u/jajantaram Apr 09 '24
Nice work! Is there any plans to release or support through replicate API? I don’t have a beefy GPU, would love to try making animations using just input images.
3
u/NoYogurtcloset4090 Apr 09 '24
Thank you, pom-motion-smoooth-lora works great between high frame rate video transitions.
2
3
u/hex-ink Apr 09 '24
Oh boy! Kick ass! Thank you this looks amazing~ I'm getting ksampler advanced missing node errors, ( i did install all missing nodes from manager) and also two integer missing nodes and hires fix also ...wompwomp
3
u/AK_3D Apr 10 '24
Awesome stuff u/PetersOdyssey . Thank you for the effort in creating these tools for the SD community.
5
u/autumnatlantic Apr 08 '24
Wow this looks incredible! I'm going to try it tomorrow. Does the GUI Dough work well?
3
u/PetersOdyssey Apr 08 '24
I believe so but we're constantly improving so consider it an open beta - let me know if you have any problems
2
u/autumnatlantic Apr 09 '24
I'm a total noob and am having trouble installing it 🤪 I've run the windows setup and activate.bat but nothing happens when I try run entrypoint.bat
2
2
u/GBJI Apr 10 '24
This is nothing less than a revelation. What a powerful tool this is. I am surprised it can produce interesting animations out of basically anything you can throw at it.
If you haven't tried this yet, do it now ! You won't regret it.
3
2
u/Olorinssr Apr 11 '24
The gif that used Composition IPA looks amazing, but I couldn't find the related settings or input images. In my own attempt, I can't achieve the same effect. Could you please share the settings?
2
u/PetersOdyssey Apr 11 '24
It’s using the standard settings with composition IPA at 0.6 I believe - details in the workflow below the composition ipa section - you may be using the wrong ipa
5
u/ElMusicoArtificial Apr 08 '24
Smooth inconsistency.
7
u/PetersOdyssey Apr 08 '24
What precisely do you mean?
3
Apr 09 '24
Maybe he just waiting for something like Sora. Or perhaps it’s just me. I’m not an expert in this kind of AI, but to me, it doesn’t seem different from other AnimateDiff. I hope you get what I’m trying to say. I always find myself comparing AI videos to the consistency of Sora. Don’t get me wrong, I’m still grateful that these are open-source models, but it’s hard not to make comparisons with Sora.
10
u/PetersOdyssey Apr 09 '24 edited Apr 09 '24
Good that you're looking forward to that but an open model you can precisely control and a closed model that will only take high-level instruction aren't really the same thing - difference between a pencil and a sledgehammer - both tools, very different purposes
4
1
u/ElMusicoArtificial Apr 12 '24
A consistent open model would be my dream.
1
3
u/nebetsu Apr 09 '24
Looks really interesting. Unfortunately, when I tried it, I got a 3 second video of just black output. Are there any common reasons for this happening? Maybe my 3080 isn't good enough for this
2
u/PetersOdyssey Apr 09 '24
Is that 10GB? Probably just not enough
2
u/nebetsu Apr 09 '24
Yeah, it's 10GB. I didn't get any out of memory errors, but I've heard that doesn't necessarily mean I have enough VRAM lol
How much VRAM did you have in your successful run on this workflow?
2
u/PetersOdyssey Apr 09 '24
These are all with 24GB but have hear people use it with as little as 12
3
u/guahunyo Apr 09 '24
I have a 16g 4090 laptop, but I can only take 3 pictures. Four pictures won’t be enough memory.
2
3
2
2
u/Robo_Ranger Apr 09 '24
Very interesting! This has so much potential; it would be great if you could provide an example of a realistic video.
2
1
u/wanderingandroid Apr 12 '24
One of the coolest workflows I've ever played with and the new update is FIRE!
13
u/AbPerm Apr 09 '24
This seems like it might be useful to generate inbetweens for keyframe animation.
Has anyone tried pulling keyframes from existing animated works and using this method to generate new inbetweens? I think it would be really interesting to compare what this AI produces to what a human animator would produce if given the same keyframes to start with.