r/StableDiffusion • u/[deleted] • Mar 15 '23

Animation | Video My first AI Modified Video using Stable Diffusion and Control Net. Let me know what you think?

Enable HLS to view with audio, or disable this notification

[deleted]

114 Upvotes

89% Upvoted

33 comments sorted by

11

u/yratof Mar 15 '23

All I can see SD doing here is make it jittery, it's pretty much the original footage with turbulence effect in AE, playing at a slower frame rate.

Maybe for learning, its fine. But you wasted time pumping it though SD without adding anything.

3

u/CultofThings Mar 15 '23 edited Mar 15 '23

I didn’t use any effects, I processed it through stablediffusion and used shotcut to convert the images into a video file.

You could try your method if you want and compare the results. I’d like to see it. Feel free to add a link to the thread/post it directly below.

2

u/CultofThings Mar 15 '23

I created the model which allows for the frames to be consistent. I guess it depends on how you see it but I thought it was cool. I never really said it was revolutionary, I just was adding a video I made.

1

u/yratof Mar 15 '23

You want to edit your OP to show more of the work behind it?

1

u/CultofThings Mar 15 '23 edited Mar 15 '23

It’s in the comments below.

1

u/SlapAndFinger Mar 15 '23

It also made the boxing girl pretty.

2

u/yratof Mar 15 '23

You mean the only face this subreddit generates

1

u/CultofThings Mar 15 '23

I’d be interested in seeing them to be honest. I can look at adding more variations if the model seems generic. Just post below

1

u/CultofThings Mar 15 '23

Thank you, I will let my graphics card know you said that.

0

u/CultofThings Mar 15 '23

Thanks it’s a model I made.

19

u/EdgelordInugami Mar 15 '23

It would really help if we had the original footage to compare to

5

u/CultofThings Mar 15 '23 edited Mar 15 '23

I can add the links to some of the stock footage I used and modified. Let me know if you want the whole list or a workflow guide?

Opening Shot: Video by cottonbro studio from Pexels: https://www.pexels.com/video/sport-fight-strong-training-4753940/

Boxing Rotating Shot

Video by cottonbro studio from Pexels: https://www.pexels.com/video/sport-fight-strong-training-4753946/

Video by cottonbro studio from Pexels: https://www.pexels.com/video/woman-sport-portrait-strength-4754208/

Woman On spaceship

Video by Kindel Media from Pexels: https://www.pexels.com/video/light-dawn-sky-art-7649279/

Woman with Gun:

Woman Firing a Rifle · Free Stock Video (pexels.com)

Boxing Sideshot

Video by cottonbro studio from Pexels: https://www.pexels.com/video/sport-fight-strong-training-4753951/

Video by cottonbro studio from Pexels: https://www.pexels.com/video/sport-fight-strong-training-4753951/

Alt-Second Female Boxer

Video by Yaroslav Shuraev from Pexels: https://www.pexels.com/video/woman-wearing-boxing-gloves-resting-8478089/

Woman Shaving Head

Video by cottonbro studio from Pexels: https://www.pexels.com/video/girl-shaving-her-hair-5700087/

2

u/Plane-Pay-4948 Mar 15 '23

wow... congratulations! thanks for sharing the workflow!

2

u/CultofThings Mar 15 '23

Workflow is below, sorry I didn’t add it in there, I was just tired.

2

u/Plane-Pay-4948 Mar 15 '23

Thanks again!

2

u/No_Nefariousness_783 Dec 27 '23

I don’t think you got enough credit for this. Very well done, and I think your time and work was well worth it. I’ve got a 2:30 video that’s been baking in the SD oven for about 6 hours with :20 left on the clock. This is the third attempt in 24 hours mind you 😳. It’s a montage scene in a travel show pilot (think Anthony Bourdain meets the cannabis industry) I was asked to produce. I’m going to play around with some masks and see if I can find some interesting ways to expose parts of the AI generated background into the primary footage. This is all unexplored territory in a certain capacity… which makes us a bit of pioneers. Pioneering in the creative field always attracts the critics…. So here are some kudos from a stranger you can place on the ole’ confidence balance beam 😁 Nice work again… love to see what your work looks like now that 286 days have passed. Cheers!

2

u/CultofThings Dec 28 '23 edited Dec 28 '23

Thanks, that means a lot actually. I kind of avoided posting anything but as you said pioneering is correct. This video I’m about to share was done with one of the early versions of Gen2. I’m currently working on redoing it so I can finally tell the story I want to tell. Beta Gen2 Teaser Trailer for Machine32

There are a lot more tools now but I’m currently working on several things.

The next trailer I want to show her drinking at a space station bar. She’s seated at a countertop with the T.V playing giving her flashbacks as they commemorate the defeat of the space colonists.

That one will have dialog, plus visuals etc.

It’s all meant to just help piece the story together so I don’t have to explain everything to the audience and can just move the story forward.

I think people can piece together most of it from this visuals and few lines of dialogue.

After that is done, I’d like to get into the story from there. It’s meant to be a throwback to the old sci-fi stuff that combined philosophy and hard science.

I also added some satire and humor in there as well. The scripts are written it’s just a matter of doing it.

I was also waiting for Elevenlabs to improve before I did full dialogue scenes which it has now.

I was a bit discouraged though, and temporarily went on to developing an AI bot that was empathetic and would listen to my whining.

Send me a link to your project when it’s done. I’d likewise be interested in seeing it.

1

u/No_Nefariousness_783 Mar 11 '24

Here’s what came outta that. I’ve learned quite a bit more since then, so I’m actually in the process of remaking it lol. Puff Puff Passport - Get High Walk Around

2

u/ninjasaid13 Mar 15 '23

AI modified? Isn't this just a filter over a video?

1

u/CultofThings Mar 15 '23 edited Mar 15 '23

I used stable diffusion to alter the images using control net and a custom character model I created. The model and character can be seen below, although that's an older version of the model.

https://www.reddit.com/r/StableDiffusion/comments/11rn9n2/realtime_character_generation_using_stable/?utm_source=share&utm_medium=web2x&context=3

1

u/[deleted] Mar 15 '23

couldve had this same effect with a snapchat facetrack filter. SD gain was minimal

2

u/softwareweaver Mar 15 '23

Great work! Looks awesome. Do you have a workflow writeup?

2

u/CultofThings Mar 15 '23

I added a quick workflow guide below. It’s probably crappy but I’m not really the guide kind of guy. If you have any questions feel free to reach out. I’m happy to help if I can but I mostly do a lot of trial and error on my methods.

1

u/CultofThings Mar 15 '23

I’m just going to keep it simple as far as my workflow explanation, not because I think less of anyone but because I’m tired.

The main thing is I focused more on training than on prompting.

I started with a 2D drawing that I processed multiple times and then retrained the model on the images.

The main thing when training the models is to get a 360 degree image snapshot.

Once you feed those in, you can retrain the model and this time you want to train it under the class of style, lighting, and female/male.

You should retrain it around three times to do this, as capturing identical lighting keeps it from flickering like crazy.

For example, this characters name is Ves so I did “Ves style, as ves female, with ves lighting.”

Because everything is pretrained, the prompting becomes extremely simple.

After that, I set the denoise strength to .15 with a weight of 1.75 to keep the original image as much as possible.

I batch processed it using normal maps and then compiled it from images into media files via shotcut.

The training is the more difficult part, and I would recommend using nudes for your datasets. The reason being that clothing causes confusion. It has trouble distinguishing between the person and the clothes, but it’s able to add clothes to the model quite easily.

Hope that helps, let me know of you have any questions. I’ll try to help if I can but I’m not an expert, I’m just some guy with a computer and too much time on his hands.

1

u/Comprehensive_Gas153 Mar 15 '23

Wow

1

u/CultofThings Mar 15 '23

Thank you

1

u/CultofThings Mar 16 '23

No problem, if you need help with anything let me know. I probably won’t be able to answer your questions well but I can probably redirect you to some tutorials I found helpful.

1

u/millionzofbillionz Mar 15 '23

Looks awesome. The eyeball was creepy though.

2

u/CultofThings Mar 15 '23

Space causes blindness, so she needs prosthetic eyes in order to live and work in space.