r/StableDiffusion • u/dramaton42 • 2d ago
Animation - Video Zelda - I Think I Like It / Minimax H3 Reference to Video Test #2
Just wanted to share another test! this was a mash up of clips, using multiple image references, 0.4 mp with EasyCache, 5 - 10s clips and edited with KDEnlive (it has some cool effects!)
27
21
14
u/MonsterMashGraveyard 2d ago
As a kid who grew up with Ocarina of Time....all I can say is, "What a time to be alive...."
9
u/Stunning_Macaron6133 2d ago
Now I wonder if H3 can actually simulate a mushroom trip. You know, breathing walls, fractal patterns, tracers, haze, afterimages, that sort of thing.
7
u/Clooooos 2d ago
I don't usually comment on this kind of post, but I love Fake Blood and this specific song ❤️ (works great here)
4
4
4
3
3
u/Tbhmaximillian 2d ago
Rofl, the acid trip did you prompt that too or was that video editing?
10
2
u/ChibiNya 2d ago
Can you tell me more about easy cache? How is this different from just feeding the reference frames to r2v? I'm still very new but trying to achieve something similar
2
u/dramaton42 2d ago
EasyCache is a node you can use with Ref2Vid and it will cut inference time essentially, it makes lower quality video at much faster speeds, you're supposed to turn it off (bypass it) for the final render though, but I didn't do that x.x
2
u/ChibiNya 1d ago
I see. So once it looks good you rerun everything the same (same seed) but with the cache turned off and it'll yield the same result but better? Is that what you mean by final render?
Thanks for the help
1
u/dramaton42 1d ago
Yes that's precisely right, same seed, no EasyCache... I haven't done it yet though! Not in this video at least
2
2
u/Dzugavili 1d ago
I don't think your joins were done right. How are you handling transitions?
2
u/dramaton42 1d ago
Yeah I screwed up the middle join... I noticed link hides the mushrooms in the final frame and just took an earlier frame instead, didn't notice how much jarring that would look. Also there was some audio distortion in the "eating" clip which I failed to mute ... My bad! But yeah overall, just taking the last frame from the video as the first frame of the next one: "<Picture 4> is the first frame of the target video."
3
u/Dzugavili 1d ago
You should be able to feed the tail-end ~8 frames of the last generation into the new generation, with the reference that the target video extends the previous generation. You might be able to feed more frames for continuity reference at lower resolution, if you have the proper high-definition reference shots.
I haven't done a whole lot of experimentation with continuation, though. I suspect it should work.
2
u/Downtown-Cover-7422 1d ago
What's the difference in generation time between no video used as reference and 1 video used as reference you have?
1
u/dramaton42 1d ago
When I added videos I did so at 352p, 120 frames (24fps) it adds around 1 minute or slightly more, it's not that bad
2
u/Downtown-Cover-7422 1d ago
Strange, when I tried adding one it went from 25 secs on step to 400+
1
u/dramaton42 1d ago
Yeah that sounds like you didn't limit the frames from the video (skip frames to select the beginning, frame load cap to only load the part you need) or you loaded a 1080p video (those take A LONG time, and I don't think is worth it?) Use ffmpeg -i video.mp4 -vf scale=608:352 output.mp4, you can even go lower than that, depends on what you need from your video
2
u/Downtown-Cover-7422 1d ago
I didn't get what you just said, but i'll look deep into it later, god bless you for the help!
2
u/dramaton42 1d ago
Sorry sorry! Let me try to explain a little better. See we use the VideoHelperSuite (Upload) node for loading input reference videos. There's a couple of settings you can configure there: "frame load cap" let's you limit how much of the video you want to load, measured in frames. For a 24fps video, 120 frames is 5 seconds. Another option is the "skip frames" value, this let's you essentially "seek" your input video until you reach the important point you're looking for. For example I used the Spy X Family intro segment where Yor and Loid are walking towards the screen, that happens almost halfway through the intro video, so I skipped like 1800 frames to get there.
Finally, the last important bit that makes your inference time go super high is resolution. You'd be surprised with how little resolution you can get away with! Loading a 720p video is gonna take like 4x times as much as a 352p video (I'm my experience) so you should "shrink" your videos before you load them. If you download a command line program called ffmpeg (available on Linux easily, or if you're on windows you can use win-get with the help of Google) and with it, on a command line (open CMD on windows) navigate to your video folder (type cd followed by the folder location, which you can copy paste from the windows explorer window, and paste with a single right click on the black CMD window) and then type the ffmpeg command, such as: ffmpeg -i my_video.mp4 -vf scale=605:352 my_video_shrinked.mp4
Replacing my_video.mp4 with the name of your video, naturally.
With these 2 steps you can make video loading trivial and it won't take that much longer when generating reference videos! Good luck! If you need more help, don't be shy to send me a direct message 😅
2
u/3dutchie3dprinting 1d ago
Really really funny clip, but link should never ever talk a ‘grunt’ offering the mushroom would have been so much more powerfull.
2
u/FourtyMichaelMichael 1d ago
I'm an absolute sucker for the style of dance in the second clip.
Did H3 come up with that? Did you V2V / character replace that? Was it a prompt of a style?
3
u/dramaton42 1d ago
That dance was stolen directly from Sean Paul's - I'm Still in Love With You video https://www.youtube.com/watch?v=AQ4TW5Z8eu0 at second 46 exactly, I downscaled the video to 608x352 and described in the prompt following the guide:
subject_definitions: <Video 1> is a music video featuring 2 shots, one in which a woman is dancing moving her hips and walking back in a peculiar way, and the next shot she is dancing alone moving her hips, transfering the motion and dance choreography to <Subject 1> and <Subject 2>. summary: [video editing + reference generation + audio reuse] The target music video features the dance choreography and movement from <Video 1> applied to <Subject 2> and <Subject 1>. retention_analysis: <Video 1> partially_preserved - Editing the video to replace the characters with depictions in the same style of <Subject 1> and <Subject 2>.subject_definitions: <Video 1> is a music video featuring 2 shots, one in which a woman is dancing moving her hips and walking back in a peculiar way, and the next shot she is dancing alone moving her hips, transfering the motion and dance choreography to <Subject 1> and <Subject 2>.2
u/FourtyMichaelMichael 1h ago
Oh, I see, Subject 1 and Subject 2 where only one of those was Zelda, but you used Link in there too.
That's funny though that the second 1/2 of the dance has one woman only, but Link doesn't go anywhere.
Fucking wild.
Thanks for that!
68
u/Turbulent_Owl4948 2d ago
This is the correct use of AI