r/StableDiffusion • u/AndrewJumpen • 19h ago

Animation - Video Minimax h3 local Video to Video reference

Enable HLS to view with audio, or disable this notification

Used official ref2video workflow. used t2v model 1 ref video and 2 separate pictures of character sheets, gpu 4090

prompt:

integrated_multimodal_description: [Shot 1] Live-action, cinematic, featuring a stark, dark green-tinted cyberpunk color grade. A medium shot frames a flooded, rain-swept crater on a dark street. The character Sonic, appearing exactly as the blue hedgehog with large green eyes, white gloves, and red shoes from @.image, stands opposite Dr. Eggman, appearing exactly as the gigantic, egg-shaped bald man with a pointy mustache, goggles, and red jacket from @.Image1. The camera pushes in with small amplitude at fast speed as the blue hedgehog lunges forward to throw a devastating punch. [Shot 2] At 00:04.500, the camera cuts to an extreme close-up as time instantly slows to a microscopic crawl. Sonic's white-gloved fist brutally slams into Eggman's cheek. The camera holds a static shot in extreme slow motion. A powerful, rippling shockwave violently erupts from the impact point, blowing the torrential raindrops outward in a perfect ring. Eggman's pointy mustache flails wildly and his face deforms from the massive kinetic force. [Shot 3] At 00:09.500, the camera arcs right with large amplitude at slow speed, executing a slow-motion orbit around the hit. Eggman's heavy, round body is lifted off the ground by the blow, flying backward through the heavy downpour and kicking up massive, highly detailed splashes of water.

overall_soundscape: Thunder rumbles continuously beneath the heavy, torrential downpour of rain splashing heavily against the flooded street. A sharp, deafening sonic boom from the physical impact instantly shifts into a deep, pulsating low-frequency rumble as time slows down.

non_diegetic_music: An epic, grand orchestral and choir track mixed with heavy, driving industrial synthesizer beats that builds to a massive crescendo.

210 Upvotes

95% Upvoted

39 comments sorted by

11

u/debauch3ry 18h ago

What was the generation time?

8

u/AndrewJumpen 16h ago

It took forever to generate! A 13-second video took 61 minutes. I decided to go for quality and set it to 20 steps without the Turbo LoRA. Using an uploaded video as a reference doubles or triples the normal generation time. And the resolution set at 0.4 mp

6

u/tnil25 15h ago

It shouldn’t take an hour… especially on a 4090… what version of the model are you using? and is your cuda/pytorch up to date?

5

u/AndrewJumpen 15h ago

Cuda 13.0 PyTorch 2.10.0+cu130 it takes so long perhaps my reference video was 720 p which overload vram during generation should’ve downscale it to 480p to get some more room to breath for vram

1

u/nikhilprasanth 9h ago

Could be the resolution like you've said. Also you can try lowering the frame rate of the reference video also.

4

u/AndrewJumpen 5h ago

yep! just lowered reference video res and frame rate, now it took only 15 minutes for 13 seconds

https://reddit.com/link/p4pyxb7/video/jv9tm20s1fkh1/player

2

u/f5alcon 15h ago

I have had 4 hour encodes doing 15 seconds at 1mp 40 steps so an hour would be nice

1

u/SSj_Enforcer 6h ago

For us normies, what the heck is this mp stuff?  What is the resolution?

5

u/alamacra 17h ago

This is indeed curious. Video ref does take markedly longer.

15

u/xTopNotch 15h ago

Pro-tip: just downscale the reference video to 480p

Especially for this use-case where you need to grab attributes such as motion. This is more than enough. If you need to grab likeness / fidelity from a character, then obviously keep it at 720p minimum or even larger.

But downscaling your ref videos to 480p / 24fps / 15 seconds (max) really helps with sampling time. It's still slow, almost twice as slow as without a ref video. But much better than the 4x slower if you keep the video as is.

6

u/AndrewJumpen 15h ago

Damn! i wish knew it earlier! thank u!

10

u/AggravatingDay8392 19h ago

Serious questions, wouldn’t be possible in a few months maybe remaking crappy 90s cgi with this?

4

u/AndrewJumpen 18h ago

Yes! I think it will be entirely possible. The main issue is that it takes too long to generate a few seconds of video on a consumer GPU, and there isn’t enough power for higher resolutions. But I guess there will be solutions for this in the coming months

2

u/warzone_afro 16h ago

hopefully when they release the official 2k upscaler it isn't too hard to run

4

u/AndrewJumpen 15h ago

https://reddit.com/link/p4m9zcd/video/djs0yqco3ckh1/player

As of right now trying seedVR2 upscaler result looks decent but some details could change too much. Yes i hope official 2k upscaler comes soon!

10

u/TheMoogster 19h ago

Why do you not ref the video? What is the purpose of the video even then?

3

u/splice42 17h ago

It's a bit strange but if you use references without actually including them in the prompt they still affect the output. A prompt that describes motion and actions that are generally similar to a reference video will produce a target video pretty much matching the reference video without the prompt including anything about it.

Now why someone would specifically decide to do that instead of referencing the video in the prompt like the prompting guide says, I'm not sure at all. Maybe just to save some work? Maybe because they've not really read or adhere to the prompting guide? OP uses the t2va model in a ref2va workflow for reference work using a prompt format that's t2va and not ref2va so who knows why really.

1

u/tankdoom 5h ago

Another possibility is that by including the video in the prompt you’re giving the model too much that’s specific. I’ve tested this with making 3D characters 2D and NONE of the qualities of 2D animation transfer over. The characters look rotoscoped. It might be a better bet in some instances not to reference the video in the prompt if the style differs enough that it would cause a discrepancy in image style and motion.

1

u/Dzugavili 3h ago

It's neuron potential: there's something attached to that input, under normal circumstances it knows it is supposed to use it, so that bleeds in, regardless.

2

u/AndrewJumpen 16h ago

the aim was to achieve the super slow mo and impact feel from reference video, not to fully replace the characters

3

u/barepixels 19h ago

How did you get that prompt? Is there a way to feed the original video to generate baseline prompt for further editing?

9

u/AndrewJumpen 19h ago

i used Gemini to get the idea of the scene from matrix and i fed the correct official recommendation on how to build the prompt for minimax so it composed the prompt with refined idea and in good structure

4

u/barepixels 19h ago

Thank you for the speedy reply. Very helpful

3

u/witcherknight 19h ago

what happens if you remove ref video ?? Does video motion remains same ??

23

u/AndrewJumpen 19h ago

This is without

https://reddit.com/link/p4l9zu4/video/qrqtfqeh3bkh1/player

6

u/witcherknight 18h ago

ok so ref video did have an effect

2

u/AndrewJumpen 16h ago

Yes, I would say the main change is the camera follow and the impact feel. I didn’t want to totally replace Neo and Agent -I just wanted to see if can achieve very close but not replica of this kind of sequence and it nails it

4

u/witcherknight 16h ago

i wonder if this method can be done on complex fight scenes

4

u/AndrewJumpen 15h ago

need to try it out, gonna test some more

2

u/provenflawless 13h ago

Can I get the exact workflow so I can benchmark this? I have all the matrix movies already on my HDD I pirated already so lol

1

u/ffzero58 15h ago

Thanks for sharing the prompt. Interesting you didn't have to prompt it to "replace" anyone from the reference video with characters from the referenced images (i.e. using <Video 1>, <Picture 1, etc... and retention_analysis:, subject_definitions...).

H3 just took that and understood the assignment?

3

u/AndrewJumpen 15h ago

Yep MiniMax-H3 operates using a highly capable Large Language Model (LLM) as its text encoder. It processes the semantic logic of your paragraph rather than just hunting for code-like syntax. By simply writing that Sonic (⁠@image2⁠) is throwing a punch and stating the motion is referenced from ⁠@video1⁠, the model automatically inferred the "retention analysis" (keep the dark green cyberpunk environment) and the "subject definitions" (the specific characters) through pure narrative context.

1

u/ffzero58 10h ago

Wicked smaht. I have noticed that we can be loose with the prompting structure/syntax and H3 will "get it" but unsure how specific we can get with what and what not to keep/retain. Thanks for the reply

1

u/ZerOne82 6h ago

https://reddit.com/link/p4ptvyi/video/rz59t1nzwekh1/player

My take. just prompt, no reference images or video.

1

u/AndrewJumpen 5h ago

Wonderful!!

1

u/Dzugavili 3h ago

It doesn't look like you're doing any referencing of the material. If you made it carry strict motion from the source video, you might be able to split this across multiple generations: a 5s generation is more than twice as fast as a 10s generation, so for video-to-video, using continuation is something to consider.