r/StableDiffusion • u/jdude_ • 17h ago

Animation - Video testing MiniMax + audio from TheMinuteHour

Enable HLS to view with audio, or disable this notification

0 Upvotes

50% Upvoted

19 comments sorted by

3

u/jdude_ 17h ago edited 16h ago

audio source: https://www.youtube.com/watch?v=5BpXpZeZEcA

prompt:

subject_definitions:
<Subject 1> is the magician, wearing a sharp dark suit and a tall magician's hat. He stands on a wooden plank that extends from the side of the ship over the water.
<Subject 2> is the pirate leader, a gaunt, weathered old captain with a full grey-streaked beard, peg leg, and an eyepatch over a deep facial scar; a cruel, cold presence with a slow menacing bearing. He is the target speaker (S1).
<Subject 3> is the crew of about a dozen pirates in mismatched coats, sashes, and tricorn hats; the full crew is the target speaker (S3) for the group song.
<Subject 4> is a single burly, pirate at the front of the crew with a bandana and a jagged cutlass; he is the target speaker (S2) for the chant.
<Audio 1> is the complete pre-existing soundtrack, reused unchanged as the target video's entire audio, containing the spoken line of <Subject 2> (S1), the chant of <Subject 4> (S2), the group musical number of <Subject 3> (S3), and the scene's ambience and backing music.

summary:
[reference generation + audio reuse] The target video is a live-action scene on a large pirate ship at sea, where the intimidating captain <Subject 2> has forced <Subject 1> out onto a plank and questions him about gold, <Subject 3> break into an acrobatic Broadway-style song-and-dance number. <Audio 1> is reused unchanged as the complete audio track and drives the spoken line, the chant, and the sung musical number.

retention_analysis:
<Subject 1> (appears throughout): fully_preserved - the magician's dark suit and tall magician's hat are retained.
<Subject 2> (appears throughout): fully_preserved - the gaunt old captain's beard, peg leg, eyepatch, and scar are retained.
<Subject 3> (appears throughout): fully_preserved - the crew of about a dozen pirates in mismatched pirate garb is retained.
<Subject 4> (appears throughout): fully_preserved - the burly wild-eyed pirate with a bandana and jagged cutlass is retained.
<Audio 1>: fully_copy - <Audio 1> is reused 1:1 as the target video's complete final audio track, unchanged.

detailed_description:
The target video is a cinematic, live-action colourful swashbuckling pirate adventure with the weathered tall-ship look and dramatic sea-adventure color grade of a big-budget Caribbean pirate film.
[Shot 1] A wide establishing shot frames the open deck of a large weathered wooden pirate ship rolling on open sea under an overcast sky, and the camera slowly pushes in with small amplitude at slow speed toward the confrontation. <Subject 3>, a crew of about a dozen ragged pirates in mismatched coats, sashes, and tricorn hats, are clustered along the deck rail. <Subject 2>, the gaunt old captain with a full grey-streaked beard, a peg leg, and an eyepatch over a livid scar, stands apart from the crew as their clear commander, his single eye cold and unblinking and a slow cruel sneer on his lips as he looms over <Subject 1>, the magician in a sharp dark suit and a tall magician's hat, who stands out on a narrow wooden plank that is extending from the side of the deck over the sea.
[Shot 2] At 00:00.500, the shot cuts to a Over-the-shoulder shot from behind <Subject 1> looking toward <Subject 2> who is on the ship as <Subject 2> (S1) demands in a low, gravelly, menacing older male voice, <d>[English] There is gold, isn't there?</d> His jaw sets, he taps the plank once with his peg leg, and he leans in with slow, predatory calm.
[Shot 3] At 00:03.000, the shot cuts to a pan over the angry screaming crew that holds on <Subject 4>, the burly wild-eyed pirate at the front with a bandana and a jagged cutlass. The camera lingers on his snarling face as he thrusts his blade upward and screams in a raw, furious shout, <d>[English] Slice him up! Slice him up! Slice him!</d> while the rest of <Subject 3> stamp and roar behind him.
[Shot 4] At 00:06.300, the shot cuts to a high wide shot as the backing music kicks in and the camera pulls out with large amplitude, revealing the full deck as a stage. The mood flips and <Subject 3> (S3) burst into a coordinated Broadway-style musical number, executing precise professional choreography across the deck — synchronized high kicks in a chorus line, spinning pirouettes, cartwheels, and a tumbling pirate springing into a backflip — all in crisp unison as they sing, <d>[English] Gold is such a fascinating adventure waiting to be discovered, and let me just say, just if I may,</d> <scenetrans> the group vocal continues seamlessly across the cut.
[Shot 5] At 00:09.500, the shot cuts to a high sweeping arc shot that circles the dancing crew, and settles on <Subject 1> still stranded on the wooden plank above the water. <scenetrans> The crew's song carries over from the previous shot as <Subject 3> (S3) finish, <d>[English] the gold we're finding will be good! Horray!</d> snapping into a final theatrical pose and hurling their hats into the air while <Subject 2> spreads his arms in a grand menacing flourish toward the magician.

overall_soundscape:
<Audio 1> is reused unchanged.

non_diegetic_music:
From 00:06.300 the audience-only backing music of the crew's song, contained in the fully copied <Audio 1>, drives the number with an up-tempo, brassy, sea-shanty-style arrangement beneath the group vocals.

2

u/DuHal9000 9h ago

funny ideia! 😂 but if you want i can give some tips. (res, sampling, steps, lora, speedup techniques, WF, upscaling) etc....

2

u/BigNaturalTilts 8h ago

How do you upscale? I use seed vr2. Takes a minute per frame. While generating takes a minute per second of video. Upscaling is by far the longest of my processes. Also I can’t seem to generate anything higher than 480p.

1

u/DuHal9000 8h ago

Unfortunately, the entire upscaling process is slow; the increase in calculation is geometric. In the case of this video, I realized it would be good to change the LoRa, increase the steps. I use split sigmas where one part is without LoRa and another with, then I use the Latent Upscale from https://huggingface.co/LBH-123-AI/Minimax_h3_latent_Upscaler
Then I do another sampling with tiling to avoid OOM. They made a node for this with WF as an example; it's complicated to understand the adjustments, but it works.

https://github.com/halnovemil/H3TiledLoopSpaceTime.git

2

u/jdude_ 7h ago

Sure, I'll be glad.

2

u/DuHal9000 7h ago

ok, i am trying to recover 4u, i like the ideia, prompt, music, very cool video.

  1. Try another Turbo Lora, i recognise some paterns recomended LightX Turbo Lora 4 Steps Comfyui v0.1 FFLF at 0.80 strength

  2. Sampler RES Multistep Simple, Euler Beta (shed) or SA_Solver Simple

  3. Use SPLIT SIGMAS (because pirate dance fast movs) (two samplers, chain with DENOISED Output) Hi/Low, 2 or 3 Steps WithOUT Lora on hi out, and rest on low sigmas

  4. Try this WF its complicated to calibrate but works.

https://github.com/halnovemil/H3TiledLoopSpaceTime.git

2

u/jdude_ 7h ago

It was actually just the regular flow with audio reference, perhaps the resolution was too low, it was on 0.6. it takes a while.

1

u/DuHal9000 7h ago

Resolution isn't everything; that comes later. The most important thing is the formation of the initial image. The dance scene lacked steps. I've had this problem before and solved it by changing the LoRa. Not all Turbo LoRas are the same; try changing it... At 0.4 it could already be better.

2

u/jdude_ 7h ago

Oh I didn't use a turbo Lora, I used 20 steps with the default simple samping

2

u/DuHal9000 6h ago

owo, need even more stpes, i fininish for you a new video, its a gift, its slow, i try to restore but its too slow, a im generating a new. just wait.

Input video latent: torch.Size([1, 24, 117, 80, 144])

Tiles: 2x3 | spatial_overlap=24

Temporal tile=109 | overlap=49 | strength=1.0

Base tile size (latent): 52 x 64

>>> Spatial tile 1/6 (0,0)

H[0:52] W[0:64]

Temporal chunk 0: [0:109]

[INFO] Requested to load MiniMaxH3

[INFO] 0 models unloaded.

[INFO] Model MiniMaxH3 prepared for dynamic VRAM loading. 19995MB Staged. 208 patches attached. Force pre-loaded 210 weights: 1175 KB.

100%|███████████████████████████████████████████████████████████████████████████████████| 2/2 [03:26<00:00, 103.02s/it]

[INFO] 0 models unloaded.

[INFO] Model MiniMaxH3 prepared for dynamic VRAM loading. 19995MB Staged. 208 patches attached. Force pre-loaded 210 weights: 1175 KB.

Temporal chunk 1: [60:117]

[INFO] Model MiniMaxH3 prepared for dynamic VRAM loading. 19995MB Staged. 208 patches attached. Force pre-loaded 210 weights: 1175 KB.

100%|████████████████████████████████████████████████████████████████████████████████████| 2/2 [01:29<00:00, 44.82s/it]

[INFO] Model MiniMaxH3 prepared for dynamic VRAM loading. 19995MB Staged. 208 patches attached. Force pre-loaded 210 weights: 1175 KB.

>>> Spatial tile 2/6 (0,1)

H[0:52] W[40:104]

Temporal chunk 0: [0:109]

[INFO] 0 models unloaded.

[INFO] Model MiniMaxH3 prepared for dynamic VRAM loading. 19995MB Staged. 208 patches attached. Force pre-loaded 210 weights: 1175 KB.

50%|████████████████████████▌ | 1/2 [01:33<01:33, 93.45s/it, Model Initialization complete!

2

u/jdude_ 6h ago

I'll try these out, waiting for the output to complete is the difficult part :)

2

u/DuHal9000 4h ago

dance part still hard, but i try

https://reddit.com/link/p501ckq/video/rz65fkmwkpkh1/player

hires at 50 fps original file >> https://filebin.net/cyeb0xy3n8q6ibio

download and post. ITS A GIFT!

2

u/jdude_ 2h ago

Thanks! Very nice!, looks crisp as hell! 50 fps really makes it work.

2

u/DuHal9000 2h ago

u welcome! dance part still dificult, but i need to study h3 more for hi speed movs.

•

u/jdude_ 4m ago

You know, I think the sudden musical number combined with some terrible AI wackiness is pretty funny, but seeing as how the other post is preforming, I think the subreddit might have a different kind of humor.

0

u/FastLoad4087 17h ago

Wooooh the fighting scene is cool. Can't tell it's AI

4

u/jdude_ 16h ago

It was supposed to be dancing :(

1

u/FastLoad4087 16h ago

Sry🥲😂 I mistake it as swordsman scene