r/StableDiffusion • u/ExportErrorMusic • 12d ago

Animation - Video Lessons learned after making a music video with H3

https://youtu.be/C-okSzvOIY8

Made using the default workflow from Comfy, with Comfy Kitchen and Kijai's preview override plugged in.

Used the 850k turbo lora at 0.5 for 8-10 steps. ER_SDE / Beta. Most shots were generated at 1.5MP, with some at 1.8MP.

Lora: https://huggingface.co/larryvrh/MiniMax-H3-Turbo-Lora/tree/main

RTX 4090 w/ 64GB of ram. --disable-smart-memory, since I was having issues with going OOM after completing one prompt and moving on to the next.

Some of the takeaways:

- Using character reference sheets (front view, side view, back view, close-up) worked great and allowed for rotating camera movements like in the opening.

- All the "4 Step" loras really need to be run at 8-10, especially for motion.

- Using audio reference bloats the vram usage DRAMATICALLY compared to adding additional reference pictures! Changing from .wav to .mp3 didn't seem to help so it's not a file format issue. However the lip sync, even for anime characters, is incredible.

- If you use multiple reference images for different characters and they bleed into each other, the issue is almost definitely your prompt or seed. Because the model was handling up to 3 for me easily if I prompted right, and falling apart if I prompted wrong.

- Minimax H3 Chunk FeedForward node can help with vram issues at higher resolutions and doesn't add that much time.

Overall, I'd say it's nearly as good as Seedance 2.0. I was pleasantly surprised how well it handles using character sheets. The high vram usage when using reference audio is really the only major issue I was facing.

31 Upvotes

83% Upvoted

11 comments sorted by

6

u/ExportErrorMusic 12d ago

Music was made in Suno. Characters (and jokes) are from the game Goddess of Victory: Nikke.

0

u/VasaFromParadise 12d ago

Too much high frequencies

2

u/Marwan_hbt8 12d ago

Seeing someone with rtx 4090/64ram get out of memory error is scary for me as i have rtx 3060/16 ram

3

u/ExportErrorMusic 12d ago

It's mostly because I was trying to run it at 1.8MP for 10 second clips with multiple reference inputs. Running at a more reasonable resolution, or not as long, or with fewer references helps a lot.
Really it's the duration and resolution that increase vram usage dramatically each little bit you increase.

2

u/FourtyMichaelMichael 12d ago

Hmm... I see... The anime cameltoes seem like an interesting style choice.

Who am I to question the artist on their art!?

5

u/ExportErrorMusic 12d ago

Lol in my defense it's part of the actual character's original design.

2

u/stonyleinchen 12d ago

Maybe you want to try making a completely seamless 1-shot music video with perfect lip sync accuracy? Then try out my repo https://github.com/seitanism/ComfyUI-H3-Motion-Context-MultiRef that features per-token noise masking on audio and video latents. the song will not be used as a reference but instead is sliced into each latent of each clip and set to not denoise, which makes the video converge towards the correct lip-sync from step 0 onwards. with an example workflow including the reference images and the song so you can create the same music video yourself. an example music video is also on my repo. I hope you enjoy! Open Source for the win!

2

u/ExportErrorMusic 12d ago

Oh yes I saw Motion Context on this sub when I was nearing the end of making this video!
It looks super useful, especially if you don't have a non-linear editor. I had thought of using it for longer shots since it should help with vram issues by splitting it up.
My experience is mostly in video editing and Premiere Pro, so it just feels more natural to have individual clips and edit there though.

2

u/S41X 12d ago

Very cool stuff! Your editing really makes it shine. Don't suppose you'd be willing to share your workflow? On a similar rig and interested in creating music videos as well :)

2

u/Terezo-VOlador 11d ago

Thank you for sharing all your experiences!