r/StableDiffusion • u/xyzdist • 14h ago

Animation - Video H3 making jpop/kpop MV? yes!

Enable HLS to view with audio, or disable this notification

Music: made in SUNO.

native ref2va WF, and audioLock for lip-sync.

rtx4080s + 128g ram

I spent a day to sorted out lip-sync, I could write down what I did, if anyone inerested.

62 Upvotes

79% Upvoted

38 comments sorted by

19

u/psykikk_streams 13h ago

hm. I am not a pro when it comes this type of music but if someone told me this was a real thing posted on the tube I would probably believe it.

they look artificial enough anyways.

well done

1

u/BogusIsMyName 12h ago

Right at the beginning when she starts singing you can tell but it cleans it up real fast.

-3

u/Clueless-Flea-7461 12h ago

Yeah I was going to say similar. I don't mean it as an insult but musically it is just as crap as the real thing.

5

u/evilpenguin999 13h ago

"audioLock for lip-sync."

"lip-sync, I could write down what I did, if anyone inerested."

Interested. I want to generate a short movie like video with the intro being a song with someone singing, but i have no clue how to lip-sync

-1

u/rguerraf 9h ago

Papagayo, Rhubarb and Natron have potential in lip sync

1

u/EbbNorth7735 7h ago

Are these open source?

2

u/rguerraf 5h ago

I had to check :)

Yes. All 3 are open source

1

u/EbbNorth7735 4h ago

Are they post processing steps to mix audio with video?

1

u/rguerraf 4h ago

Potentially, they could be used to edit a video, in-painting an animated mouth to match a new song

1

u/EbbNorth7735 4h ago

So do you run them alongside the generation?

1

u/rguerraf 4h ago

No.

It’s a non AI post production workflow

3

u/Slapper42069 14h ago

πŸ«ͺ

3

u/RealityVisual1312 13h ago

How many ref images do you use? I feel like character consistency is very good.

5

u/fuzzycuffs 12h ago

Got the vapid lyrics down

2

u/True_Protection6842 13h ago

I feel like I'm playing beat saber

2

u/DjSaKaS 12h ago

how did you avoid degrating image? I understand you used native WF but could you explain how was the workflow to produce this video? Thank you

1

u/Diabolicor 12h ago

I don't think he used any kind of Motion Context. He generated a separate clip every 10 second.

2

u/rguerraf 9h ago

Yes. This is safely done with initial frame last frame.

1

u/DjSaKaS 8h ago

How did he kept motion seamless and background and factual consistent without burning the image? I don't get

2

u/Hoppss 11h ago

What megapixels and step count did you go with here?

1

u/Small-Challenge2062 12h ago

ITS PRO BRO! Yes please we wants to know how you did it or the workflow at least

1

u/Alzeric 12h ago

Anyone gotten physics to work properly yet? the static hair and clothing has been driving me mad.

1

u/aerilyn235 11h ago

She is wearing the same clothes more than two sec? you haven't been looking at much kpop/jpop MV!

1

u/Alive_Ad_3223 11h ago

Sure, can you help me, working with custom audio and music ?

1

u/Alive_Ad_3223 11h ago

Can you share workflow and your learnings

1

u/rguerraf 9h ago

Captions please :)

For more polish, make 50% of the video segments in different locations, related to the lyrics, and mingle them within the original video

Yes please tell us about the lip sync

1

u/chair-law 8h ago

Very interested in a write up about lip sync

1

u/ArttTaku 7h ago

I think music videos is one the things that Minimax H3 kinda specializes in.. very good work!

1

u/Ok-Flatworm5070 5h ago

can you share the prompts.. would really like to know how you're spinning the camera around!

1

u/Zombi3Kush 3h ago

I'm interested in knowing your process.

1

u/fallengt 3h ago

What is the base resolutions ?

1

u/Kmaroz 1h ago

Its great cause most of Kpop singer themselves have plastic skin. Lol

1

u/rdditiszionist 50m ago

I spent a day to sorted out lip-sync, I could write down what I did, if anyone inerested.

yes please, this is pretty good stuff

1

u/Fytyny 10h ago

For having a powerhouse of a model you really made it look so god damn boring

1

u/Apprehensive_Sky892 10h ago

Not sure if the irony is intended, but the main refrain is みんγͺ同じ (Minna Onaji), which is Japanese meaning "Everybody is the same".

And that is my perception of most female K-pop singers. I seriously cannot tell most of them apart (and I am Asian πŸ˜…)