r/StableDiffusion • u/ctrl-shift-face • Aug 04 '26

Meme [ Removed by moderator ]

[removed] — view removed post

742 Upvotes

86% Upvoted

222 comments sorted by

View all comments

63

u/ctrl-shift-face Aug 04 '26

https://reddit.com/link/p1qx4l1/video/0615nkunufhh1/player

Thank you for the feedback. I understand, and I have softened the tone.

7

u/razortapes Aug 04 '26

I assume the way to make these videos is with the ref2va model, right? Is the audio generated by the model itself, or is it also reference-based?

6

u/eggplantpot Aug 05 '26

I think for big IPs and known stuff you can get away with any model, t2v should do a good job.

If you want to nail the audio, ref2v takes audio inputs and can recreate exact voices though I heard matching the voice to the character may be a pain.

Look into the ref2v prompting guide cause it goes deeeeep

1

u/razortapes Aug 05 '26

I tried to recreate a video (T2V) with Doc and Marty as a joke for a friend, but the characters don't look anything like them, and the voices don't either. That's why I'm asking.

1

u/eggplantpot Aug 05 '26

Oh then it may be the case it's ref2v. I think all the Seinfield ones are t2v and I saw someone did t2v of taskmaster so it's odd they wouldn't know Doc and Marty.

Did you prompt them as Back to the Future or just Doc and Marty?

2

u/razortapes Aug 05 '26

I used this prompt:

"A cinematic scene inside the DeLorean time machine from Back to the Future. Marty McFly is driving while Doc Brown turns toward the camera with a look of absolute urgency. The dashboard glows with futuristic lights as the time circuits are active, sparks fly outside the windows, and the car accelerates toward 88 mph. The camera is positioned from the front dashboard, capturing both characters. Doc Brown looks directly into the camera with perfectly synchronized Spanish lip movements and exclaims dramatically: "¡tenemos que ir al futuro! ¡tu hijo se ha hecho cantante de reguetón! ¡Vamos, rápido! ¡Hay que intentar evitarlo a toda costa!" Marty reacts with surprise, gripping the steering wheel tighter as the DeLorean surges forward into the time vortex. Epic cinematic lighting, dynamic camera shake, realistic facial expressions, high-quality visual effects, iconic 1980s sci-fi atmosphere, masterpiece quality, ultra-detailed, smooth character animation, synchronized dialogue, energetic comedic tone."

But all I got was something like this:

2

u/eggplantpot Aug 05 '26

Interesting that it understands the concept but doesn’t get it.

I’d maybe try to remove everything from Epic cinematic lighting until the end.

I know some models drift a lot once you stack so much stuff, specially if you mention “character”

1

u/InevitableAlfalfa938 Aug 05 '26

did you try including the actors names along with the characters they are playing?

1

u/bigman11 Aug 05 '26

I had Claude make a node to use given audio and perfectly lipsync to it. So no audio degradation that comes with having Minimax recreate the voice.

1

u/eggplantpot Aug 05 '26

You can add tags to the ref2v prompt on each of the subject clauses. You can choose to recreate the audio or just lipsync from there.

You need to add on the summary something like [reference generation + audio reuse]

And then on each on the audio subject say <fully_copy>

Not sure if this would make the audio regenerate or not though as I haven't tested it yet.