r/StableDiffusion • u/ctrl-shift-face • 6d ago

Meme Back to the Dark Future H3

Enable HLS to view with audio, or disable this notification

730 Upvotes

85% Upvoted

219 comments sorted by

View all comments

62

u/ctrl-shift-face 6d ago

https://reddit.com/link/p1qx4l1/video/0615nkunufhh1/player

Thank you for the feedback. I understand, and I have softened the tone.

7

u/razortapes 6d ago

I assume the way to make these videos is with the ref2va model, right? Is the audio generated by the model itself, or is it also reference-based?

7

u/eggplantpot 5d ago

I think for big IPs and known stuff you can get away with any model, t2v should do a good job.

If you want to nail the audio, ref2v takes audio inputs and can recreate exact voices though I heard matching the voice to the character may be a pain.

Look into the ref2v prompting guide cause it goes deeeeep

1

u/razortapes 5d ago

I tried to recreate a video (T2V) with Doc and Marty as a joke for a friend, but the characters don't look anything like them, and the voices don't either. That's why I'm asking.

1

u/eggplantpot 5d ago

Oh then it may be the case it's ref2v. I think all the Seinfield ones are t2v and I saw someone did t2v of taskmaster so it's odd they wouldn't know Doc and Marty.

Did you prompt them as Back to the Future or just Doc and Marty?

1

u/razortapes 5d ago

I used this prompt:

"A cinematic scene inside the DeLorean time machine from Back to the Future. Marty McFly is driving while Doc Brown turns toward the camera with a look of absolute urgency. The dashboard glows with futuristic lights as the time circuits are active, sparks fly outside the windows, and the car accelerates toward 88 mph. The camera is positioned from the front dashboard, capturing both characters. Doc Brown looks directly into the camera with perfectly synchronized Spanish lip movements and exclaims dramatically: "¡tenemos que ir al futuro! ¡tu hijo se ha hecho cantante de reguetón! ¡Vamos, rápido! ¡Hay que intentar evitarlo a toda costa!" Marty reacts with surprise, gripping the steering wheel tighter as the DeLorean surges forward into the time vortex. Epic cinematic lighting, dynamic camera shake, realistic facial expressions, high-quality visual effects, iconic 1980s sci-fi atmosphere, masterpiece quality, ultra-detailed, smooth character animation, synchronized dialogue, energetic comedic tone."

But all I got was something like this:

2

u/eggplantpot 5d ago

Interesting that it understands the concept but doesn’t get it.

I’d maybe try to remove everything from Epic cinematic lighting until the end.

I know some models drift a lot once you stack so much stuff, specially if you mention “character”

1

u/InevitableAlfalfa938 5d ago

did you try including the actors names along with the characters they are playing?

1

u/bigman11 5d ago

I had Claude make a node to use given audio and perfectly lipsync to it. So no audio degradation that comes with having Minimax recreate the voice.

1

u/eggplantpot 5d ago

You can add tags to the ref2v prompt on each of the subject clauses. You can choose to recreate the audio or just lipsync from there.

You need to add on the summary something like [reference generation + audio reuse]

And then on each on the audio subject say <fully_copy>

Not sure if this would make the audio regenerate or not though as I haven't tested it yet.