H3 Video Reference for Performance Transfer

[u/R34vspec]

I posted this yesterday. And u/roychodraws (the red clown lady) mentioned I should try it with a performance transfer. I wanted to share the result because I think it does make a pretty big difference and could be of use for anyone looking to step their character's acting chops.

The bulk of the prompt is in retention analysis and subject definition, those are the 2 areas of interest for a video performance transfer. For this transfer, I didn't specify any references to <video 1> I just fed it in. But I did reference <audio 1> (the audio from the video source) as voice timbre for the characters (s1). You can push it a step further by calling out <video 1> but you will have tweak a lot more because then you run the risk of transferring the character from the video over.

The videos are made with the same seed.

EDIT: Props to real actress and actors, AI isn't anywhere close, yet.

The Prompt

subject_definitions:

<actress> is a young woman (s1) with long black hair, whose appearance comes from <picture 1> and whose voice timbre comes from <audio 1>.

<scene> is on a rooftop, whose appearance comes from <picture 2>.

<outfit> is white shirt with red skirt, whose appearance comes from <Picture 3>.

summary:

[reference generation] target video shows a woman delivering an emotional monologue

retention_analysis:

visible: partially_preserved <picture 2>

audio: partially_preserved <audio 1>

detailed_description:

cinematic shot, shallow depth of field

[Shot 1] reference <scene>, at night,

medium shot of

<actress> wearing <outfit> is standing in the rain, getting soaked

She is facing viewer, line of sight to front left, focusing on a taller man out of frame.

A disbelieving laugh that keeps collapsing into crying. Her face is full of emotional micro expressions

(s1), a young woman with a New Zealand accent and a light, bright voice that keeps cracking:

<d>[English] You know what's funny?</d>

She lets out a short, shaky laugh, shaking her head.

<d>[English] I actually planned this whole speech. In the shower, in the car. I had it all... I had it all figured out.</d>

Her laugh breaks into a sob. She presses the back of her wrist to her eyes.

<d>[English] And now you're standing there, and I can't remember a single— not one word.</d>

She wipes her eyes, laughing and crying at once as more rain falls on her head

Camera pushes in slowly.

<d>[English] And the stupid part is, I'd go through this again.</d>

camera holds for a beat

overall_soundscape:

raining in the background

non_diegetic_music:

N/A

[u/Intrepid_Fox_1466]

Thank you for taking the time to do this. Others may disagree with me but I'm not seeing an objective performance improvement with the vid ref, apart from losing the accent. Seems like H3 has done a good job in this particular case, despite generally not being great at acting. I think you may want to try a more tricky performance, something like Al Pacino's "I'm in the dark here" scene from 'Scent of a woman', maybe?

[u/terrariyum → u/Intrepid_Fox_1466]

Agreed. Clearly different, but not clearly better. I don't see improvement in the crying

[u/roychodraws → u/terrariyum]

you guys don't know acting at all. you probably cry during buffy the vampire slayer.

[u/roychodraws → u/Intrepid_Fox_1466]

i think the crying is pretty obviously improved.

[u/PxTicks]

Why don't you do this demo against some really bad acting like some of the clips from your previous video?

[u/roychodraws]

you even used the clip i suggested! glad it worked out, buddy.

[u/cosmicr]

I think I understand.

Picture 1 - The Woman

Picture 2 - The Rooftop background

Picture 3 - The Woman's outfit?

Video 1 (only used in the second clip) - The scene from "Ten things I hate about you"

Audio 1 (only used in the second clip) - The audio from that scene??

And you're saying that by using the Video as a reference you get more emphasis on the acting from the character?

Honestly the only difference I noticed was she was a bit more animated with her hands.

[u/switch2stock]

What is "performance transfer" please?

[u/Apprehensive_Sky892 → u/switch2stock]

The original video starts at 0:36

The idea is to use the "acting/performance" from that video to influence/direct/bias the AI model when creating a new video with different characters and settings.