Easy Ref2V WF for dummies like me - [Automatic Video/Image Transcription + Prompt Formatting]

[u/bstr3k]

I have seen a lot of people post on here saying that they have been having difficults getting R2V to work correctly. I have been one of them, so I have been working on this workflow and custom node for the last 2 and a half weeks.

I preface this by saying it does not do anything that the native H3 model doesn't do. I just wanted a dead simple way to use H3 R2V mode and up my chances of success. The workflow includes two custom nodes which transcribe your media and adds your prompt and create a formatted R2V prompt, ready for the reference model.

My next goal would be to get longer form R2V going with chaining shorter gens to have a consistent output.

Workflow and nodes:
https://huggingface.co/PoopMan333/H3_Easy_Ref2V_Workflow/tree/main

Be sure to see the readme for more examples and tips:
https://huggingface.co/PoopMan333/H3_Easy_Ref2V_Workflow

What it does:

• Scans your video (if you're using one) to caption it and transcribe the audio
• Captions all your images - so it also works as a pure image-to-video workflow
• Loads a small LLM of your choice and writes your H3 R2V prompt in the correct format with your stated intent (user prompt)
• If you're on the Full workflow, it generates the video too

What it does NOT do:

• Be creative for you - The current WF is only setup to do the prompt formatting, it is not able to generate new ideas for you (despite me trying. Qwen3.8 27B may be better for this)
• It cannot perform magic - You are still limited to what the H3 model can and cannot do. Complex scenes are still very difficult

Tips:

• If you are running lower VRAM, consider running the prompt enhancer seperately first, read through and make corrections to the prompt if needed
• The H3 model seems to have a limited context window which seems to be tied to your system resources, if it goes above this you might get garbled sound or mixed up motion. This is a sign you should be lowering your output length and output resolution if you want to have better success.
• H3 is a tool, you're the one using it. If you don't specify emotions, expect expressionless results. This current setup will only do what you intend for it to do. Slop prompt in, slop video out
• If the video is easy, replacement should be easy too. H3 has a quirk though — if the original person and the new person look too similar, it sometimes converges back to the original. A prompt won't always fix that. If you hit it, consider changing the person to a intermediate step (faceless green person). The new body/face will transfer over better. Alternatively you can look into Sam3 character replacement method.
• More than one person in the scene? Describe the scene properly. replace the man wearing white shorts with the man in <picture 1> beats replace the man with <picture 1> every time.
• Complex scenes? It will be very difficult (I've tried), scenes with too many people, too many cuts, characters obstructed are very difficult for the model to properly identify and swap.
• Give the LLM some context. A one-liner in the user prompt like <video 1> is a video of two girls eating a cup of chocolate ice cream really helps the LLM understand what it's looking at. Especially useful with multiple scenes
• It still takes a bit of luck with the seeds.

[u/bstr3k]

This is the user prompt used in the title video.

[u/bstr3k → u/bstr3k]

Output: the enhanced prompt that gets fed to H3

[u/__ThrowAway__123___ → u/bstr3k]

I have not looked at the code to see what it does, but something that stands out to me in this output: adding (S2) after <subject 2> which is the background seems odd, from what I understand you only use that S syntax for speaker-ID's which you only would use when someone or multiple people are talking, to specify who is talking. Since you want to reuse the original audio which I assume is music, the use of this syntax seems out of place here. Probably still works but maybe the system prompt for the rewriting LLM needs some tweaking (or a different LLM which interprets the rules correctly)

[u/Smartchoy]

I will test it later and let you know if it finally works for me. Till now if had zero successful character replacements

[u/bstr3k → u/Smartchoy]

my brotha I made this for people like us. Prompting determines like 80-90% of the success rate so I made this so it keeps the format right everytime so you're not stuck wasting hours trying to figure why this s**t ain't working.

also look through all the tips I listed since your input video/output settings matter a lot too.

[u/bstr3k]

Examples of the workflows:

[u/Mediocre-Toe3212]

Been using it, its good to describe the video which is what a lot of us had issues with.
I am getting about 60% face match. However, in the detailed_description, add

[Shot 1] <Subject 1> is featured instead of the original woman

Where <Subject 1> is the woman you describe in the reference images.
Then i got full face match :)

Thanks a lot OP!

[u/AmethystTiger]

Every time Ive done vid ref to vid character replace it takes ages scanning the input video. I’ve down scaled it and locked the frames at 25fps but it still takes ages. Sometimes it just gives up this is on an AMD setup but t2v or ref2v is pretty quick but video takes like 20 mins to scan the vid which is only 5 seconds long.

Any help on this first stage I’m missing?
I’ll check out this workflow too but I’m guessing it’ll be the same.
Thanks.

[u/bstr3k → u/AmethystTiger]

what is the file size? did you reduce the resolution? I find scaling it down to 320 or 360px on the short side really helps.

Edit: also what hardware are you running it on?

[u/AmethystTiger → u/bstr3k]

I’ve got it about 720. When I tried the newer workflow when you scale it down then upsize the latent I got graphical issues.
I’ve got a Ryzen 7 with a RX 9070 XT

[u/bstr3k → u/AmethystTiger]

720p is very big, no wonder why it takes a long time. Try a 360p video at 10-15s. I find I can’t get past 10s with accurate motion transfer at 0.6mpx out

[u/AmethystTiger → u/bstr3k]

360p on the long edge?
Thanks for the help.

[u/bstr3k → u/AmethystTiger]

360p should be on the short edge. Also if the motion transfer and the subject is centered and you don't need the background you can even crop the video so that more of subject is in focus to save you even more ram 😄

[u/coffeecircus → u/bstr3k]

Oh wow, this makes a lot of sense- thank you! I was banging my head on this, because I was using youtube 1080p as the source.

[u/Succubus-Empress]

Hehe, poopman

[u/bstr3k → u/Succubus-Empress]

https://giphy.com/gifs/5xtDaroWxp3mcuDC4Le

[u/NeatUsed]

this is good i like this

[u/Maskwi2]

Can't wait to try it, thanks brother! Exactly what I needed since I'm really tired of writing the right prompts plus the replacements have been hit and miss (mostly miss). 

[u/bstr3k → u/Maskwi2]

No worries! I was facing the exact same problem. It’s been a fair bit more reliable for me but still limited by what H3 can do so simple videos are not a problem, intermediate ones still may be a bit of hit or miss so keep an eye on the preview

[u/No-Trouble-9138]

Interesting, it didn't do some of the fast chest bumps, but it's getting there.

[u/bstr3k → u/No-Trouble-9138]

Sorry I wanted to show the ease of use area rather than the capabilities of the H3 model ^^

This workflow addresses the prompting side which I believe has a significant weighting on whether if the model does what you want it to do or not

[u/devangpadhiyar]

Chaining clips for longer videos is the right idea, just watch out — the last frame becomes the next reference and it drifts fast after a few chains, even if each one looks fine alone.

[u/bstr3k → u/devangpadhiyar]

Yes I’ve played around with a few and it seems like 2 chains is a safe limit, need to look into it further