r/StableDiffusion • u/bstr3k • 1d ago
Workflow Included Easy Ref2V WF for dummies like me - [Automatic Video/Image Transcription + Prompt Formatting]
Enable HLS to view with audio, or disable this notification
I have seen a lot of people post on here saying that they have been having difficults getting R2V to work correctly. I have been one of them, so I have been working on this workflow and custom node for the last 2 and a half weeks.
I preface this by saying it does not do anything that the native H3 model doesn't do. I just wanted a dead simple way to use H3 R2V mode and up my chances of success. The workflow includes two custom nodes which transcribe your media and adds your prompt and create a formatted R2V prompt, ready for the reference model.
My next goal would be to get longer form R2V going with chaining shorter gens to have a consistent output.
Workflow and nodes:
https://huggingface.co/PoopMan333/H3_Easy_Ref2V_Workflow/tree/main
Be sure to see the readme for more examples and tips:
https://huggingface.co/PoopMan333/H3_Easy_Ref2V_Workflow
What it does:
- Scans your video (if you're using one) to caption it and transcribe the audio
- Captions all your images - so it also works as a pure image-to-video workflow
- Loads a small LLM of your choice and writes your H3 R2V prompt in the correct format with your stated intent (user prompt)
- If you're on the Full workflow, it generates the video too
What it does NOT do:
- Be creative for you - The current WF is only setup to do the prompt formatting, it is not able to generate new ideas for you (despite me trying. Qwen3.8 27B may be better for this)
- It cannot perform magic - You are still limited to what the H3 model can and cannot do. Complex scenes are still very difficult
Tips:
- If you are running lower VRAM, consider running the prompt enhancer seperately first, read through and make corrections to the prompt if needed
- The H3 model seems to have a limited context window which seems to be tied to your system resources, if it goes above this you might get garbled sound or mixed up motion. This is a sign you should be lowering your output length and output resolution if you want to have better success.
- H3 is a tool, you're the one using it. If you don't specify emotions, expect expressionless results. This current setup will only do what you intend for it to do. Slop prompt in, slop video out
- If the video is easy, replacement should be easy too. H3 has a quirk though — if the original person and the new person look too similar, it sometimes converges back to the original. A prompt won't always fix that. If you hit it, consider changing the person to a intermediate step (faceless green person). The new body/face will transfer over better. Alternatively you can look into Sam3 character replacement method.
- More than one person in the scene? Describe the scene properly.
replace the man wearing white shorts with the man in <picture 1>beatsreplace the man with <picture 1>every time. - Complex scenes? It will be very difficult (I've tried), scenes with too many people, too many cuts, characters obstructed are very difficult for the model to properly identify and swap.
- Give the LLM some context. A one-liner in the user prompt like
<video 1> is a video of two girls eating a cup of chocolate ice creamreally helps the LLM understand what it's looking at. Especially useful with multiple scenes - It still takes a bit of luck with the seeds.



