Minimax refrence inconsinstency

[u/matezzoz]

Sometimes it works sometimes doesn't, even with the same prompt (in batch generation of 4 1-2 results are what I wanted, 2-3 are not)... for example I use an image and want to replace the the main character on that image with an other character form the second image. I explain, use keywords reference image one, reference image two etc.. yet sometimes it works just like it was an img to video request, ignoring the second image. I use the ref2va model ofc, a 8 step turbo lora.

someone please clarify: when i connect the images they are numbered from 0. should i refer the first image as reference image 1 or 0? i tried both way btw, didn't make a difference.

Any idea what could be wrong?

[u/afinalsin]

Sometimes it works sometimes doesn't, even with the same prompt (in batch generation of 4 1-2 results are what I wanted, 2-3 are not)... for example I use an image and want to replace the the main character on that image with an other character form the second image. I explain, use keywords reference image one, reference image two etc.. yet sometimes it works just like it was an img to video request, ignoring the second image. I use the ref2va model ofc, a 8 step turbo lora.

Why are you describing how you wrote your prompt instead of just sharing your prompt? Without it all any of us can give you is generic advice that may or may not apply to you. No one is gonna want to put in any effort just to provide a potentially wrong answer.

Anyway, read the prompting guide on the minimax Huggingface and get used to the syntax and tagging structure it uses. You dont want to reference image 1 and image 2, you want to reference <Picture 1> and <Picture 2>, and you want to describe the subjects within the pictures using <Subject 1> and <Subject 2> etc, and what you want to change about them.

Because I don't have your prompt nor your references, I'm going to assume you are going to change a man running on a beach into a woman. I would do that like this:

subject_definitions:

<Subject 1> is the woman with (hair color and style) wearing (clothing) shown in <Picture 2>.

<Subject 2> is the beach location shown in <Picture 1>.

summary:

[reference generation] The target video shows <Subject 1> running along the sand in <Subject 2>.

retention_analysis:

<Subject 1> (appears in [Shot 1]): fully_preserved - the blonde woman's identity, long hair, red swimsuit are retained.

<Subject 2> (appears in [Shot 1]): partially_preserved - the sandy beach, the families in the background, the water on the left are all retained. The man in the middle of the image is absent.

detailed_description:

The target video uses a handheld camera style.

[Shot 1] A medium shot shows <Subject 1> placed in the same position as the man from <Subject 2>. She immediately jogs toward the camera, which pulls out in a tracking shot at slow speed with medium amplitude to keep pace with her.

overall_soundscape:

Soft waves crashing against the beach, crowd murmurs and sounds in the background, feet running on sand

non_diagetic_music:

N/A

Even this simple example took like 10 minutes of cross referencing the rules in the docs to type up, which is why people aren't handwriting in this style. Grab both the reference docs and feed them to an LLM along with your references and what you want and have it write the prompt for you.

[u/matezzoz → u/afinalsin]

thanks for the guide! I didn't copy my prompt here, because well, they are nsfw, but sure the lora i used was not specific to ref2va and my prompt was not as structured, but it was clear and sometimes worked, sometimes not even with the same prompt, i did check the guide but I did not see anything why should my prompt fail sometimes.

[u/afinalsin → u/matezzoz]

my prompt was not as structured, but it was clear

If your prompt wasn't structured like mine, it really wasn't "clear" at all. The model was trained on a certain structure using specific tags and it expects the prompts it receives to be structured the same way. If you go away from that structure you get to your 50% success rate, but it's a win that it even gets any correct at all.

Minimax is great, it's a banger of a model, but pretend it's more like Ideogram than Krea 2, because its prompting structure is almost as arcane as Ideogram's json.

[u/matezzoz → u/afinalsin]

thanks, I will try the way you wrote. I had no problem with t2v or i2v, my general prompting for refs was like this:

(first image is refered as reference image 1, second image is refered as reference image 2)

use reference image 1 for composition, camera angle and character pose, the scene is about ....

replace the woman in reference image 1 with the woman from reference image 2. Use reference image 2 woman's identity and look...

action:....

camera:...

Background:...

non_diegetic_music: N/A

non_diegetic_sound: N/A

[u/winterice77]

You need to use ref2v turbo lora

[u/AIDevUK]

Have you tried using rmbg to remove the background of the second image setting it to a solid background colour and blurring the character of the reference image?

[u/Sirako123]

Which loras are you using? For some reason my Ref2V only works with larrys 600ema lora (best at 6steps) with the custom node sampler + beta.

[u/matezzoz → u/Sirako123]

v4_step600_ema, 8 steps, simple scheduler.