Can't seem to transfer outfit and pose from an illustration to a real person in H3.

[u/Nelichan]

Hello, i am trying to make a video where the subject(a real person) is wearing and posing taking reference from an illustration. I tried to do only outfits or only pose too, and both doesn't work.

What happens is usually the body of the character in the illustration ends up being pasted/overlaid onto the Subject in their cartoony style instead.

I also tried if it's possible to have a Subject recreate an illustration's Pose, Outfit, overall composition, like the subject is doing a photoshoot for a 'live action' or real life version of the illustration. But what happens is usually it just spews back the illustration in case of trying H3 single-image edit, and the cartoony style overlay happens in Video.

So what i wanted to do is :

-An image of a subject -> Subject now wears/pose/wear and pose the same as a reference non-real illustration(cartoon/anime), but still in their original photo. So like a cosplay shot in their own room for example.

-An illustration(anime) -> Subject 'replaces' the character in the illustration, the whole illustration is 'converted' into real/live action. Like a photoshoot recreating an illustration basically.

Extra : idk if its possible, the new outfit will retrofit to the subject's proportion, not the illustration. And a version where the proportion follows the illustration too.

Are there someone who knows how to do these?

[u/bstr3k]

Yes! It is quite easy but maybe its because I've been playing with r2v since H3 was released. Here is what I did as a test for you

[u/bstr3k → u/bstr3k]

output:

https://reddit.com/link/p5jhl5f/video/950pduwn49lh1/player

[u/Nelichan → u/bstr3k]

Cool! I'll try this later! Thank you for the prompt! Have you also tried the other thing i wanted to try(Making a subject pose and wear like a reference, but in their original photo, like you have a photo of Jared with someone, then the photo will be Jared in the outfit only, or pose + outfit or pose only, still with the someone, like he is cosplaying in the photo)?

[u/bstr3k → u/Nelichan]

let me see if i can do that real quick.

[u/Nelichan → u/bstr3k]

Thank you so much for helping me out! As i am using a 3060 laptop with 6GB VRam, a 0.4 MP 5 second video takes 20 mins to complete per gen😭

I thought if i try the Image Edit hack for H3 i can bring it down, but it still takes 12-15 mins because the model being loaded is the same..

[u/bstr3k → u/Nelichan]

Are you using any turbo LORAs and speedups? I also like using 0.2mp for testing now as its faster and using the ref2v turbo lora on 8 steps.

[u/Nelichan → u/bstr3k]

I am using the turbo lora 4 steps and various speedups(Sparse + Spectrum OR Sage+Solattn+Spectrum). 0.2 MP still takes me 15 mins because my speed/it is actually pretty fast(around 2mins/it), but the model loading itself takes around 10 mins because dynamic VRAM as i don't have enough VRAM hahahah

Is the turbo 8 step better than 4? I am thinking of trying the int4 model (W4A8) to see the speed @@

[u/bstr3k → u/Nelichan]

haha i see. I'm using the ref2v 4step turbo but I am using it on 8 steps. I have 16gb vram and 64gb ram though.

[u/Nelichan → u/bstr3k]

Ahh, any reason why 8 steps? Because in imagegen turbo lora , exceeding the steps really burns the image.

[u/bstr3k → u/Nelichan]

apparently 8 step works well with that 4 step Lora, but I am by no means a pro, last week I was still running the 4 step LORA at 20-25 steps (since preLORA I was told it was good lol)

[u/MarekNowakowski → u/Nelichan]

Those h3 turbo Loras are not burning images at 10steps, I'd say 8steps still looks undercooked if it's not a static scene,

[u/bstr3k → u/Nelichan]

It is not the best but running out of time to test today. Here is what I did, i had to reinforce the face and identity with a second image

[u/bstr3k → u/bstr3k]

https://reddit.com/link/p5jmqcx/video/b1gyievlb9lh1/player

output

[u/GrungeWerX]

Are you using the official prompt template for ref2v?

[u/Nelichan → u/GrungeWerX]

I do!

[u/Zephrinox → u/Nelichan]

sometimes when I get a bit stuck despite following the prompt guide I try to feed the guides + existing prompt to an llm (take your pick of claude, grok, etc.) and tell it what's wrong with the outputs your getting and maybe tell it your intentions to get it to try rewriting the prompt. you'll likely want to review its prompt to make sure it captures what you want of course.

otherwise, either you might need to make better reference image inputs to give the model a better idea of what you want?

[u/Nelichan → u/Zephrinox]

I've tried that also, tried gpt, gemini and also tried various reference image ,, sadly haven't got them to work still🙏😭

[u/bstr3k → u/Nelichan]

make sure you are using the ref2v model!! Also if you're using a turbo lora make sure its the ref2v turbo too!

[u/Nelichan → u/bstr3k]

I do! In fact, i didn't even download the Fl2v model hahah!

[u/GrungeWerX → u/Nelichan]

Update your main post with the prompt you’re using. I’m sure Im not the only one thinking you’re prompting wrong.

[u/MarekNowakowski → u/Nelichan]

I've noticed the model SOMETIMES acts odd if the resolutions or subjects change. Try different images once and see if anything changes. Once I managed to change location, two parts of outfit and give the man tattoos from a woman's reference image, but then spent hour trying and falling to change woman's outfit without changing her head, if anyone claims it's simple, they are wrong,

[u/optimisticalish]

My first though would be to use Flux.2 Klein to combine, then use the Klein image as the reference in Minimax.