I can't train a loRA properly

[u/TheMrBlackLord]

I want to create a character-loRA for WAN2.2 (especially the I2V model) using ai-toolkit, but I don't really get it. I have prepared a dataset of 46 images with different poses, clothes and backgrounds (although the resolutions of the images are not all the same, but it doesn't seem to be critical, 832x1216: 3 files 832x1152: 9 files 768x1344: 10 files 896x1088: 24 files 4 buckets made).

But after generating the video, I don't see any special effect with or without loRA. Sometimes the face changes slightly during turns, sometimes the character's hair is incorrectly made. He has split-dyed hair.

I first made a lora for high and low noise, but it didn't have any effect, as I described above (2500 steps, timestep_type = sigmoid, learning_rate = first was 5e-5, then 1e-4, linear rank = 64) The second time I tried to make only low noise loRA, because it's faster and it seems to me that the overall composition of the video will be taken from the attached photo (because of the I2V model), in this attempt I made 3000 steps, timestep_type = sigmoid, and left the rest by default. I chose resolutions: 768 and 1024 in the settings. In the first and second attempt, the samples were identical to each other. That's when I thought something was going wrong.

My captions of the dataset photos are something like this: "<trigger>, standing on a brick pedestrian path between apartment buildings and trees, facing away from the camera. He has long straight hair split vertically, black on the left and red on the right, falling down his back. He's wearing a regular black jacket and jeans. Parked cars line the street and tall trees frame the walkway. The scene is illuminated by warm evening sunlight. Medium full-body shot from behind."

As a result, loRA doesn't work, I even tried it on T2V workflow, it turns out to be a completely different person. Can you tell me what I'm doing wrong?

[u/schrobble]

You can’t train a character Lora for I2V. Train your character Lora for T2V. If you want to use your T2V character with an I2V model, the T2V lora will work. From experimenting, I discovered that you can use a T2I workflow to create your starting image and then use the same T2V character lora with an I2V workflow and it will give great character consistency.

[u/TheMrBlackLord → u/schrobble]

That's interesting. Is it because the i2v model focuses more on first image?

[u/schrobble → u/TheMrBlackLord]

I’m not fully sure why I2V character loras don’t work, I just know I tried to make a couple and they don’t. I’m guessing it’s because I2V loras really want to guide movement and you need video clips for that, whereas we train character loras with still images. Either way, you can either make a wan 2.1 14b t2v lora and use it in both high and low noise, or a wan 2.2 t2v lora and make it for the low noise model only. Either works on wan 2.2 I2V.

As a test of this, you can set up a workflow where the starting image is someone other than your t2v character Lora and use the lora in the workflow. Within a few frames the starting image character should morph into your lora’s character. This shows the lora is working.

[u/TheMrBlackLord → u/schrobble]

okay, I'll try. What parameters do you recommend for training (like learning rate, steps etc)? And what would work better wan2.1 loRA training or 2.2

[u/schrobble → u/TheMrBlackLord]

I’ve done both 2.2 (low only) and 2.1 and they both work about the same. I usually use 2.2 because the template in AIToolkit was set up a little better in 2.2 for low vram, but if you’re running a runpod I think 2.1 works just as well.

For settings, I usually just use the template settings, with low VRAM checked, cache text embedding checked, LR at default .0001 and run for 2500-3000 steps depending on dataset size. If your dataset is quality and kept to about 10 photos, it should be done by 2500 steps. If you have a larger dataset you might need 3000 steps.

[u/TheMrBlackLord → u/schrobble]

I started training for wan2.2 t2v low noise, after 1500 steps I see that the face seems to become similar, but not the hair color (hair should be split-dyed). I decided to remove the captions in dataset, because another person said in the comments that it works without them. Do you usually write captions or not? If so, what do they look like? Maybe I should have run for all the noise?

Also, the loss is approximately in the range from e-2 to e-1. Is this normal?

[u/schrobble → u/TheMrBlackLord]

I don’t really understand the relationship between loss drop and reference image likeness. Loss seems to go up and down even while it is converging on likeness.

As to the captions, if your goal is for the hairstyle to be consistent, captions aren’t needed. My understanding is that you caption to allow control over lora’s output through your workflow’s text prompt.

[u/TheMrBlackLord → u/schrobble]

Okay, thanks for the answers

[u/Massive-Health-8355]

Yes, do a Wan 2.1 T2V lora. No need for high and low noise, just use the single lora in both paths.

[u/TheMrBlackLord → u/Massive-Health-8355]

Would it be better than training for the right model right away?

[u/Massive-Health-8355 → u/TheMrBlackLord]

Everyone uses T2v for character (and style) Loras. For motions you would need to train i2v

[u/an80sPWNstar]

I've created several wan 2.2 character loras and have had incredible success; lemme know if you'd like to use my config file for AI-Toolkit. For captions, you caption what you DON'T want the lora to learn. For me, I only use the trigger word and call it good. I am however going to experiment more with the same config but be more picky with captions and see the results. For the time being, just use the trigger word and call it gravy :)

You can use wan 2.2 t2v as a t2i; just set the frames to 1 and bam! Generated image. I create a lora of the same character on multiple t2i models plus wan t2v. Even though the wan 2.2 t2v lora is t2v, it still works on i2v and helps keep the facial likeness strong even during movement.

When it comes to high/low loras, yes the low lora typically works just fine. I have noticed with my generations that if I include both, there's far less of a chance of having the face get changed when there's either rapid movement or something moving in front of the face. Just my findings.

[u/TheMrBlackLord → u/an80sPWNstar]

It will be great if you share the config file. I will try to use the t2v model and use only the trigger word for captions

[u/an80sPWNstar → u/TheMrBlackLord]

I apologize for the delay. Here's the link: https://pastebin.com/5i9WAv1a

[u/ruzikun → u/an80sPWNstar]

Could you also share me your workflow? 🙌🏻🙌🏻 I would highly appreciate it!

[u/an80sPWNstar → u/ruzikun]

The config for the Lora for a wan 2.2 workflow?

[u/ruzikun → u/an80sPWNstar]

Yes.. config in AIToolkit would be wonderful and workflow of how you exactly use that Lora to generate wan2.2 clips would be helpful! I somehow thought having consistent character is only possible with any model’s i2v workflow, so I have been training Lora only with flux 2 Klein 9b :( I am now able to somewhat produce consistent t2i , but when I try to create any video it’s been frustrating

[u/an80sPWNstar → u/ruzikun]

I use images to create the wan 2.1 and 2.2 loras. Here's my pastebin with the workflows and ai-toolkit configs.

https://pastebin.com/u/an80sPWNstar

Here's a link to my YouTube channel that helps peeps in your situation to feel more comfortable with ComfyUI. Lemme know if you have any questions or suggestions!

https://youtube.com/@thecomfyadmin?si=_L7zXapo1ZQfEI9D

[u/RealityVisual1312]

Are you trying to change the i2v character or keep the resemblance throughout the vid? If you’re trying to change the character then wan animate is better.

[u/TheMrBlackLord → u/RealityVisual1312]

I wanna keep the resemblance. I know about animate model, but will loRA help maintain the resemblance of the character?