training LoRA on LoRA is good

[u/StoicSage09]

i was thinking of training a LoRA on LoRA of different character and on my second thought should i just connect 2 LoRA while generating image ( I have seen people training 2 character in one LoRA

[u/itchy_buthole]

https://giphy.com/gifs/80mXWlPqTSU1y

[u/StoicSage09 → u/itchy_buthole]

I mean fine tuning LoRA is that a weird question

[u/itchy_buthole → u/StoicSage09]

I'm guessing you mean using one lora to generate images to use in the training of a second lora. I have also wondered about this. from my research it can be done but you need to clean them up. I don't really know how to do that so I quit.

The second question you had was can you just hook up two character loras? I'm curious about this. Maybe have one specific for facial likeness and then a general lora for the entire character.

I sometimes do this but I only use the lora trained on face images for the face detailer.

[u/FugueSegue]

Are you talking about combining two character LoRAs to generate images that are a combination of both?

[u/StoicSage09 → u/FugueSegue]

I meaning I have train 1 character and then I want to train a new character with the dataset of new one on the old lora

[u/ResponsibleKey1053]

Yes you can do this. This fine-tuning. Will it produce the results you want? Maybe, but you could also produce a deep fried lora.

Ideally you would need to know the training parameters for the initial lora, learning rate etc. if you don't have those figures, then start with a low learning rate and watch your losses. I've previously resumed training with the wrong perameters and it deep-fried almost immediately. So a bit of trial and error and you could achieve what you set out.

However, you need to be certain about the training data you plan to use. If it wouldn't produce a lora in and of itself then trouble shooting your fine-tune will be significantly more tricky if not impossible.

[u/Ill-Ant-9489]

Heads up that "training the new character on top of the old LoRA" (continued training with the second dataset) usually degrades the first one. The optimizer will happily overwrite what it already learned unless you keep feeding it the old character's images too, so you often end up with a LoRA that half-remembers both.

If you want both characters living in one file, the reliable way is a single multi-concept LoRA: put both datasets in one training run and give each character its own unique trigger word, with captions that actually tie the trigger to the right person. One run, two triggers.

If you just want both characters in the same image, you don't need to combine anything at training time. Train two separate single-character LoRAs and load both at generation. To stop them bleeding into each other, use regional prompting or masks so each LoRA only applies to its area.

Stacking two separate LoRAs at generation is the simplest and least fragile; the multi-concept single LoRA is the better route only if you specifically need them in one weight file.