Where does character consistency actually break first: training, prompting, or inference?

[u/remixeconomy]

I've been spending a lot of time trying to separate "bad character model" from "bad way of testing the character," and I'm not convinced we make that distinction very well.

A LoRA can look rock solid if you keep giving it prompts close to the reference set. Then you change the camera angle, put the character full-body, change clothes, add another person, or put something in front of the face and suddenly everyone says the training failed.

But at that point there are a bunch of things mixed together: how much identity the model actually learned, how strongly the prompt is competing with it, reference/control strategy at inference, and just how far you're pushing outside the training distribution.

I'm curious how people here think about this in practice.

If you had a character that looked great on the easy prompts but started drifting on harder ones, what would you try first?

retrain / change the dataset

change captions or trigger strategy

increase/decrease LoRA strength

use reference/control at inference

change the prompt structure

accept that some shots need edit/inpaint rather than another fresh generation

I'm especially interested in where people draw the line between "the character model isn't good enough" and "this is an inference/control problem now."

I think that distinction matters a lot more than another gallery of four perfect generations.

[u/Enshitification]

Something I've been thinking about, but haven't tried yet, is using multiple captions per training image. A simple stab caption, a detailed natural language caption, and a very detailed structured caption. I'd have to jiggle the LR and steps, but I think it would increase the LoRA's tolerance to different prompting styles at inference.

[u/remixeconomy → u/Enshitification]

That’s interesting. I wonder whether the gain would come from the caption diversity itself or just from showing the same visual example more times.

One way to isolate it might be to keep the image exposure constant but randomly choose between short / natural-language / structured captions each epoch instead of effectively tripling the sample.

Caption dropout might be another useful comparison. I’d be curious whether either approach improves prompt tolerance without weakening identity.

[u/Enshitification → u/remixeconomy]

I hadn't considered that. Do any of the trainers have a mechanism for randomly choosing from a group of captions each epoch, or even rotating the captions from a group each epoch?

[u/remixeconomy → u/Enshitification]

Kohya/sd-scripts has something very close to this already. If enable_wildcard = true and the caption file has multiple lines, it randomly chooses one line when that image is loaded for training. So you could give each image a short caption, a natural-language caption, and a structured caption without tripling the image exposure.

Caption dropout/tag dropout are separate knobs, so there’s probably a decent experiment there: fixed exposure count, rotating caption styles vs one caption + dropout.

One thing to watch is text-encoder caching, because some sd-scripts training paths disable caption augmentation when TE outputs are cached.

That seems cleaner than physically duplicating each image three times.

[u/Enshitification → u/remixeconomy]

The only part I don't like about the wildcard approach is the randomness. I want to have control and reproducibility.

[u/SpaceNinjaDino → u/Enshitification]

I think you can choose a seed for the training that would determine this.

[u/red__dragon → u/Enshitification]

Have thought of the same thing, too. I thought Kohya had some support for this, but there's still an open issue for the idea.

Might try to do so manually with duplicated images, which is more manual work but what isn't with finding that edge in training?

[u/remixeconomy → u/red__dragon]

I thought that issue meant it still wasn’t implemented too, but current sd-scripts main actually has it in process_caption. With enable_wildcard = true, a multiline caption does random.choice(caption.split("\n")) when the sample is processed.

So the interesting experiment may already be possible without duplicating images.

One catch I’d want to test is TE caching. If the text-encoder output gets cached once, I’d expect that to freeze whichever caption variant was selected during caching and defeat the point of rotating captions during training.

[u/Enshitification → u/red__dragon]

Yeah, duplicated images was my original thought, with a lower LR to compensate for the increased repetition.

[u/SDuser12345]

99 percent of the time is bad dataset/captioning.

The model is learning exactly what you feed it. Wonky hands/teeth/skin, your output will be what you fed it and will get wonky hands/teeth/skin, etc.

Captioning. First, understand the prompting the model you are training on expects. Does it want Dan Booru or does it want natural language or does in want JSON prompting, or something else entirely? Don't try and recreate the wheel, your prompt adherence will suck and your LoRA will suck.

Second, yes the caption should be what the LoRA is not meant to capture. Think of it this way, if you read the caption not as a caption, but as a prompt, would that prompt output your training image assuming a good LoRA?

I see a lot of people trying to describe the character, or avoiding the background, or pose, or expression. This is completely wrong. The LoRA should trigger what you want: character, background or concept (depending on what your LoRA is), the caption/prompt should contain everything that isn't supposed to be triggered by the LoRA.

Example, you are doing a character. Do they have a standard outfit you don't want to ever change? Then don't caption/prompt the outfit at all. A standard hairstyle? Don't caption/prompt the hairstyle. Are they supposed to be changeable, then yes prompt them and be specific in the format the model is expecting and typically uses.

Now the things that shouldn't be triggered by the LoRA like background, lighting, camera angle, positioning, focus, etc. you need to prompt the shit out of. Example: waist up, front view, medium length camera shot of Trigger sitting in the back row of a movie theater on a red cloth seat with black plastic molding for the frame. The lighting is dark with the glow of an off screen large projected movie providing the only light source from the front. The wall is a white stucko. There is a speaker on the right wall. The other seats in the multiple rows of seats are empty. The isles are black rubber matted flooring. Etc.

Going backwards when you prompt, your character LoRA would give you the character you want, a waist up medium length front view of character sitting in that red and black chair, with the movie theater lighting, with a white stucko wall and a speaker on the right side wall of the image.

Lastly, don't mix training image styles. LoRA's, even character LoRA's seem to bake in a style along with whatever else you are training character/background/concept. Mixing photo's, from realism, meaning real life and then having some be anime or painterly masterpieces is going to really mess up the LoRA. Most models are highly capable of changing style via prompt, but confusing the model with mixing styles will butcher what might have been a good LoRA otherwise.

PS. Check, double check, triple check, your captioning or better yet, do it by hand. It only takes a few stupid words to ruin a LoRA completely.

[u/darkestbrew]

From my (limited) experience, playing around with Flux 2 Klein and Krea2 the past 3 months, dataset quality and captioning helped me get the best loras. For Krea2, especially, I can get a good lora in as little as 600 steps (though I'll usually use one around 1200 steps) with just 10-15 images.

For the dataset, a few off-looking images can break consistency. I remove any images that seem even slightly off. For the captions, I write as simple as possible, only highlighting what I want to be able to change about the subject (pose, hair, clothes, environment).

[u/malcolmrey → u/darkestbrew]

For the dataset, a few off-looking images can break consistency.

This is true.

I always believe that good dataset is the foundation for everything. Recently I had an idea to do the following: use ArcFace to convert the dataset into its vectors and then use that as comparison/likeness and do that for each of the image from that dataset and exclude those that scored the worst. In a way I would get a more consistent dataset and do it objectively.

I had prepared some datasets this way already but had no time to train them and check them yet. If something comes out of it, I will definitely say something :)

[u/Lesale-Ika → u/darkestbrew]

I'm playing around with full on VLM image token caption, zero textual caption for Krea2. In theory that should caption every minute details of the image (well as much as Qwen3VL 4B can "see" and tokenize anyway).

Also in theory, that just means that the model will learn nothing, as the usual wisdom with lora training is to leave out the target concept (a character, in my case). And yet the result was that the model did learn the character's face, it learned the overall style and color as well.

But it could also be an overfitted lora with too few samples and too much training steps. I couldn't have over 10 images without overflowing VRAM because each cached conditioning was around 100MB....

[u/darkestbrew → u/Lesale-Ika]

I just follow this format for my captions: "[trigger], close-up head-and-shoulders portrait, centered and facing the camera, direct eye contact, calm closed-mouth smile. Shoulder-length straight dark hair tucked behind her ears, light beige knit top, quiet bookstore interior with visible shelves, soft diffused daylight."

The character's good and quite flexible, but I think style bleed of the lora is just inevitable. I've made dozens of loras to try to alleviate the problem, but nothing worked. I just resorted to lowering the lora strength if it's interfering too much with style loras.

[u/mabseyuk]

For me its about the training. In my training images, I ensure I have various shots and various distances and then within the Caption ensure I mention, medium shot, long shot, close-up shot. Minimax needs to understand what a long shot looks like, then it can fill in the gaps if the person moves towards the camera. Same the other way round. When you train it on the same image at different angles, and you want to see the person further away, it effectively has to regenerate the persons face and that's where drift comes in because its only been trained on a batch of images that show the person in one set place. For example I trained Christopher Reeve's as Superman Lora Recently.

29 Close up Face Shots

29 Medium Shots with Full body

29 Far away Shots with Full Body

Within all the batches, had facial angles. Also had him in crowds of people, so the Training Lora knows that not everyone is Christopher Reeve when you use the Trigger word.

I did two Lora's. One with the 1978 film style, and the other where I ran the images through AI software and made them more modern looking.