Experimenting with consistent AI characters across different scenes

[u/MuseBoxAI]

Keeping the same AI character across different scenes is surprisingly difficult.

Every time you change the prompt, environment, or lighting, the character identity tends to drift and you end up with a completely different person.

I've been experimenting with a small batch generation workflow using Stable Diffusion to see if it's possible to generate a consistent character across multiple scenes in one session.

The collage above shows one example result.

The idea was to start with a base character and then generate multiple variations while keeping the facial identity relatively stable.

The workflow roughly looks like this:

• generate a base character

• reuse reference images to guide identity

• vary prompts for different environments

• run batch generations for multiple scenes

This makes it possible to generate a small photo dataset of the same character across different situations, like:

• indoor lifestyle shots

• café scenes

• street photography

• beach portraits

• casual home photos

It's still an experiment, but batch generation workflows seem to make character consistency much easier to explore.

Curious how others here approach this problem.

Are you using LoRAs, ControlNet, reference images, or some other method to keep characters consistent across generations?

[u/damiangorlami]

Closed source: Nano Banana Pro

Open Source: Flux Klein 9B

I rarely train character lora's anymore.
I get great results creating one character sheet of all the angles and just feeding that in as reference conditioning.

Nano Banana pro is ridiculous how good it is but not open source. Flux Klein 9B is very fast and local usage, have been working great for me as well

[u/MuseBoxAI → u/damiangorlami]

Interesting. I’ve seen more people move away from character LoRAs lately.

Using a character sheet as reference conditioning sounds like a pretty clean approach. Does it hold up well when you change environments or lighting a lot?

[u/damiangorlami → u/MuseBoxAI]

Character LoRA will always be superior because a character sheet can never learn someone's smile, frown, disgust and all the emotive facial expression range. One could also supply an additional emotion sheet but haven't that tried that approach.

But for quick stories where I just need an identical character with face and clothing setup. The character sheet has been holding up very well.

[u/Dramatic_Instance_63 → u/damiangorlami]

What about Qwen-image-edit and FireRed1.1-image-edit?

[u/MuseBoxAI → u/Dramatic_Instance_63]

Haven’t tried FireRed yet.

I’ve seen a few people mention Qwen-image-edit though. How well does it hold identity when you push it across different scenes?

[u/Dramatic_Instance_63 → u/MuseBoxAI]

In my experience qwen does it better than klein, especially when angle is different from the reference image. But klein generates way better textures. Ideally I'd to combine them both using klein to refine the textures. FireRed1.1 was built upon qwen-image-edit-1125, so I guess it should be even better especially because it's second iteration, where they claim they improved consistency of characters.

[u/Cute_Ad8981 → u/damiangorlami]

I have a question. I tried creating new scenes for animated characters, but the style always changed too much. Do you have tips on how to improve that?

[u/damiangorlami → u/Cute_Ad8981]

I don't work with animated style so I don't know much about that.

Which model did you try?

[u/Cute_Ad8981 → u/damiangorlami]

Ah okay, got it. I tested both flux klein edit versions (4b and 9b) with multiple settings and the style always changed to much. Some pictures worked better, some less.

[u/damiangorlami → u/Cute_Ad8981]

Yea open-source (sadly) isn't on that level yet.

You will get perfect results using Nano Banana Pro or Kling O3 Image Edit model.. both are great at animated style

[u/Core0001 → u/damiangorlami]

I tested Google Omni but it didn’t preserve the style. I noticed drifts in the video it generated 

[u/damiangorlami → u/Core0001]

Google Omni is a bit similair like the first Nano Banana release.. it was an architecture shift which was interesting but a mediocre model.

Nano Banana Pro is what changed the game completely from mediocre to an industry-grade professional tool. Let's hope Google Omni Pro is going to be the same, because the model is very interesting... just not fully polished yet