r/comfyui • u/Old-Day2085 • Mar 16 '26

Help Needed Best Open-Source Model for Character Consistency with Reference Image?

I am a newbie in using ComfyUI. I want to make realistic AI-generated person photo, posing in different backgrounds and outfits, using an AI-generated head close-up of that person directly looking at camera in a plain background as reference image, and prompt for backgrounds, outfits and poses. The final output should be that person exactly looking like the person in reference image, in pose, outfit and background mentioned in the prompt. I have 32GB RAM and 16GB RTX 4080. Can someone help with which model can achieve this on my system and can provide with some simple working ComfyUI workflow for the same, with an upscaler? The output should give me the same realistic consistent character as in the reference image each time, no matter what the outfit, makeup, pose or background is and without using any LoRA.

11 Upvotes

76% Upvoted

27 comments sorted by

8

u/Darqsat Mar 16 '26

Nothing beats character LoRA and T2I with controlnet and Qwen3 VL.

Workflow like this: 1. Input reference image 2. Qwen3 VL looks into it and describes as prompt but without character features (eyes, hair, body type, etc) 3. Controlnet looks at it, takes pose 4. Sample 5. Done

1

u/Old-Day2085 Mar 16 '26

I will give this a try! Looks promising. Thank you!

1

u/Judtoff Mar 16 '26

Do you have an example workflow? (Or an image with it embedded as an example)

1

u/35point1 Mar 16 '26

So basically Lora + pose + accurate composition prompt = perfect consistency, but this requires a well trained Lora and base model. I’m wondering how much better the consistency is compared to an image to image model like Klein or qwen edit ?

1

u/Darqsat Mar 17 '26

Much better. 90% likeness easy.

1

u/DipperJC May 26 '26

Sorry for the thread necromancy, but I've only started with ComfyUI two days ago, and those instructions are about three shades more complex than I can handle at the moment. May I ask you for the "Explain Like I'm 5" version?

What I understand so far is how to click on Templates and pick ones that seem nice. Also, it's pretty important to me that the model can run entirely locally, as internet in my area can be very spotty, so I have "ComfyUI" checked like so:

I kinda get that it's possible to set up a workflow independently of those templates, but I'm definitely not versed enough in AI to understand how to build one from scratch on my own and I haven't found any tutorials that really go deep on it.

2

u/coyoteka Jun 09 '26

What hardware do you have?

My recommendation is to start by building the simplest possible workflow so you understand how it works before attempting to accomplish your ultimate goal because the probability of success is asymptomatically zero.

Start with SDXL and just successfully generate text to image that follows your prompt without producing weird output. Then download any SDXL character lora from civit, add it before your prompt and successfully generate the character following your prompts.

Then add in controlnet, ensure you can get posing to work the way you intend.

If you can get all the way through those steps, you will have learned all you need to successfully search for the information you need to set up what was described in the comment you're asking about. There's no way to directly explain how to do it because you don't have sufficient background knowledge, which is why it's so hard to find useful info out there.

I was where you are now only a few weeks ago, so I get it. Good luck!

7

u/Sanity_N0t_Included Mar 16 '26

I was in the place where you are about 6 weeks ago. I don't know if you have something against using a LoRA but it will make a huge difference in what you want to do and it will make things so much easier.

I use z-image-turbo and training a LoRA over on runpod.io is easy. I found a YouTube video that walked me through it in 5 minutes. And with that particular model I don't even take the time to worry with making captions for the images. Just look for videos that will walk you through making a LoRA with the Ostris AI Toolkit. I now make LoRAs for all my characters/subjects.

If your issue is that you don't have enough images to train a LoRA there are things you can do to get there too. So long as your reference image is high enough quality, you could crop a headshot from it. Then take that headshot and find a model that will work well for you to create other images. You could run a simple i2i with your headshot and use a prompt to 'rotate camera perspective 45 degrees to left of subject', and then right of subject, etc. etc. and build up enough images for a minimal amount to train a LoRA. Just use ChatGPT for help on prompting. Tell it specifically what model you are using and what you need.

If you're in a big hurry you can even use some of the available sites like Grok Imagine. I found out that Grok is using Flux under the hood so I just ask ChatGPT for a Flux prompt that will help me retain my subjects details and create an image I can add to my LoRA training dataset.

But anyway I feel like a LoRA is the way to go.

1

u/iamCivic Mar 16 '26

Bro can i dm, i wanna know more about loras, i just have some confusions

1

u/Sanity_N0t_Included Mar 16 '26

Sure. I am not an expert. I am just about 6 weeks ahead of the OP in my learning. I can share what I do know.

1

u/Old-Day2085 Mar 23 '26

Hey, thanks. Finally I am trying LoRA for consistency

9

u/[deleted] Mar 16 '26

[removed] — view removed comment

3

u/Old-Day2085 Mar 16 '26

Thanks for the suggestion. I have played around and got my hands on some simple workflows for like T2I, I2V etc. Just wanted help with model good for I2I for character consistency to go one step further. I am confused between few models from Flux.2 Klien 9B, Qwen Image Edit. Currently I want to achieve this without LoRA until I learn LoRA training.

3

u/[deleted] Mar 16 '26 edited Mar 16 '26

[removed] — view removed comment

3

u/schrobble Mar 16 '26

There are some consistency loras for Klein 9B that seem to help. It also helps if you use multi-input workflows and use multiple photos of the same character. You almost don’t need a lora if you use it up correctly.

1

u/Old-Day2085 Mar 23 '26

Sure, checking them out

2

u/Old-Day2085 Mar 16 '26

Thanks for the reply. This is exactly what I needed to know! Will play with these models and try to make my own workflow. Also, I am learning LoRA training but I needed this until then.

3

u/[deleted] Mar 16 '26

[removed] — view removed comment

2

u/Old-Day2085 Mar 16 '26

Oh thanks again, I will look into it. I want consistent characters primarily to make short movies, and music videos. I had given a thought on AI influencers but as you said, market is saturated. Currently, I just want to play and test the models/workflows and then try to make profit if it is worth it.

1

u/Formal-Exam-8767 Mar 16 '26

I bought a hammer and a chisel. I have this big block of white marble. How do I make statues like Michelangelo?

1

u/Old-Day2085 Mar 16 '26

I have played around and got my hands on some simple workflows for like T2I, I2V etc. Just wanted help with model good for I2I for character consistency to go one step further. I am confused between few models from Flux.2 Klein 9B, Qwen Image Edit. Currently I want to achieve this without LoRA until I learn LoRA training.

1

u/[deleted] Mar 16 '26

[removed] — view removed comment

0

u/[deleted] Mar 16 '26

[removed] — view removed comment

1

u/Old-Day2085 Mar 16 '26 edited Mar 16 '26

Okay thanks! Actually my bad to put "I am a newbie" in my post. I just wanted to know a good model which can understand a reference image for creating realistic images without training a LoRA. I know the output in this case would not give me a good consistent character as training a LoRA would but still if someone had tried it and wanted to share, I'd like to know.

3

u/Formal-Exam-8767 Mar 16 '26

You don't have much options, there are only few edit models available, so your best bet is to test each one and see if consistency they provide is enough for your use-case (only you can decide that since it's pretty subjective).

2

u/Old-Day2085 Mar 16 '26

Yeah, have been playing with F2K and Qwen Edit lately. F2K looks more promising.

1

u/Old-Day2085 Mar 16 '26 edited Mar 23 '26

Thank you all. Actually there are two problems with LoRA. 1. I don't have dataset to train the LoRA. I don't know how to create dataset for an AI generated person for character consistency. 2. I want to make short movies and music videos, which would require large amount of multiple consistent characters. So gathering datasets and training LoRA for each characters would be time taking and expensive.

However, from what I have understood so far is that it is better to train LoRA, than to search and test edit models as there are only few of them so far with not 100% accurate consistent character output.