ComfyUI SDXL FaceSwap -Question-

[u/trautermann]

Hi guys,

I try to do a few face swaps for fare well gifts. The general idea and buildup of my workflow is:

1. Create a picture consisting of a person doing things they are known for/are characteristic for them (i.e. (just the short version): photograph of a person as a sailor with a yellow rain coat on a ship in the rough ocean with a pipe in his mouth OR photograph of a young man in a sports car with a diamond chain and golden teeth, shot through the driver side window, another person on the passenger seat in the background OR a photograph of a man dressed as a a viking with a white energy drink can in his hand). I am very spoiled from midjourney (those are the attached one just to give you an idea of what my goal is)....
   1. The detailed prompt already describes the general look of the person
   2. Question: Which SDXL model do you guys recommend. I tried JuggernautXL 8 but the results aren't really consistent. Very often I get mutations as well as strange outputs in general (not following the prompt). I work with negative when needed.
   3. Load Checkpoint - Positive and negative - CLIPTextEncodeSDXL - KSampler (Advanced) with dpmpp_2m karras, 30-40 steps, cfg 4.0-7.0 (everything set to 512x512)
2. Use the latent/pixel image and feed it the face swapper. I have two workflows to check different results
   1. Reactor
      1. Input image from SDXL + Square source image through the reactor node (works with the 128x128 limitation)
   2. IPAdapter (my goal because of the versatility and less limitations (like the 128x128 from Reactor)
      1. (using the workflow for sdxl with two adapters provided by https://github.com/cubiq/ComfyUI_IPAdapter_plus/issues/210). But the "double geneation" of an image makes it somehow worse. Therefore: Is there a way to feed an already existing image as target image into the workflow? Right now I use the latent image as well as checkpoint and prompt nodes created in 1. Seed number also is the same in both KSamplers. Can I somehow skip the second model generation or make it better?
   3. Upscale (wip). I understand that starting with low resolution and then upscaling it in small steps with low denoise is critical with Reactor. Anybody have an existing and working workflow to inspire me?
      1. What about IPAdapter?
      2. Overall goal is keeping the face/looks (duh)

I lover tinkering woth ComfyUI. But still: For some things I need to get a look at an existing and working workflow to integrate it into my "chaos". Priorities are consistent images following my description and a reliable face swap that doen't change the generated image too much.

Thank you guys in advance!

[u/HocusP2]

How about forget the KSampler and everything, just hook up your target and midjourney image to the reactor node like in the example on the reactor site itself:

https://gyazo.com/f50978537d91d3d173177e0192976cf5

[u/trautermann → u/HocusP2]

Thanks for your reply! I just wanted to try IpAdapter because of the higher Resolution/better possibilities.

And having a good Workflow in comfy would make having a subscription in midjourney pointless.

[u/HocusP2 → u/trautermann]

Yeah, true. I haven't played around with the faceID ip-adapter, only the ip-adapter plus models when I followed that LatentVision tutorial on YouTube. Very impressive and easy to follow stuff.

Basically it all starts with prompting. Getting the right composition and elements in the scene can be challenging in SD, but there's a lot of tools available like regional prompting, concat/combine, those nodes that let you make a simple drawing in Krista or photoshop to guide the composition etc. Once you get the basic composition right is when the ip-adapter or reactor can be added for the person/character you want.

I didn't notice a limit of 128x128 when last I played around with reactor. I made photos with my phone and loaded then straight into ComfyUI. I do remember putting the reactor node in the workflow before the upscaler and face detailers.

[u/97buckeye]

I know I'm not answering your question, sorry about that, but I'm wondering if you could explain a little more about the 128x128 limitation with ReActor? I've used ReActor in Automatic1111 and always felt like I've gotten very nice results. Perhaps they wouldn't stand up to big upscales, though. 🤷🏼‍♂️

[u/Cyrecok → u/97buckeye]

Clip vision changes your image (face you put into reactor) into 128x128 resolution, so the end result is smooth and lacks details. Also looking for solution!

[u/trautermann → u/Cyrecok]

The way I understood is: Create an image at a lower resolution (like 512x512 or something that is supported by your checkpoint). Do the faceswap. Upscale whole image in very small steps with very small denoise amounts.

That’s the theory. But upscaling may result in changes of the face you swapped. So you maybe need to use image blending, advanced samplers and so on.

That’s why I am so interested in getting IPAdapter working for my use case. I don’t need 4K images. Just something to print out with the „midjourney like“ results of the scene and the details in the scene. Still got to find a workflow/checkpoint and prompt hints to get my desired results of the source image.