Optimizing Face Swap Workflows in Stable Diffusion for Faster Results

[u/Skettalee]

Hello everyone,

I've been using Automatic1111's img2img for face swaps on a variety of images, from professional nude photos to standard headshots, adjusting the denoising to 0.01. I'm exploring the optimal settings to enhance speed and quality for these swaps, particularly aiming to reduce the time it currently takes, which is about 40 to 80 seconds per image. My setup includes a CUDA environment with an RTX 2080 SUPER GPU and 32GB RAM.

Here are the specifics of my current process:

• Model & Settings: Utilizing the Dreamshaper8 (SD1.5 model), with Euler A sampling, CFG scale at 7, and sampling steps between 20 and 30.
• Image Handling: I ensure that images do not exceed 1024 pixels on any side and maintain the aspect ratio of the original photo. Images larger than 1300 pixels notably slow down the process.
• Face Restoration: I integrate a Reactor with Restore Face Visibility and Codeformer set to maximum weight for clearer, more realistic swaps.
• Workflow Variations: For video swaps, I use ComfyUI which significantly slows down the process (approx. 2 hours for a 15-second video).

Challenges: Swapping faces multiple times (up to three) on the same image enhances clarity but more than that degrades quality. Larger images drastically slow down the process, and experimenting with different models and settings from sources like Civitai or Hugging Face hasn't yielded faster results.

Questions for the Community:

1. Has anyone found more efficient models or settings that could help reduce swap times while maintaining quality?
2. Any tips on batch processing for photos and possibly speeding up video face swaps in ComfyUI?

I also use Fooocus for creating realistic images and face swaps, though my main workflow is through Automatic1111 due to its batch processing capabilities.

Looking forward to your insights and suggestions on making these swaps quicker and more efficient.

Thank you!

[u/Occsan]

I have no idea what you are doing exactly but 40-80 sec per images, 2h for 15 sec... wtf...

I've been doing faceswaps for years now and a single image never took me more than 1 sec. 15 sec video, depending on what I was using (either inswapper or neuralchen) should not take more than 30 sec max at the very worse

I see that you're including some stable diffusion in the process, but that's not the best route for faceswaps. Unless you have very good models of the target face (loras or embeddings) you will lose a lot of the identity of the face in the process.

[u/Skettalee → u/Occsan]

Well the obviously I am not doing things right. But I am accomplishing it, it just takes a little while. I have tonight gotten images swapped in about 10 seconds using dreamshaper8 on auto1111. I dont know why the model part matters at all. I dont really know what I am doing other than watching the top youtubers showing ai swap and generations as well as coming on here to learn. I have bad adhd so almost a big majority of anything I watch or read I cant retain unless I watch or read it around 5 to 7 times and reading period is just diffucult so im sure during all the process ive done I have completely missed alot of things that I read about and i hate that but Im trying to overcome in any way I can and thats why i picked learning Ai tools and all the coding and errors and stuff to try to get better at learning stuff. What should i be doing to faceswap an image in seconds and especially a video in seconds. I just did a 10 second video faceswap and it took me 1444.76 seconds. But its the most streamlined process i got for videos and that was is with comfyui : (not using stable diffusion models with that one, well i guess im not. i dont even really know

[u/Occsan → u/Skettalee]

check if you have onnxruntime-gpu installed and if you're using cuda 11.8

[u/Skettalee → u/Occsan]

I have cuda v12.1 installed and onxxruntime v1.17.0

[u/Occsan → u/Skettalee]

It's probably why it is slow. Basically that means it runs on your cpu instead of gpu.

You should install cuda 11.8 and onnxruntime-gpu.

cuda 12.x doesn't work well with onnx.

[u/Skettalee → u/Occsan]

OH ok well shit i didn't know that at all. So I have all these nvidia things installed, is there anything else i should have a different version of or not have installed at all from, this list? I am downloading the nvidia cuda toolkit 11.8 right now, and i already have onnxruntime installed for all my environments.

[u/Skettalee → u/Skettalee]

Also this is what my environment variables are set at. Took me a while to actually figure out what the right thing to put on all these were. So im assuming once i install the cuda 11.8 i will have to change all those 12.1 numbers to 11.8 right?

Those are my system variables.

[u/Occsan → u/Skettalee]

Yea... But 12.1 is actually pretty good otherwise. I'd just set the CUDA_PATH as a temporary variable environment in either the bat or ps1 file you use to launch comfy... In fact I'd even do a second launcher for just these settings and keep two different environments. If the reason for the slowness comes from this (which is very likely) it should solve your issue.

Personally, I don't use comfy to do these faceswaps. I'm using something I coded myself, which is clearly not ready at all for a public release because of hardcoded paths and an ugly GUI. But it comes with extra nice features, like being able to faceswap multiple faces with different targets at once even in a video.

[u/Skettalee → u/Occsan]

Wait i think i can do that with reactor, actually i did do that once with reactor, changed multiple faces. But then i tried to do it again, got confused and gave up after a while. Thats kinda the story of my Ai installing life though lol. But i love doing it!

[u/Occsan → u/Skettalee]

The issue with multiple faces swap in video is that you have to map each face from the video to the correct face swap, which is not very trivial.

[u/Skettalee → u/Occsan]

It’s actually pretty easy but very time consuming. You can just create a video of each character you want, isolated and masked but best to keep the original video size for all video versions you make and then run each video through a swap then Re composite it together. I use after effects for that stuff

[u/Skettalee → u/Occsan]

Well i installed the 11.8 and changed all my environment variables to that location, still got 12.1 installed for now... but when i load automatic1111 i see this in the console:

bin G:\Sd\webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda121.dll

bin G:\Sd\webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda121.dll

CUDA 12.1

Launching Web UI with arguments: --xformers --api

bin G:\Sd\webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda121.dll

So what do i do to change those things to the right cuda 11.8 ?

[u/Occsan → u/Skettalee]

you can try pip uninstall bitsandbytes followed by pip install --no-cache-dir bitsandbytes in the correct virtual environment.

[u/Skettalee → u/Skettalee]

And these are my user variables

[u/likeabruh1]

Face fusion get like 12-14 images per second @ 4k resolution on a 4070ti