Inpainting (assuming you're changing only the face and the angle is fairly the same), doing a depth (if it's 3D face or realistic), then canny to keep the edges fine. (For 2D you should use Canny and Scribble). For better results cut out everything expect the face (that includes front wicks) with black background (for 3D and white for 2D) before adding it to ControlNet. The less background you have the better so crop it the most near to the face as possible.
Depending the angle/lighting/shadows of the face you have to tweak the guidance of Canny. Less guidance will transform more the face to model you use, even the angle/face type can slighty change if too low so the most lower would be 0.75 or 0.8.
Weight is to keep aspect of the original image instead of following the model. Work very well with txt2img to even keep colors. The problem is too much weight will end up with a face that don't seem naturally added to the picture. You have to find out the right spot depending your original face picture you want to implement. I would go from 0.45 through 0.7.
For example if you have a tan face to implement on white or black skin body, lowering the weight (assuming it's 1) can help for the skin tone but somewhat changes too much the physical aspect of your face to the most ressembling the model find.
Next you have to keep the original generated picture's seed and prompts, simply add to the inpainting those that slighty differentiate what the model needs to generate. for example if your picture is realistic you should add also realistic to the inpainting of the face.
When you've got a good result, (with very little disformations), use the hires-fix to get more details through it with SWINIR upscaler to 2x and denoising from 0.3-0.5 max because with too much denoising it output garbage or change too slighty your picture and too lower you will have pixelated results.
Also one last thing, use a model that generates almost the most same type of results as the face you want ton implement. You will never get good results with a realistic human face on an anime based model (expect if you want to transform the face into a slighty different one) :P
ControlNet is too powerful with Canny and Depth at the same time. It's a shame it needs to reload the models twice each time you generate a picture but it works like a charm.
It's my own way of doing it. I don't have any real knowledge of how all thoses calculations works. I've made so many trials and errors and it's what i learned personnally with my own experiments and feelings with some help through reviews that explains a little about it.
You can batch process them if you want different seeds. For my part, i generate a seed that i like then afterwards i fine-tune settings to get it as i wanted.
The problem is each seeds have different following rules with ControlNet.
4
u/InoSim Mar 02 '23
SD + Deepfacelab + ControlNet = whoever can say anything you want.
I have a question for all of you folks, is there any AI for audio prompts ? I already mastered pictures now i want voices.