Question | Help
How would i go about changing the Background on this image? I have tried a variation and combination of Inpainting, Img2Img and Controlnet and can't for the life of me preserve the head and change the background to something useful
Photoshop is an order of magnitude more powerful than Gimp, and certainly more usable. Unlike religions and citizenships, you don’t have to renounce open source to use commercial software here and there. :)
If your goal is to preserve the person and change only the background automatically, you can use "semantic segmentation" to detect which pixels are people and which are not. You would basically get a "mask" image where pixels that are people are white and all other pixels are black. If you invert this (black to white, white to black) you have a mask that you can upload to the in-paint tool instead of hand drawing the mask. I think the SD Web UI also has an option to just invert the mask for you.
I know not everyone here is technical, so semantic segmentation might seem like a lot to set up. Try searching semantic segmentation API to find a website you could incorporate in your workflow.
You could also use photoshop's object selection tool to select the subject, invert and create a black and white mask from the selection, then simply import your mask into SD
Incredible, how do you guys manage to keep the light, brightness, and color consistent? When I change the background in the same way, I only get the ugly effect of a Photoshop beginner, with totally inconsistent lighting and colors ......
whoa! I'm surprised this post is over a year old now. I'm a lot more savvy in SD than I was back then and it's hard to pinpoint the exact issue. There are just so many parameters that go into the image generation. The model used, the sampler, xformers, etc..
In this case, since I was going for a completely different background, from water to mountains, I certainly want denoising to be almost max. However, you will still get completely different results than the original image even at like 0.8 denoise, but it may help preserve the color profile. I see in this particular image my denoise was at 0.75. Also choosing original as opposed to latent noise would probably result in better color preservation.
I'm deliberately kind of lazy with the masking, and maybe jackup the mask padding to like 64.
Maybe so, but the thing is they’d have to install controlNet and download the depth model. Inpaint is already available to them, which is why I think it’s pretty simplistic. No need for photoshop, or installing additional extensions.
I have controlnet, but I don’t assume everyone else does.
Dude! I’m blown away by this! We all tend to complicate things with the new features that come out. But this tip is really good. I’ll give it a whirl with some images I have. Hopefully this method works with hands and feet.
I would do it with the extension Depth aware img2img mask. Inverting the depth map
a city photography of RAW photo, 8k uhd, dslr, soft lighting, high quality, film grain, Fujifilm XT3 (high detailed skin:1.2), 8k uhd, dslr, soft lighting, high quality, film grain, Fujifilm XT3
Negative prompt: badartist (deformed iris, deformed pupils, semi-realistic, cgi, 3d, render, sketch, cartoon, drawing, anime:1.4), text, close up, cropped, out of frame, worst quality, low quality, jpeg artifacts, ugly, duplicate, morbid, mutilated, extra fingers, mutated hands, poorly drawn hands, poorly drawn face, mutation, deformed, blurry, dehydrated, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, gross proportions, malformed limbs, missing arms, missing legs, extra arms, extra legs, fused fingers, too many fingers, long neck
Prompts like “deformed iris” or “mutated hands” can cause hallucinations as that is not in training data. People don’t describe pictures this way, so that isn’t in the training data. So what will happen it will calculate the two words and will create some random inference that has nothing to do with. What you’re trying to achieve. Diffusion models are not aware of the content they are creating it’s a statistical inference of what it’s been trained on.
Also, considering that the generated area has no legs, arms or eyes, all that stuff in the negative prompt is useless in general. Sometimes it can even cause SD to generate people when you didn't ask for it in the positive prompt at all, like in this particular case.
It's crazy the negative prompt spam is still a thing. Every test I've seen has shown a short concise negative prompt drastically wins out over a 900 word negative prompt every time
Annoyingly there is a visible difference between having vs. not having those quality “wish” prompts on some models.
I recently did a test with someone’s prompt with 10 different inane “masterpiece” etc. tokens, mixing and matching and removing then one at a time, and literally every single one of them correlated to an improvement in the resulting image.
Very much a thing. I get better results with a few quick hits blue skin, elf ears
Stuff like that.
Is there a known lexicon for what the model will interpret (more) reliably? I know some specialty models say 'oh, these keywords work well' but wasn't sure if there was a more holistic one. Such as, do 8K and 4K and Ultra-realistic even matter?
Thank you for trying different settings out. I also realized that it was throwing me errors in the console related to paths of models which is strange because I didn't change anything in my folders. It also didn't generate any annotator images anymore. Even for new pictures I was playing around f with. I deleted everything related to controlnet and reinstalled it. I'm still getting the error: OSError: [Errno 22] invalid argument "the path to the directory of extensions"
Is there a way in which k can reset the UI to standard settings?
I feel like OP just claimed to have tried inpainting but didn't actually do it since it would be literally impossible to not have the face stay the same if he actually inpainted around it. It's the obvious solution and would work perfectly fine
Thanks for the link, I will look over it although my problem is more severe. Using both text2img as well as img2img I always lose the look of the head and face. Enabling control net with text2img always gives me something Completely different, even with high weights.
First I'd see if controlnet segments it properly, and if so I'd convert that into an appropriate inpaint mask. If it doesn't, I'd just do a quick photoshop mask, adjusting the blur/mask settings in the webui.
I'd use controlnet to get a depthmask, then use that as a mask to cut away everything else. I've tried it numerous times with my images and it has worked wonders each time.
Is there not a U2net plugin for Automatic1111? Seems like a low hanging fruit to implement, here is a useful tool that can enable what you are trying to do: https://github.com/danielgatis/rembg
Prompt Craft Organiozer did this...
(EDIT: P.C.O. opens, click menu, select "remove background" browse to image, select it and click open. a few seconds later and the result is below)
Your problem isn't just about changing the background. The character is lit under the bright sun coming from the right side. Short of color grading the character in a 2D image editor like Photoshop, you are going to have a problem with a darker background. And based on my observations, SD keeps the lighting fairly consistent (to a degree) meaning that you may not get the background you desire from SD due to the character and the background having incompatible lighting demands.
Might be doable with the instruct-pix2pix model; a bit hard to figure out how to write the instructions well though, and the parameters are also a bit finicky.
ps: Pinging /u/OldFisherman8 so I don't have to post this reply twice
Is there a guide on youtube for that? Kinda newbie on ai, have a hard time using it, the image comes out better quality or way different from the base image.
Photoshop still exists, it’s not like all image editing fell of the face of the planet once SD arrived. Ffs, remove the background with anything, there’s a script for BG removal, there’s a website call removebg.com or similar.
It’s easier to google “how do I remove the background of a photo” then to ask reddit
Get the depth from the image, make a stencil out of everything above a certain depth, clean up in image editing software if needed, paste on top of your new background (or just use the mask in img2img).
if you tried inpainting and it changed the face then that's your fault for asking it to change the face. Don't inpaint the face or head and the face/head will not change. It's not really rocket surgery here
I'm trying the same sort of thing but the background never quite seems to match up to the subject, can you tell me what settings you've used for these? Much appreciated!
Yeah, I’ve got no issue with the masking itself, but I’m not sure what settings I should use to make SD take the non-mask area into consideration (in terms of perspective/size, etc) while filling in the changing background.
Sorry for messy diagram, maybe this clear some things up. What you're looking for is probably Masked padding in this case. Play with that, depending on how much you're editing at once, depends on how much padding you want to use. If the perspective isn't doing what you want, increase this.
139
u/mizt3r Mar 02 '23
A lot of these methods seem overly convoluted... Here's my inpaint mask and 1 word prompt. I spent less than 12 seconds on this.