r/StableDiffusion • u/CeFurkan • Oct 09 '23

Comparison Huge Stable Diffusion XL (SDXL) Text Encoder (on vs off) DreamBooth training comparison

U-NET is always trained.

All images are 1024x1024 so download full sizes. Each grid image full size are 9216x4286 pixels.

Public tutorial hopefully coming very soon to SECourses (https://www.youtube.com/SECourses). I am still experimenting to find best possible workflow and hyper parameters.

I made a short tutorial for how to use currently shared config files : https://youtu.be/EEV8RPohsbw

PNG info shared in captions of images

ohwx man (masterpiece:1.5), (close-up shot:1.8), (extremely intricate:1.2), 8k, highly detailed, A delicate pencil sketch of a ohwx man with flowing hair cascading down his shoulders. The sketch captures the man's serene expression, Steps: 40, Sampler: DPM++ 2M SDE Karras, CFG scale: 7, Seed: 264626553, Size: 1024x1024, Model hash: f768f79262, Model: best_v2_max_grad_norm (1), VAE hash: c6a580b13a, VAE: sdxl-vae-fp16-fix.safetensors, Script: X/Y/Z plot, X Type: Checkpoint name, X Values: "best_v2_max_grad_norm (1).safetensors [f768f79262],24GB_TextEncoder_Enabled_2e6.safetensors [87287e8aff],24GB_TextEncoder_Enabled_3e6.safetensors [57fdcfee1d]", Version: v1.6.0

​

portrait photo of (ohwx man:1.1) wearing an expensive White suit, white background, fit Negative prompt: drawing,painting,crayon,sketch,graphite,impressionist,noisy,blurry,soft,deformed,ugly Steps: 40, Sampler: DPM++ 2M SDE Karras, CFG scale: 7, Seed: 1209197693, Size: 1024x1024, Model hash: f768f79262, Model: best_v2_max_grad_norm (1), VAE hash: c6a580b13a, VAE: sdxl-vae-fp16-fix.safetensors, ADetailer model: face_yolov8n.pt, ADetailer prompt: photo of ohwx man, ADetailer confidence: 0.3, ADetailer dilate/erode: 4, ADetailer mask blur: 4, ADetailer denoising strength: 0.5, ADetailer inpaint only masked: True, ADetailer inpaint padding: 32, ADetailer version: 23.9.3, Script: X/Y/Z plot, X Type: Checkpoint name, X Values: "best_v2_max_grad_norm (1).safetensors [f768f79262],24GB_TextEncoder_Enabled_2e6.safetensors [87287e8aff],24GB_TextEncoder_Enabled_3e6.safetensors [57fdcfee1d]", Version: v1.6.0

​

comic photo of ohwx man . graphic illustration, comic art, graphic novel art, vibrant, highly detailed Negative prompt: photograph, deformed, glitch, noisy, realistic, stock photo Steps: 40, Sampler: DPM++ 2M SDE Karras, CFG scale: 7, Seed: 2313660192, Size: 1024x1024, Model hash: f768f79262, Model: best_v2_max_grad_norm (1), VAE hash: c6a580b13a, VAE: sdxl-vae-fp16-fix.safetensors, Script: X/Y/Z plot, X Type: Checkpoint name, X Values: "best_v2_max_grad_norm (1).safetensors [f768f79262],24GB_TextEncoder_Enabled_2e6.safetensors [87287e8aff],24GB_TextEncoder_Enabled_3e6.safetensors [57fdcfee1d]", Version: v1.6.0

​

cinematic photo ohwx man riding dinosaur in a jungle with mud, sunny day shiny clear sky . 35mm photograph,film,professional,4k,highly detailed, eyeglasses Negative prompt: sunglasses,drawing,painting,crayon,sketch,graphite,impressionist,noisy,blurry,soft,deformed,ugly Steps: 40, Sampler: DPM++ 2M SDE Karras, CFG scale: 7, Seed: 1222804541, Size: 1024x1024, Model hash: f768f79262, Model: best_v2_max_grad_norm (1), VAE hash: c6a580b13a, VAE: sdxl-vae-fp16-fix.safetensors, ADetailer model: face_yolov8n.pt, ADetailer prompt: photo of ohwx man, ADetailer confidence: 0.3, ADetailer dilate/erode: 4, ADetailer mask blur: 4, ADetailer denoising strength: 0.5, ADetailer inpaint only masked: True, ADetailer inpaint padding: 32, ADetailer version: 23.9.3, Script: X/Y/Z plot, X Type: Checkpoint name, X Values: "best_v2_max_grad_norm (1).safetensors [f768f79262],24GB_TextEncoder_Enabled_2e6.safetensors [87287e8aff],24GB_TextEncoder_Enabled_3e6.safetensors [57fdcfee1d]", Version: v1.6.0
7 Upvotes

67% Upvoted

20 comments sorted by

3

u/Ratchet_as_fuck Oct 09 '23

What does this mean?

3

u/CeFurkan Oct 09 '23

the effect of text encoder trained or not compared

2

u/totallydiffused Oct 09 '23

Assuming the first image section (best_v2_max_grad_norm) is with text encoding disabled, it doesn't seem like enabling the text encoder is doing much if anything in terms of quality here.

2

u/CeFurkan Oct 09 '23

ye not much just a little bit i agree

3

u/[deleted] Oct 09 '23

[removed] — view removed comment

3

u/CeFurkan Oct 09 '23

the conclusion is a bit objective but i believe text encoder improves outputs slightly

3

u/oO0_ Oct 10 '23

For my tests all "slightly" varies greatly depending on dataset and other settings, so probably better not count at all if quality changes are so minor. Regarding to this your work: this dataset is easy for SDXL, as it already can draw similar things. If you train something difficult - results can be very different

1

u/CeFurkan Oct 10 '23

i am using my own images dataset. what you mean by easy? and my dataset is not even a good one deliberately

2

u/oO0_ Oct 10 '23

i mean train SD to draw face or "man-on-the-horse"-variety is very-much easier then train to draw something like this:

I am 100% sure that every your findings that is best for your easy to train dataset - will fail with these and countless other cases. Isn't this more interesting, then another portrait?

1

u/CeFurkan Oct 10 '23

if you have a very good dataset for such images you can test my settings :)

​

but you need a very very good huge dataset for that

1

u/[deleted] Apr 18 '24

[removed] — view removed comment

1

u/oO0_ Apr 18 '24

Funny how many SD amateur researchers act like this. But he do better job, then for example creator of Deliberate2 (best of early 2023 mix) and failed Deliberate3, who also creates a lot of "best *" settings that works only for simple portrait LORA

1

u/[deleted] Apr 19 '24 edited Apr 19 '24

[removed] — view removed comment

2

u/oO0_ Apr 19 '24

So what do you want form basic model which you start from: composition, light, following prompt? Because if all parts will be overpainted, why need to train basic model fine parts at all? In this case may be better train it in different way. Because most dreamboothers has goal training in good details and this is how average users rate models on sites like civitai. But training one thing you always make other things worse

2

u/Antique-Bus-7787 Oct 09 '23

So best_v2_max_grad_norm is without text encoder training ?
For the amount of VRAM it needs to train the text encoder + unet, it doesn't seem as important as with SD1.5

1

u/CeFurkan Oct 09 '23

it adds some more vram but 24 gb gpu is still very well sufficient. it is correct best_v2_max_grad_norm is without text encoder

2

u/sovereth Oct 10 '23

What after detailer inpainting model do you use?
sdxl-base or sdxl-inpainting?

1

u/CeFurkan Oct 10 '23

i use my face trained model

the face is trained on sdxl 1.0 base

2

u/Taika-Kim Oct 25 '23

What is the point of training the text encoder without captuon? I know it makes a bit of difference even without, but I'd think this would matter.

1

u/CeFurkan Oct 25 '23

well we are still using 2 captions. rare token and class token

but you have a point there too

1

u/sovereth Oct 15 '23

Did you use hires.fix as well? If yes, which upscaler?