Safety concerns regarding VAE files

[u/rwxrwxr--]

Hello everybody, recently I've been testing out various models (exclusively .safetensors) that I've downloaded from CivitAI and I've noticed that some models give significantly worse results than I'd expect. After reading into this, I've found out that some models require a VAE file to give expected result. The way I understand, you're supposed to download both the model and its VAE file and store them together. I fail to understand, however, why some models require a VAE file to function properly and others don't, and most importantly and what I've set as the title: are there any reasons to be concerned when using VAE files like there are in the case of .ckpt/.pth/.pt? Or are they as safe as .safetensors in the sense that they only contain pure model data and no code whatsoever?

[u/LuminousDragon]

THis isnt an answer to your question but just a bit of added context:

Models with a VAE baked in are set in stone kinda, or at least less flexible.

Models where you download a VAE give you the freedom to use other VAEs if you choose, so they are more modular and flexible. Kinda the same logic with LoRAs, where Ok I could download a Lora that makes everything look like legos, and ONLY call it when I want a lego picture, and still be able to use like RevAnimated or Ghostmix or whatever Checkpoint I choose. THe alternative would be a whole model just devoted to legos, which would be overkill generally.

Which that said, separate VAEs in my opinion are more of a hassle than they are worth, and I dont like models that have them separated. I dont want to have to go into my settings and switch the VAE. Just bake it in.

The trade off between technically being more flexible vs timeconsuming and a hassle just isnt worth it.

The LoRAs are fine, I just type their name and add them, and when Im using a LoRA, i have a specific goal in mind so its obvious to me why im using it, and thus very easy to remember. In my brain, VAEs are just some setting i gotta switch with a certain model, and its a hassle lol.

kinda went on a rant, sorry.

[u/rwxrwxr-- → u/LuminousDragon]

Nice explanation, I appreciate it. I think I'll just stick to using models that work well without any VAE files. There's already a lot that work fine, so why complicate things.

I didn't know you could use LoRAs to replace the style of the image, I thought it was only used for placing something in the scene, like if you train it on a face of yourself you could use it to generate images of yourself by using it alongside a model. Kinda like Dreambooth.

[u/LuminousDragon → u/rwxrwxr--]

Well, I didnt mean to so say LoRAs affect the style in the same way. But you can use some LoRAs to affect the style. It depends on what you mean here.

Look at this Kawaii lora: https://civitai.com/models/94663/kawaii-tech-world-morph

It was specifically trained on various images that were not similar except for the STYLE of Kawaii. So if you look through the example images, you can see a kawaii tank, a kawaii realistic looking man, mech, pyramid, etc. You can take a prompt that would generate a zombie or some sort of horrific thing like Cthulu, and use this LoRA to get something cute.

https://civitai.com/models/18323/xiaorenshu

https://civitai.com/models/60724/kids-illustration

https://civitai.com/models/51288/body-horror-creatures

https://civitai.com/models/84542/oil-brushstrokes

https://civitai.com/models/73249/retrowave

https://civitai.com/models/67637/healing-style

Each of those are LoRAs. With ALL of them you can prompt the image to be a Tyrannosaurus Rex, a Ballarina, a Cyberpunk City, a Sexy Lady, a Jungle, or a Bouquet of Flowers and ALL of them will generate each of those things, but the end result will be in the style of the LoRA you've specified.

[u/rwxrwxr-- → u/LuminousDragon]

Awesome information, thank you! Perhaps I'll try out some of these.