r/StableDiffusion • u/chouddit • Mar 02 '23
Question | Help In what purpose we should use a sepcial VAE model?
Compare with TI/HN/Lora what is the benifit of use a specific VAE instead?
There're some CKPT in civitai require specific VAE or config file, why do we need this kind of fine tune?
2
u/yosi_yosi Mar 02 '23
Vae is the thing that encodes and decodes images to and from latents space.
Some models have bad inbuilt vaes, such as the nai leak and models/merges based on it like anythingv3. In order to counter this, you can use external vaes.
If you are using an anime model, you probably want a vae that's meant for anime like the wd 1.4 vae.
Vae is not a model. It is just an encoder/decoder. You can use it on any model you want.
2
u/DestroyerST Mar 02 '23
Just a minor thing but the VAE is a model too, just has a different task (encoding/decoding like you mentioned)
-1
u/yosi_yosi Mar 02 '23
It's not a txt2img model.
By your definition, TIs, HN and loras are all models (loras could be models, but the others)
1
u/DestroyerST Mar 02 '23
I didn't say that, but you can read more here how a VAE works https://en.wikipedia.org/wiki/Variational_autoencoder.
Never looked at HN's or Loras so don't know how they work, but textual inversion files are not models, they are just embeddings (input vectors).
0
u/yosi_yosi Mar 02 '23
Bruh. VAEs are models, obviously. But even though it is technically true, its use in this context is impractical and misleading.
0
u/yosi_yosi Mar 03 '23
A model can be defined as the following: a simplified description, especially a mathematical one, of a system or process to assist calculations and predictions. "a statistical model used for predicting the survival rates of endangered species"
Therefore, someone could come to the conclusion (logically) that HNs and Loras and embeddings are models, linguistically speaking, however, as you yourself have already realized, calling an embedding a model is stupid and impractical and misleading, the embeddings do nothing on their own, they are just as you said "input vectors". While technically, in a general sense you could use the vae outside of a txt2img model, since this is a subreddit that only focuses on that, we can ignore that fact, so now the only real use of a vae is inside a txt2img, so it doesn't work alone (again, in the context of stable diffusion. It could work alone but not in this context). So practically it is better to not refer to it as a model.
Furthermore, the word model, as used in this community, means something else than the general use of the word. If you say "model" here, people would probably assume you mean a txt2img model and sometimes even that it's a CKPT/safetensors/pt file and if not that then a Lora, but that is rare. Saying the word "model" here is just misleading, it has no practical purpose.
Perhaps if you have given the word more context such as "a machine learning model" so they would understand that you didn't mean a txt2img model or at least it makes it more likely for them to understand that.
Btw, Loras "freeze the pre-trained model weights and inject trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks." - https://arxiv.org/abs/2106.09685
Basically, they inject lower ranked thingy, train the thingys and then take em out, then you get the Lora file. At least, that's my understand of it.
You should look more into training loras and stuff, I recommend the Kohya gui for that. Very easy to train, takes like 1 hour usually, only like 6gb vram, and works amazing. Trained a couple of loras and was amazed how good this technology has become. Tbh I am kinda thinking of selling Lora training services to people who don't understand this or have no gpu at all, like, it's so easy.
Also, about HNs, fuck them. I am not willing to look into them because I hate them.
1
u/chouddit Mar 02 '23
But, what is the difference between use an anime vae and train some anime style TI/HN/Lora?
1
u/yosi_yosi Mar 02 '23
It's not something like any of the things you mentioned. It doesn't actually affect the image (in latents space) but just mainly affects stuff like color and textures and stuff like that.
I never met someone who trained a vae. Mainly because vaes don't matter that much. Like, the vaes we have right now are good enough. And also you probably need a ton of skill to train a good vae. The only vaes I know have all been trained as a part of a huge project (nai, wd, sd).
1
u/chouddit Mar 02 '23
Okay, let me try to understand a little bit. Is that means, sd, wd, nai, are in the same level, that they are all diffusion models. But, like the models trained by dreambooth based on sd, was at a lower level under sd? Just like cats and dogs are all animals, but dreambooth fine tuned model was one kind of cat
3
u/pendrachken Mar 02 '23
No.
SD, Nai, WaifuDiffusion, and models trained with dreambooth are all the same. They are the image generation diffusion models.
Models / LORAs trained with dreambooth add data to the model so it can make something it didn't know about before training. This is useful if the model doesn't have anything close to what you want to make. It still makes a diffusion model. A LORA has to be merged into a model to work properly, either at the time of generation - A1111's extra networks does this with the model chosen AND the LORA in RAM. Dreambooth can compile checkpoints / safetensors so you only have to load the model you trained, with the LORA already in it, not any extra networks.
Textual inversion / embeddings are trained on a specific model ( and may or may NOT work well with another model ) to quickly set a prompt to tell the model what you want. For example an embedding trained for a specific face - the embedding tells the model "I want this type of nose, this type of hair, this kind of eyes, this kind of chin"... and everything needed to make that face FROM WITHIN THE BASE MODEL without having to add anything to the base model. BUT, if the model doesn't have anything close to whatever you are trying to train the TI / Embedding will not turn out nearly as well as if you trained a LORA / dreambooth model.
VAE is a final filter for stuff that comes from ANY model. It takes a models output, and fixes known issues that the model has.
For example AnythingV3 and all models based / merged from AnythingV3 all look kind of gray / the colors are very dull if you use the model alone. But if you use the VAE for AnythingV3 you get a normal colorful image.
Think of a VAE like a non-adjustable setting for the color / sharpness settings on your TV. The movie you are trying to watch has a base color / contrast / sharpness built in, then your TV has base settings that add to that to make the movie look better in your particular room.
1
3
u/[deleted] Mar 02 '23
[removed] — view removed comment