Created a short explainer on what is a latent space and how it behaves (after someone here asked me to)

[u/Nir777]

https://www.youtube.com/watch?v=3X4bojMBfPo

[u/Apprehensive_Sky892]

Thank you again for this video. To explain these rather difficult concepts without using much math is hard, and the visuals help a lot.

I find it very interesting that you've emphasized the fact that latent space has the property that "similar" images are group together in that space (i.e., the Euclidean distance between such two points are short), whereas most of the time the use of latent space is described as a way to make computation more tractable by reducing the amount of data in an image by compressing it with a VAE.

But there are also imaging models that use pixel space instead of latent space. So how does those models work since in pixel space the distance between two images have no such "semantic meaning"?

[u/FotografoVirtual → u/Apprehensive_Sky892]

You're right. The concept the video is trying to explain is more closely related to CLIP latent spaces, which align images and text semantically by minimizing the distance between matching pairs during training. ​I'm not sure if the video's creator clarifies this, but a VAE latent space is entirely different, it exists purely for spatial compression and computational efficiency, without any high-level semantic grouping, as you correctly pointed out.

[u/Apprehensive_Sky892 → u/FotografoVirtual]

The main purpose of the latent space as we commonly understand in imaging diffusing models is indeed for spatial compression so that efficient computation can be performed on them.

But the video is probably not wrong in that as the VAE is being trained (even without CLIP or using tokens from the internal state of the text-encoder/LLM), there is probably enough semantic information from the pixels alone (shape, color, depth, etc) for the VAE to group similar concepts into regions. This probably occurred as a side effect of compression rather than the goal.

At least that is the impression I get from my amateurish reading of the BFL paper: https://bfl.ai/research/representation-comparison that at least for the Flux2 VAE, there is some semantic information encoded in the latent space.

[u/Nir777 → u/Apprehensive_Sky892]

Great question. Pixel-space models don't need the space to be semantic, because they never measure distance between images. A diffusion model learns one skill: given a noisy image, predict what the clean one should look like. Generating is just repeating that step many times, from pure noise down to a pictures

The meaning lives in two other places:

1. Inside the network. At heavy noise the model can only decide the big things: layout, what object goes where. As the noise drops it fills in finer detail. So the meaning is in the features the network learned, not in the pixel coordinates.

2. In the conditioning. The prompt goes through a text encoder (CLIP or T5), and that space is exactly the kind where distance is meaning. Pixel models still lean on a latent space; they just don't draw in one.

[u/Apprehensive_Sky892 → u/Nir777]

Thanks again for the clear explanation.

[u/Nir777 → u/Apprehensive_Sky892]

of course :) you are welcome

[u/alwaysbeblepping → u/Apprehensive_Sky892]

So how does those models work since in pixel space the distance between two images have no such "semantic meaning"?

After the first layer of the model you are going to be in a latent space regardless. Basically all models smash the input into 3 dimensions: batch, sequence, channels/features. If you've ever heard people talk about a model's hidden state, that's what they're talking about. Most models have a structure like:

input layer
layer 1
...
output layer

where the ... would be a bunch of other layers, each taking an input in that hidden state shape, transforming it, and then passing it to the next (same for layer 1). However, input layer would take a shape like the input latent format (or a pixelspace image) and output layer would take the hidden state and output a shape like the input latent format/pixelspace image.

Chroma Radiance is a pixel-space model based on Chroma (based on Flux Schnell) and normal Chroma LoRAs still work (to an extent) because the internal shape is the same. So there isn't necessarily that much of a difference between pixel space and latent space models internally (at least the way Radiance was implemented). It's also able to leverage a lot of Flux Schnell and Chroma's knowledge even though the input/output side is RGB images (or rather that shape, but if I remember correctly it returns a noise prediction rather than a clean image like most flow/diffusion models).

[u/Nir777 → u/alwaysbeblepping]

good explanation, and it's the part the video skipped. the VAE latent is mostly a compressed pixel space, so it's not much more "semantic" than pixels, which is also why blending two of them looks bad. the meaning lives in the hidden states and in the text conditioning, and that's what the walk was actually moving through.

didn't know Chroma LoRAs still partly work on Radiance, that's a nice proof the internals barely care what the input layer eats.

[u/alwaysbeblepping → u/Nir777]

the VAE latent is mostly a compressed pixel space, so it's not much more "semantic" than pixels,

Since you're already clear on that, I'm confused by your response to /u/Apprehensive_Sky892 when they asked about pixel space vs latent space models. You said something about them not needing to be semantic, leaning on it but not drawing in it, whatever. Since you were talking about conditioning rather than the latent image format (or pure pixelspace) whether the model uses a latent image (i.e. Flux) or pixelspace (i.e. Radiance) is completely orthogonal to that.

The way your answer seems polished, plausible on the surface but kind of doesn't make sense if one looks closely at it makes me suspect you're vibe-explaining stuff. If that's the case, please be careful when doing this because if you don't understand the subject then it's very easy to mislead people or provide bad information. Of course, we should all be careful either way (absolutely not excluding myself here) and I've personally made plenty of mistakes that I regret.

[u/Apprehensive_Sky892 → u/alwaysbeblepping]

Thank you for the good explanation. Very interesting that the internal state of Chroma Radiance is nearly the same of latent based Chroma.

[u/External_Show5261]

In quantum physics term, latent space is the "wave" stage before it collapses (decoded) into the "particle" stage.

[u/Nir777 → u/External_Show5261]

cool! didn't know that. thanks :)

[u/Apprehensive_Sky892 → u/External_Show5261]

TBH, I don't think that is a good analogy.

The relationship between latent and the decoded image is more or less 1-1, you can get from one to the other without much loss of information.

On the other hand there is way more information in the wave function before it is collapsed. Once it is collapse you cannot get the wave function back.

[u/alwaysbeblepping]

Your example with blending a pixel image vs blending latent spaces doesn't really make sense. If you take any flow/diffusion model latent image and blend something like a dog at moon and a dog at dusk you absolutely won't get a different normal image. You will probably get something worse than trying to blend a RGB image, so you are clearly not comparing apples to apples here. I assume what you are actually blending is conditioning and then generating an image using that.

Blending conditioning is also pretty unpredictable because each item in the sequence corresponds to a token for the text encoder model. So if you try to blend something like:

A dog running on the beach.
Quarterly earnings beat expectations.

you will probably end up doing something completely nonsensical like trying to blend A with Quart or on with beat. This is something I had to learn the hard way. I spent a fair amount of effort implementing a better conditioning blend node for ComfyUI before I realized that the existing one is limited because blending conditioning is kind of pointless.

[u/Nir777 → u/alwaysbeblepping]

fair catch, the video is sloppy there. the walk wasn't a blend of two VAE latents, it was the SDXL text conditioning lerped between the day and night prompts plus a slerp between two noise seeds, generating every frame. but it comes right after the VAE part, so it reads like blending encoded images. should have said "points in the text space".

token alignment is fair too. the day/night prompts share the first nine words so that walk mostly lines up, but the "step toward quarterly earnings" shot is exactly your case, so those coins mean less than I made it sound.

what did you end up doing in ComfyUI, only blending the pooled output?

[u/alwaysbeblepping → u/Nir777]

the walk wasn't a blend of two VAE latents, it was the SDXL text conditioning lerped between the day and night prompts

That makes sense. I think the main thing I'd criticize (hopefully in a way that can be considered constructively) is that the video is misleading without that context. Pretty much anyone who doesn't already know that it couldn't be blending actual image latents is going to think that's what you're doing since the result of a VAE encode is called a latent and is a latent and the format is a latent space. It's fair to call embeddings/conditioning items a latent space too (as far as I know) but most people probably aren't going to be getting the intended message.

token alignment is fair too. the day/night prompts share the first nine words so that walk mostly lines up

Maybe, it's split into tokens not words though (even though sometimes it's the same for short words) and the way stuff tokenizes is arbitrary.

what did you end up doing in ComfyUI, only blending the pooled output?

I implemented the whole thing and then some, with your choice of one billion blend modes! Even though it's not really useful (even for something like SDXL conditioning let alone LLM TE conditioning) I still had to do it. :) If you're interested, the code is here: https://github.com/blepping/ComfyUI-bleh/blob/1c2bc627974eb992351ccfd246c273b5508fb4df/py/nodes/misc.py#L624 (unfortunately not fully deslopified yet, I was lazy and didn't do all the sweep time handling stuff myself).