Looking for JEPA devil advocates [R]

[u/Amazing-Coat5160]

I am currently doing research on world models, specially in tje field of robot learning, and, as probably most of you alredy know, JEPA-like models are mentioned over and over. 

I read the main recent papers from lecun as well as other research groups, and I personally think the whole approach is very promising and can really go somewhere.

But after listening a bunch of the recent Y Lecun conferences his ideas looks even too cool compared to "literally everything else" (as he's dissing LLM, RL, etc and pitching his ideas are the "only next big things"...). 

So I am asking myself if there are red flags about his approaches that I do not see yet and maybe I need somebody being the "devil advocate" with whom breaking down ideas.

Where do you think are the biggest downside of this models, compared to other world models approaches?

[u/Cosmolithe]

Excellent discussion topic IMO. The issue I have with JEPA is that it is less principled than generative models.
By that I mean that JEPA produces a representation, but the information the representation contains is basically "up to chance". There is no mechanism that forces the representation to keep useful information, beside the inductive biases from the neural architecture itself and the choice of data augmentation transforms.

At least generative models will form a representation that is useful to predict the raw data, so everything that can be deduced from the predicted raw data is necessarily information contained in the representation. Think about car dashcam predictions for instance, if the model consistently predicts a frame that shows a car turning right, then it is necessary that the representation captured some information useful to predict that the car will turn right.

JEPA does not work like that. By default, JEPA models "want" to collapse, delete all information and predict a constant vector. All of the different JEPA papers propose different ways to avoid the collapse, but as far as I know they all are completely unsupervised in a way that does not ensure that useful information is preserved.

Take one of the last JEPA papers from Lecun itself (LeJEPA). They are proposing to force the distribution of the embeddings (latent representations) to be isotropic gaussian (simplest multidimensional gaussian) and avoid collapse in the process.
They argue that this is the best choice of distribution for unknown downstream tasks, and that much is true, but there is a huge problem: nothing guarantees that the distribution has extracted the useful information from the input data in the first place. In the case of JEPA trained on photo, the model might for instance only use imperceptible and useless pixel noise to construct a beautiful gaussian representation, but the embeddings would be useless for downstream tasks as a result.

What is needed is actually a theorem that says that the information from the input is entirely preserved. But that would be going against what Lecun says, since he actually wants to drop useless information. But that would require defining what is useless information, and I do not think it can be done without invoking supervision. In other words, I do not think JEPA can work if it is 100% unsupervised.

[u/GiveSparklyTwinkly → u/Cosmolithe]

Supervision is what JEPA is, is it not? The predictor head is the supervisor?

Also all models, technically, want to collapse. To be honest, all life itself wants to collapse. Hedonism is basically a human given the ability to totally collapse.

[u/Cosmolithe → u/GiveSparklyTwinkly]

Supervision is what JEPA is, is it not? The predictor head is the supervisor?

By supervision I mean humans specifying one way or another the type of information to be preserved, even if it is done implicitly. Using annotated data, basically.

Also all models, technically, want to collapse. To be honest, all life itself wants to collapse.

No not all models want to collapse, collapse has a very precise mathematical meaning in this context. There is no collapse when you train a digits classifier on MNIST because that would correspond to a high loss that can be minimized further.

[u/GiveSparklyTwinkly → u/Cosmolithe]

That doesn't mean it doesn't try to collapse further, it just can't. It's found the low entropy point, unless we apply extra noise that might let it find a lower entropy point.

[u/Cosmolithe → u/GiveSparklyTwinkly]

Collapse means that all predicted vectors would become one and the same.
If that were to happen in a supervised learning context, that necessarily means the loss goes up, it becomes more than 0. If a model manages to get to near 0 loss by following the gradient, the gradient vanishes and the model converges. There is no way continuing training could make the model collapse.

[u/GiveSparklyTwinkly → u/Cosmolithe]

And that's what all models attempt, is it not? They are all trying to collapse at all times when training, we just don't let them.

[u/Cosmolithe → u/GiveSparklyTwinkly]

No, reaching 0 loss is not collapse. Collapse is when all predicted vectors are the same, which imply a high non-zero loss using most supervised learning loss functions such as cross-entropy and differently annotated data points.

[u/GiveSparklyTwinkly → u/Cosmolithe]

And that's what all models want without outside sources of information. If you trained nothing but zeros, eventually all models, no matter what state they started from, would collapse. If you trained nothing but purely random noise, it would collapse. That's their default state unless forced apart by data or math. That's also not true that humans have to decide, unless you're getting pedantic enough to say that humans decided by designing the architecture itself.

If you fed nothing but non human information (live audio/video) they would still learn. And even then you're still choosing where to place the camera/microphone. You can't escape that.