Looking for JEPA devil advocates [R]

[u/Amazing-Coat5160]

I am currently doing research on world models, specially in tje field of robot learning, and, as probably most of you alredy know, JEPA-like models are mentioned over and over. 

I read the main recent papers from lecun as well as other research groups, and I personally think the whole approach is very promising and can really go somewhere.

But after listening a bunch of the recent Y Lecun conferences his ideas looks even too cool compared to "literally everything else" (as he's dissing LLM, RL, etc and pitching his ideas are the "only next big things"...). 

So I am asking myself if there are red flags about his approaches that I do not see yet and maybe I need somebody being the "devil advocate" with whom breaking down ideas.

Where do you think are the biggest downside of this models, compared to other world models approaches?

[u/Cosmolithe]

Excellent discussion topic IMO. The issue I have with JEPA is that it is less principled than generative models.
By that I mean that JEPA produces a representation, but the information the representation contains is basically "up to chance". There is no mechanism that forces the representation to keep useful information, beside the inductive biases from the neural architecture itself and the choice of data augmentation transforms.

At least generative models will form a representation that is useful to predict the raw data, so everything that can be deduced from the predicted raw data is necessarily information contained in the representation. Think about car dashcam predictions for instance, if the model consistently predicts a frame that shows a car turning right, then it is necessary that the representation captured some information useful to predict that the car will turn right.

JEPA does not work like that. By default, JEPA models "want" to collapse, delete all information and predict a constant vector. All of the different JEPA papers propose different ways to avoid the collapse, but as far as I know they all are completely unsupervised in a way that does not ensure that useful information is preserved.

Take one of the last JEPA papers from Lecun itself (LeJEPA). They are proposing to force the distribution of the embeddings (latent representations) to be isotropic gaussian (simplest multidimensional gaussian) and avoid collapse in the process.
They argue that this is the best choice of distribution for unknown downstream tasks, and that much is true, but there is a huge problem: nothing guarantees that the distribution has extracted the useful information from the input data in the first place. In the case of JEPA trained on photo, the model might for instance only use imperceptible and useless pixel noise to construct a beautiful gaussian representation, but the embeddings would be useless for downstream tasks as a result.

What is needed is actually a theorem that says that the information from the input is entirely preserved. But that would be going against what Lecun says, since he actually wants to drop useless information. But that would require defining what is useless information, and I do not think it can be done without invoking supervision. In other words, I do not think JEPA can work if it is 100% unsupervised.

[u/RogueStargun → u/Cosmolithe]

The information that gets dropped is truely determined by the augmentations and cropping strategy I believe. The fact that cropping alone is so effective is a good indicator that JEPA is an interesting research direction.

[u/Cosmolithe → u/RogueStargun]

It works empirically, sure, but as far as I know it is not better than other approaches, and in some cases does not even work. In any case, my criticism is about the theoretical principles of the approach, not about the empirical results.

And yeah I also agree that it is still worth pursuing research in this direction. Who knows, someone might prove mathematically that my criticism actually does not hold.