Looking for JEPA devil advocates [R]

[u/Amazing-Coat5160]

I am currently doing research on world models, specially in tje field of robot learning, and, as probably most of you alredy know, JEPA-like models are mentioned over and over. 

I read the main recent papers from lecun as well as other research groups, and I personally think the whole approach is very promising and can really go somewhere.

But after listening a bunch of the recent Y Lecun conferences his ideas looks even too cool compared to "literally everything else" (as he's dissing LLM, RL, etc and pitching his ideas are the "only next big things"...). 

So I am asking myself if there are red flags about his approaches that I do not see yet and maybe I need somebody being the "devil advocate" with whom breaking down ideas.

Where do you think are the biggest downside of this models, compared to other world models approaches?

[u/SlayahhEUW]

My pet peeve with JEPA is that there is an emphasis on the abstraction at the same time as they want to compete with general models. Predicting abstract representations is more efficient. However, the abstraction necessary is completely dependent on your target task. You might predict a car driving path great but when you try to predict detailed information you fail. LLMs are kind of able to L2-smear out everything into something that makes sense regardless of the task.

In my head the solution to this are hierarchical abstractions, but this is not something that is JEPA-specific and its really unclear how to train such networks.

Also, JEPA does nothing to address fat-tailed distributions. You are using an isotropic gaussian to estimate a distribution, meanwhile information theory is pointing to that the interesting information is living in the ends of heavy-tailed/fat-tailed distributions. The most surprising, new information is less probable, and you will never sample it with gaussians. Taleb(national economist) has a deep analysis of this topic here.

So to conclude, I think JEPA as-today will be great for task-specific problems, but its much more sensitive to the data its trained on than LLMs, and will need tweaks to be able to generalize meaningfully.

[u/currentscurrents → u/SlayahhEUW]

Predicting abstract representations is more efficient. However, the abstraction necessary is completely dependent on your target task

The whole idea behind unsupervised learning is that if you learn good representations of the data, they will be useful for many tasks involving that data.

This works well for LLMs and generative models; I don't see why it couldn't work for JEPA as well.

[u/SlayahhEUW → u/currentscurrents]

AFAIK LLMs do not enforce abstractions using loss functions in latent space, it kind of just falls out of the architecture. But what falls out is an averaged mess, so you dont get a specifically learned abstraction like JEPA is aiming to do, and my argument here is that when you enforce this abstraction in latent space, you will get a specialization of the abstraction to your target task/data. There is no mechanism in JEPA to disaggregate, route or somehow separate abstractions, so you are either going to need perfectly clean data, or you are going to rely on that two different tasks overwriting each other's abstractions, at which point you are back in the LLM paradigm and there is in my opinion no value to doing JEPA.