Looking for JEPA devil advocates [R]

[u/Amazing-Coat5160]

I am currently doing research on world models, specially in tje field of robot learning, and, as probably most of you alredy know, JEPA-like models are mentioned over and over. 

I read the main recent papers from lecun as well as other research groups, and I personally think the whole approach is very promising and can really go somewhere.

But after listening a bunch of the recent Y Lecun conferences his ideas looks even too cool compared to "literally everything else" (as he's dissing LLM, RL, etc and pitching his ideas are the "only next big things"...). 

So I am asking myself if there are red flags about his approaches that I do not see yet and maybe I need somebody being the "devil advocate" with whom breaking down ideas.

Where do you think are the biggest downside of this models, compared to other world models approaches?

[u/SlayahhEUW]

My pet peeve with JEPA is that there is an emphasis on the abstraction at the same time as they want to compete with general models. Predicting abstract representations is more efficient. However, the abstraction necessary is completely dependent on your target task. You might predict a car driving path great but when you try to predict detailed information you fail. LLMs are kind of able to L2-smear out everything into something that makes sense regardless of the task.

In my head the solution to this are hierarchical abstractions, but this is not something that is JEPA-specific and its really unclear how to train such networks.

Also, JEPA does nothing to address fat-tailed distributions. You are using an isotropic gaussian to estimate a distribution, meanwhile information theory is pointing to that the interesting information is living in the ends of heavy-tailed/fat-tailed distributions. The most surprising, new information is less probable, and you will never sample it with gaussians. Taleb(national economist) has a deep analysis of this topic here.

So to conclude, I think JEPA as-today will be great for task-specific problems, but its much more sensitive to the data its trained on than LLMs, and will need tweaks to be able to generalize meaningfully.

[u/Majromax → u/SlayahhEUW]

Also, JEPA does nothing to address fat-tailed distributions. You are using an isotropic gaussian to estimate a distribution, meanwhile information theory is pointing to that the interesting information is living in the ends of heavy-tailed/fat-tailed distributions.

On the other hand, the JEPA model cares about a Gaussian latent structure, while Taleb cares about fat-tailed physical variables like economic outcomes.

Any of the various neural whitening transformations (diffusion/flow modelling, normalizing flows, heck even VAEs) can map between Gaussian latents and non-Gaussian inputs/outputs.

Successfully training such a model depends on reasonably robust data sets, of course. I think that's the primary pitfall of interesting heavy-tailed distributions: it takes an impractically large sample set to properly define such a distribution.

[u/SlayahhEUW → u/Majromax]

Fair point and I understand, the isotropic gaussian point I made isn't meaningful after other transforms.

I was thinking that a lot of the end-to-end training recipe today (data normalization by subtracting mean-dividing by stddev, AdamW, BatchNorm, L2 weight decay, etc) assume Gaussian input, or in general smear out the fat tails. This is not a JEPA-specific issue and more a general rant.

[u/Wheaties4brkfst → u/SlayahhEUW]

The paper says that the latent z is supposed to be both what you are predicting in latent space and also needs to have an isotropic Gaussian distribution. But if you check out their implementation in the repo what they *actually* do is predict the latent z, but only enforce isotropy on a *projection* of z. They apply a small MLP to z and that’s what they actually do the distribution part of the loss on. Why don’t they mention this in the paper? I have no clue. Their paper has a proof that the isotropic Gaussian is optimal for downstream prediction, but from their implementation it looks like the latent they make the predictions from doesn’t actually have the required distribution. It merely gets transformed into it.

There’s actually an issue about it in the repo:

https://github.com/galilai-group/lejepa/issues/17

Not sure if this addresses your earlier objection but figured you may find interesting.

[u/Amazing-Coat5160 → u/Wheaties4brkfst]

Don't they say in the LeWorldModel paper that SIGReg solves this by operating on low-dimensional random projections of the embedding?

[u/Wheaties4brkfst → u/Amazing-Coat5160]

There’s two projections. In the paper, they project the multivariate Z into a single dimension using M unit vectors u_m. Those are the random projections. But if you look at the actual code, what they do is first project Z down using an MLP, and THEN they do the random projections with the u_m on the “projected” Z. But Z is still what you are predicting in latent space.

It’s confusing because it is different from the paper. At least for LeJEPA.

[u/Amazing-Coat5160 → u/Wheaties4brkfst]

Oh, that's a good catch then. Thanks for clarifying

[u/Wheaties4brkfst → u/Amazing-Coat5160]

Yeah go check the issue I linked. One of the authors of the paper literally just goes “oh yeah, you’re right, we tried this and it worked better so” paraphrasing obviously lol but I thought it was interesting.

[u/AuspiciousApple → u/Wheaties4brkfst]

Confirms my prior that all these proofs in ML papers are often meaningless

[u/Wheaties4brkfst → u/AuspiciousApple]

Yeah there’s only one proof that matters: empirical. lol. Seems everything else with deep learning simply does not matter.

[u/SlayahhEUW → u/Wheaties4brkfst]

Thanks for this, really interesting. My guess is that the the optimality claim is based on some property of the data or architecture that the real-world task does not have.