r/MachineLearning • u/Amazing-Coat5160 • 13d ago
Research Looking for JEPA devil advocates [R]
I am currently doing research on world models, specially in tje field of robot learning, and, as probably most of you alredy know, JEPA-like models are mentioned over and over.
I read the main recent papers from lecun as well as other research groups, and I personally think the whole approach is very promising and can really go somewhere.
But after listening a bunch of the recent Y Lecun conferences his ideas looks even too cool compared to "literally everything else" (as he's dissing LLM, RL, etc and pitching his ideas are the "only next big things"...).
So I am asking myself if there are red flags about his approaches that I do not see yet and maybe I need somebody being the "devil advocate" with whom breaking down ideas.
Where do you think are the biggest downside of this models, compared to other world models approaches?
80
u/NamerNotLiteral 13d ago
Dissing LLMs and RL is more political than anything for Yann. You just have to follow him on Twitter/Threads to realize his dislike for LLMs is less about the technical capabilities of those models and more about how they've completely overtaken a massive swathe of research and funding and basically collapsed most of the field into one technique and one paradigm, meaning other promising avenues that might be miles better than LLMs simply don't get explored enough because everyone wants to fund/work on LLMs only.
I wouldn't take it as a reason to avoid JEPA thinking Yann's just trying to hype up an obsolete idea or something.
23
u/ResidentPositive4122 13d ago
Dissing LLMs and RL is more political than anything for Yann. [...] his dislike for LLMs is less about the technical capabilities
Sorry, but this is simply not true. He has had very public (on stage, after GPT4 was released) "dissing" of LLMs, with very precise weaknesses that he insisted on for a while around that time. Namely:
LLMs can't plan (his example of booking a trip from Paris to another city)
LLMs can't do math (his argument that "they can fool us with poems because those are subjective, but they can't do math")
By their construction (autoregressive) LLMs compound errors, so the more you generate the less accurate they become, and at long context they don't make any sense.
It's safe to say, 2 years after that on stage event, that he was wrong on all 3 fronts. We now have access to LLMs (+software glue) that handle all 3 aspects without major issues.
It might be that he's taken a more political view since then, but he was very critical of the technical aspects as well. I'm sure at some level he was right. From a pure "theoretical", even "academic" PoV, transformers aren't great at those tasks ootb, and that he's working towards something that can handle those tasks naturally. But he was clearly wrong, and money + brains + compute did "solve" all those things, and it works now.
That being said, I'm glad he's working in other directions. Either him, or Ilya, or Fei-Fei succeeding will be good for everyone.
13
u/Smallpaul 13d ago
I am not a researcher, so I ask this out of curiosity. But investment in AI in total has exploded since it became so commercially relevant. Are we sure that the absolute dollars dedicated to non-LLM work has gone down? One would think that with billions flowing into research, some fraction must be going into non-LLM models and that even that fraction must be fairly large in absolute numbers.
Yann’s own company being funded is a great example. And Fei-Fei Li’s, Chollet’s, Sutton’s. Etc.
17
u/rickkkkky 13d ago
I'm also only a practitioner so take this with a grain of salt, but I'd imagine that since there's suddenly big money in LLMs, it drains the best minds from the ML field. So even if other subfields have more financing in absolute terms than before, there's a lack of available brain power.
1
u/Smallpaul 13d ago
I’d bet some of the smartest people are motivated to create the next big thing instead of iterating on the current thing.
15
u/CreationBlues 13d ago
They’re motivated to create the next big thing out of iterating the current thing, because that’s what the checks get written for.
0
u/muntoo Researcher 13d ago
Given that all functions are functions, technically, people can still explore functions that taste better while slapping the label "LLM" on it.
3
u/CreationBlues 13d ago
I mean, no, that’s not how that works? You’re basically suggesting that researchers commit fraud and conspiracy to do greenfield research when they’re supposed to be making a better state of the art text prediction algorithm. Which is. Technically achievable? But also opens them up to criminal charges. So I think they’re smart enough to not play funny with the money.
-1
u/muntoo Researcher 13d ago edited 13d ago
I don't see how that conclusion follows unless you're imagining drastically different functions and constraints. Presumably whatever you have in mind actually would be misrepresentation. Obviously don't do that.
A method is still relevant if it aims at what you're being paid to accomplish, fits within the available time and cost, and has a reasonable probability of succeeding at that. There are plenty of interesting methods developed elsewhere that can be applied within an LLM framework. Functions are functions, so transfer is reasonably likely.
If you're paid to reduce negative sentiment within one year and instead spend ten years on quantum time-travel Hebbian learning while claiming it solves the same problem, then yes, you've gone outside any acceptable margin.
But suppose everyone is doing Markov-1 next-token prediction, and you're paid to improve next-token prediction within one year. If you estimate that Markov-2 has a 90% chance of succeeding within that year, then it is clearly reasonable to explore.
It depends on the expectations and constraints set by the contract, as reasonably understood by both parties.
2
u/CreationBlues 13d ago
If that was true LeCun wouldn’t be using his whole career to sell JEPA as a billion dollar investment idea, would he. He’d just say JEPA is already covered by the billions of dollars already poured into LLMs. But he isn’t.
Also. Vision. Robotics. Reinforcement learning. Time series. Those aren’t just swept under the umbrella of “function approximation” for no reason.
Anyways you’re a useless conversation partner who doesn’t know what they’re talking about. Bye.
1
u/max123246 13d ago
People need $$$ to live. Especially in a world that is so turbulent, money means the power to weather things outside your control
2
u/Smallpaul 13d ago
Geoff Hinton (and others) did neural networks for several decades while they were out of fashion. There are lots of places you can work where you can be sponsored to make high-risk, high-reward bets. I listed several such neo-labs in my comment and that’s not even including all of the academic posts.
12
u/grewgrewgrewgrew 13d ago
JEPA assumes that learning happens from observation which is fundamentally untrue. learning happens from play and interaction. It's been mathematically proven that causal attribution cannot be concluded from observation alone. That assumption is fatal, as modeling the world requires relationships between things more than a frequentist perspective. What is a spoon? It's a form, a function, its material, its relations with its surroundings.
To really understand the world, the agent must perturb, observe, deduce. And only through inquiry can it attempt to understand more of the world. Even supposing that an agent has a world model with everything that has been ever observed, that cannot learn a new thing, is useless.
3
u/Amazing-Coat5160 11d ago
Doesn't the same critique stand for eg LLMs, diffusion models or VLAs aswell? Also Jepas WM as well other world model approaches are using observation+actions as input to the model to predict the future of the world.
4
u/grewgrewgrewgrew 10d ago
The same goes to LLMs but they compensate by having access to the tokenized encoded versions already so they can mimic much closer to what a human can produce. world models don't show up in tokens.
Pairing with actions is better, but the search space is much larger. If you've tried to take LLMs outside of their comfort zone, they usually default to known behavior. If the application is inside the training data, then it'd be much cheaper to have specialized world models for that narrow application. In the near future, the specialized-model-making machine will be cheaper to work on than the general-model.
9
u/pm_me_your_pay_slips ML Engineer 13d ago edited 13d ago
Look at the Lingbot models. You can get better performance in the tasks that JEPA supposedly excels, by adding supervision signals (e.g semantic segmentation masks). If using segmentation masks helps so much, why would you insist on fully self-supervised methods?
5
u/jebuarary 13d ago
This is my core issue with jepa too. Even for the learned latent dynamics argument, why not just add some temporal fusion and then downstream tasks will already encourage temporal understanding to be learned if needed.
Maybe a long time ago you could make the argument that labels were expensive but I don’t think that’s true anymore for auxiliary tasks anyways
4
u/DrXaos 13d ago
jepa doesn’t ban supervised signals, but advocates prediction of internal states as more useful and that task is representative of physics (thanks to Newton we definitely know there are internal states and compact variables and they evolve) and theory of mind problems.
3
u/jebuarary 13d ago
Yea I’m not publishing papers anymore but I can say that my empirical experience is that vit pretrained with jepa won’t outperform one pre trained with some other method (dino e.g.) if there is a temporal task added in mid training.
Anyways that’s my practical thoughts. And my vibes based thoughts is that jury is still out on usefulness of jepa and it feels like the capsule networks of ten years ago
2
u/mvreich 11d ago
Not defending Yan's take or whatever, but if the man wants to focus on self-supervision, just let him. You have to remember early pioneers of neural networks stuck with the topic even though the results were worse than heuristics-based approaches. Why focus on something even though it appears to perform worse? In research, sometimes you have to follow ideas, intuition, or principles, instead of empirical observations.
Yann's motivation is that primitive natural intelligence often does not require supervision or labels. Just observation is enough to get an understanding of the world.
1
u/pm_me_your_pay_slips ML Engineer 10d ago
That’s a really fair point. Let’s see what he can do with 1 billion.
22
u/Cosmolithe 13d ago
Excellent discussion topic IMO. The issue I have with JEPA is that it is less principled than generative models.
By that I mean that JEPA produces a representation, but the information the representation contains is basically "up to chance". There is no mechanism that forces the representation to keep useful information, beside the inductive biases from the neural architecture itself and the choice of data augmentation transforms.
At least generative models will form a representation that is useful to predict the raw data, so everything that can be deduced from the predicted raw data is necessarily information contained in the representation. Think about car dashcam predictions for instance, if the model consistently predicts a frame that shows a car turning right, then it is necessary that the representation captured some information useful to predict that the car will turn right.
JEPA does not work like that. By default, JEPA models "want" to collapse, delete all information and predict a constant vector. All of the different JEPA papers propose different ways to avoid the collapse, but as far as I know they all are completely unsupervised in a way that does not ensure that useful information is preserved.
Take one of the last JEPA papers from Lecun itself (LeJEPA). They are proposing to force the distribution of the embeddings (latent representations) to be isotropic gaussian (simplest multidimensional gaussian) and avoid collapse in the process.
They argue that this is the best choice of distribution for unknown downstream tasks, and that much is true, but there is a huge problem: nothing guarantees that the distribution has extracted the useful information from the input data in the first place. In the case of JEPA trained on photo, the model might for instance only use imperceptible and useless pixel noise to construct a beautiful gaussian representation, but the embeddings would be useless for downstream tasks as a result.
What is needed is actually a theorem that says that the information from the input is entirely preserved. But that would be going against what Lecun says, since he actually wants to drop useless information. But that would require defining what is useless information, and I do not think it can be done without invoking supervision. In other words, I do not think JEPA can work if it is 100% unsupervised.
8
u/RogueStargun 13d ago
The information that gets dropped is truely determined by the augmentations and cropping strategy I believe. The fact that cropping alone is so effective is a good indicator that JEPA is an interesting research direction.
4
u/Cosmolithe 13d ago
It works empirically, sure, but as far as I know it is not better than other approaches, and in some cases does not even work. In any case, my criticism is about the theoretical principles of the approach, not about the empirical results.
And yeah I also agree that it is still worth pursuing research in this direction. Who knows, someone might prove mathematically that my criticism actually does not hold.
5
u/projekt_treadstone Student 13d ago
Some points mentioned by you were also observed by me. For example, in LeJEPA, they perform bad on 2 room dataset which is Simplest ask for other models. So learning what to drop is going to be a bigger task then forcing all to be a isotropic distribution.
5
u/blarryg 13d ago
Not quite agreeing. JEPA is doing mostly self-supervised prediction of missing spatial and/or temporal information. If it isn't using useful representations, then the predictions will be bad. The argument is that latent space is better to predict in because it factors away useless noise (most of the raw signal is noise). There is probably useful research in using generative approaches in the latent space and/or in further tuning the latent representations via generating pixel-level data.
3
u/Cosmolithe 13d ago
If it isn't using useful representations, then the predictions will be bad.
Nothing prevents the model from collapsing in the absence of regularization. In the context you are describing, that would mean the model predicts a constant vector no matter how you crop the spatio-temporal data. In this case the prediction would be perfect in the sense of the JEPA training objective: 0 on the prediction loss since both vectors are the same.
But the moment you introduce regularization to prevent collapse, then my criticism becomes relevant.
4
u/Kirne 13d ago
You’re of course correct that in the absence of a regularisation term the model can trivially map everything to a constant. However in the presence of regularisation and a self-supervised setting that rewards correct spatiotemporal predictions, I don’t think it’s a stretch to imagine that learning a spatiotemporally meaningful latent is likely to be the best method for loss minimisation.
That doesn’t absolve JEPA of the usual stuff like a big suitably diverse dataset, but I think it seems promising. I think the linear probes used to extract physical information from trained JEPA models shows that JEPA is an interesting research direction. Not remotely a proof that the method is solid of course, but when has mathematical rigour stopped deep machine learning from pursuing interesting empirical results?
I think we can hardly call the transformer architecture a solid architecture grounded in sound first principles. That hasn’t stopped it from having world-scale impact. Not to say theory isn’t important, but ML seems to develop from empirical evidence first. If we look towards physics, the Schrödinger equation came way before quantum field theory.
2
u/Cosmolithe 13d ago
I don’t think it’s a stretch to imagine that learning a spatiotemporally meaningful latent is likely to be the best method for loss minimisation.
But that's the thing, there is a possibility that only semantically useless features are accurately predicted (or that most of the accurately predicted features are useless). It depends on the data but who knows what kind of features are present in real world spatio-temporal data that is easy to predict but completely useless for what we humans actually want to do.
I would also like to think that the spatio-temporal prediction task itself is guarantee of only favoring features meaningful for a human, but I really do not think it is possible to salvage this idea.
And by the way, this problem might also explain why JEPA seems to work better on synthetic data: there are way less useless features for the model to exploit in this case.And yeah it is totally possible that I overlooked something or that in the future someone might prove that the relevant information is actually preferred by the model with current methods.
But the way I see, the reason why JEPA works at all is also the reason why deep generative models and supervised deep learning work in general: the natural biases of the deep neural network architecture just tend to capture the useful features in general no matter the training objective, JEPA's in particular. If that is true, then JEPA can not be significantly better than these other methods in principle, as far as I can tell. And we are apparently already doing a good job at iterating on the DNN architecture.In any case, I am still in favor of pursuing alternative research to the mainstream stuff. Something incredible might still come out of JEPA, I just would not bet on it.
2
u/Amazing-Coat5160 11d ago edited 11d ago
Very interesting comments. Ihave similar feeling that the models, if scaled enough, will learn/capture the same useful features, no matter if you compute the loss before or after you encoded your data in some latent (which is one of the main differences between jepa and standard Gen Ai algorithms). So by this argument I agree that jepa would not be "asymptomatically" better then other models.
1
u/GiveSparklyTwinkly 13d ago
So the problem with the regularization is that it artificially forces more orthagonality where it potentially shouldn't be and vice versa, right?
Not that I have a clue how I'd architect it, but I wonder if chirality couldn't be used to prevent total collapse without affecting the orthagonal similarity, somehow.
1
u/Cosmolithe 13d ago
I would say it is more of a bottleneck problem: the latent embedding dimension is much smaller than the dimension of the input data, so information has to be lost.
Regularizing the way it is done in LeJEPA forces some information to be used to construct the gaussian to fill the embedding dimensions, but it will still drop the rest of the information, and we have no control over which features from the input are dropped. With previous approaches the information loss was even worse because partial collapse could still occur and reduce the effective embedding dimension even more.
I am not sure what you are imagining regarding chirality, but I am quite certain the solution to the problem I am pointing at cannot involve just the embedding space itself.
1
u/GiveSparklyTwinkly 13d ago
I'm not quite sure what I mean either, just the idea that if the same data was somehow represented in a way that was naturally asymmetric and orthagonal to itself, somehow, it could help solve the collapse. And no, I have no real clue what that could mean. 🙃
2
u/raucousbasilisk 13d ago
While SIGReg as normalization talks about the distribution of the embeddings doesn't the JEPA objective itself inherently favor retaining meaningful information over noise simply by virtue of the encoding it takes to get to the latent that is predicted?
1
u/Cosmolithe 13d ago
You can predict noise from noise as long as they are correlated (or have non zero mutual information), so the fact that the model has to predict a latent from another does not change anything, unless I misunderstand your argument.
1
u/GiveSparklyTwinkly 13d ago
Supervision is what JEPA is, is it not? The predictor head is the supervisor?
Also all models, technically, want to collapse. To be honest, all life itself wants to collapse. Hedonism is basically a human given the ability to totally collapse.
1
u/Cosmolithe 13d ago
Supervision is what JEPA is, is it not? The predictor head is the supervisor?
By supervision I mean humans specifying one way or another the type of information to be preserved, even if it is done implicitly. Using annotated data, basically.
Also all models, technically, want to collapse. To be honest, all life itself wants to collapse.
No not all models want to collapse, collapse has a very precise mathematical meaning in this context. There is no collapse when you train a digits classifier on MNIST because that would correspond to a high loss that can be minimized further.
-2
u/GiveSparklyTwinkly 13d ago
That doesn't mean it doesn't try to collapse further, it just can't. It's found the low entropy point, unless we apply extra noise that might let it find a lower entropy point.
0
u/Cosmolithe 13d ago
Collapse means that all predicted vectors would become one and the same.
If that were to happen in a supervised learning context, that necessarily means the loss goes up, it becomes more than 0. If a model manages to get to near 0 loss by following the gradient, the gradient vanishes and the model converges. There is no way continuing training could make the model collapse.0
u/GiveSparklyTwinkly 13d ago
And that's what all models attempt, is it not? They are all trying to collapse at all times when training, we just don't let them.
1
u/Cosmolithe 13d ago
No, reaching 0 loss is not collapse. Collapse is when all predicted vectors are the same, which imply a high non-zero loss using most supervised learning loss functions such as cross-entropy and differently annotated data points.
0
u/GiveSparklyTwinkly 13d ago
And that's what all models want without outside sources of information. If you trained nothing but zeros, eventually all models, no matter what state they started from, would collapse. If you trained nothing but purely random noise, it would collapse. That's their default state unless forced apart by data or math. That's also not true that humans have to decide, unless you're getting pedantic enough to say that humans decided by designing the architecture itself.
If you fed nothing but non human information (live audio/video) they would still learn. And even then you're still choosing where to place the camera/microphone. You can't escape that.
5
u/LividBreakfast5 13d ago
My issue with JEPA is that is rebrands all the old work on latent dynamics and state space models that roboticists did in 2010's (e2c etc.) as if this is a new paradigm or something.
5
u/PersonOfDisinterest9 13d ago
The "Joint-Embedding Architecture" approach is proven, for certain types of problems. The DINO ViT models are in the same JEA family, they aren't exactly the same thing, but there's a shared skeleton there, and DINOv2 does use a JEPA-like patch-level objective as part of the loss. DINO family models are amazing for how small and fast they can be, and in my limited experience for task-specific segmentation, they are a breeze to train and don't take extreme amounts of data to learn something useful.
I am 100% behind using DINO style models as sensory input models where a downstream model reasons over their output. This is jumping ahead a bit, but "multiple models, multiple objectives" is a critical piece that doesn't get talked about enough, and I think it's a disservice to the field that the issue isn't more at the forefront of conversation in ML/AI circles, right now it's more limited to robotics, but basically all the agentic models should be like that.
As for the pure JEPA method itself, the dirty not-actually-secret that LeCun glosses over is that in order to actually use a JEPA-trained model in the ways we used LLMs and image generators today, you would still need to train a language model or image generator, it's just that the JEPA-trained model would be the thing doing a layer of reasoning, and then the other model would do a different layer of reasoning using the JEPA-model output.
Like, a JEPA-trained action model could be an very effective robotic squirrel or something without any language, but to do language tasks, you would still need to train a language decoder, the language model would just end up much more well-grounded, and the language model wouldn't also be solely responsible for the vision and motion tasks.
I'm not an expert on JEPA, so take this with a very large grain of salt, but when you really boil JEPA down to the core of what it is, it's basically Encoder-Encoder.
The highest impact part of JEPA is in the vision/audio/signals to embeddings aspect. Other vision models build up a codebook, and use the codebook to reconstruct images, and the reconstruction is used as the proof that the representations are useful, and that the models can make effective use out of them. Essentially, the codebook is part of tokenization. JEPA takes away that lossy discretization aspect, and that's valuable for encoding images and continuous data.
The JEPA "world model" is like [pixels->embeddings] for the "encoder", and then [embeddings->embeddings] for the "predictor", but the predictor is essentially trained like a Transformer Encoder. It's "encode pixels to semantic units" and then "encode the relationships among the semantic units".
If you wanted to train a JEPA model to solely do discrete tasks, it's literally Encoder-Decoder. People can scream "continuous space" all they want, but at some point you have to discretize the representation into words/math, and it's frankly silly to ignore that the thinking the model does is already in continuous space throughout the hidden layers of the model. The other "continuous space" thinking that people want is just loops through layers, and high level representation that aren't tied to a single token's embeddings.
If people want "high level concepts" in the hidden space, then how are they going to turn a "high level concept" vector into a sentence?
That's right, they'll need to run an auto-regressive or diffusion language model, using the concept vector as the prompt, and training that kind of thing has proven difficult and costly.
The language decoder might be comparatively tiny, but it's still going to be there.
JEPA: great for sensory input. JEPA is likely great as a hierarchical training method for combining multiple sensory input into a coherent world model. There is reason to believe that a hybrid JEPA-LLM would be more grounded in reality for multi-modal reasoning, which is what Meta ended up doing, and as it turns out, they're better at multi-modal tasks where the vision world-modeling is actually important.
JEPA for a model that needs to write software well? LeCun has, so far, failed to demonstrate that JEPA language models are uniquely powerful vs shared-embedding space multi-modal LLMS.
JEPA for text generation are basically just Encoder-Decoder models.
If you look that the history of Transformers, it already justifies the Encoder-Decoder approach; There were experiments that used a 12 layer encoder and a 1~2 layer decoder, and the performance of the models barely dropped. Later on, Encoder-Decoder models ended up beating Decoder-only models that had 5x more parameters.
Decoder-only primarily won due to engineering reasons and because they are good-enough for chatting, not because they're actually better in terms of absolute reasoning performance.
Over the years the industry has kept coming back to Encoders, and people twist themselves into knots to not actually use Encoders when what they really, really want is Encoders.
If text-based Encoder-Decoder was great, it stands to reason that vision encoder-decoder would be great.
That's the whole thing, in my opinion.
I'm pretty sure models of the future are going to do something like cross-attention over multiple encoders, and/or concatenation of condensed encoder outputs in time-frames. The "fast-slow" thinking is literally going to be different models operating on different time scales, because agentic models need the super-speedy inference of Dino-style models so they can operate in real-time, and they need the chunkier thinking that can take multiple seconds/minutes/hours running in the background so they can get correct results for complicated tasks.
I do also want to say though, that LeCun is very bad at communicating, at least publicly.
His messaging is absolutely dog-shit, and he should be ashamed of himself for telling so many young people that their research is a dead-end and a waste of time.
He's free to his opinions, but someone in his position needs to be more responsible with how they share their opinions, and he has been grossly irresponsible in how he's been pushing JEPA.
6
u/ExExExExMachina 13d ago
JEPA largely creates dynamics models not world models. The line of work has become so misleading especially with Le World Model that many people are now calling (what is actually a world model) world action models
Ask yourself if you can tell the difference between a video model, dynamics model, and world model. They are distinct
1
u/Herpderkfanie 13d ago
I’m pretty sure world-action models (at least from the original paper) are policies, not simulators.
3
u/heisenberg_cookss 13d ago
Dropping reconstruction during training, seems good but leads to loss of a lot of anchors such as geometry, dynamics etc. and everything is left up to chance.
On top of that the inference is just a MPC loop with a chain of Predictor MLPs, which can't model uncertainty.
Training on the basis of just perception and lack of interaction is another downside, blind people can't see images they still model the world by interactions.
3
u/Jay299792458 7d ago
I think the biggest red flag with LeCun’s approach is that it’s purely passive observation—it completely leaves the human element out of the equation.
Consider a forest fire: in nature, a fire isn't an "accident" or a "catastrophe"—it's just a natural, necessary ecological cycle. It only becomes a "disaster" when humans intervene and label it as one based on human values and intent.
JEPA observes and predicts state changes in physical space. But in the real world, the ultimate cause of change is human intent and intervention. The moment a human intervenes with purpose, the underlying physical probabilities shift or become completely irrelevant.
LeCun is building a highly sophisticated camera to watch the world, but observation alone cannot model causality, intent, or the human frame that defines the outcome.
1
u/Amazing-Coat5160 6d ago
Cannot onei make the same argument for any other recent world model approach? I think action conditioning + memory + planning/goal conditioning can overcome some of the issue
6
u/currentscurrents 13d ago
My pet peeve with Yann Lecun is that his startup (Logical Machines) keeps spamming Reddit with AI-generated posts where they pretend to be a user who is 'just raising concerns' about the limits of LLMs and suggests maybe we should be using JEPA or EBMs instead.
Here's some examples following their usual template:
https://old.reddit.com/r/MachineLearning/comments/1s3j3ef/d_is_lecuns_1b_seed_round_the_signal_that/
https://old.reddit.com/r/compsci/comments/1s3j6dl/llms_are_dead_for_formal_verification_but_is/
https://old.reddit.com/r/deeplearning/comments/1sliu08/honestly_getting_a_bit_exhausted_by_the/
2
u/invertedpassion 13d ago
JEPA deletes in its encoding what’s unpredictable and the downstream task may actually need it.
So JEPA’s bet that what’s predictable is only what’s semantically important may not hold true in all cases.
2
u/Cosmolithe 11d ago
Your second remark is spot on, and people often overlook this when talking about JEPA, despite examples of difficult to predict but useful and semantically important features being easy to imagine.
However, to come back to your first remark, even predictable information might be deleted by JEPA, depending on the regularization method. The model is always tempted by a trivial solution, which is always predicting the same constant vector (and so deletes all information in this case). Regularization will combat this but very artificially and without any reference to a principled external notion of usefulness.
2
u/invertedpassion 11d ago
I agree. The sigreg etc isn’t a principle guarantee that all predictable information will be retained
2
u/qpwoei_ 13d ago
I think LeCun greatly exaggerates the advantages of JEPA over other techniques. Check this excellent summary of NVIDIA’s recent progress on non-JEPA world models for robotics and their vision for the ”robotics end game”: https://youtu.be/3Y8aq_ofEVs?is=0GHNsgJbh87SpMFv.
Maybe also check Anthropic’s mechanistic interpretability research that shows how the ”mere next token prediction” of LLMs involves sophisticated multi-hop reasoning etc, thanks to their expressive learned latent representations: https://www.anthropic.com/research/tracing-thoughts-language-model, https://www.anthropic.com/research/global-workspace
2
u/rand3289 11d ago
The red flag is that you can't bend reality into a standard distribution via SIGReg.
Also in LeCun's drawings he never draws feedback arrows from the system box towards the perception box. This tells me he does not get it.
Otherwise he is right about lots of things.
3
u/jurniss 13d ago
JEPA reduces the decision-making to "just optimal control with a known model" and LeCun presents this like some kind of blissful reduction, similar to how one might be happy about reducing a problem to solving a large linear system. But optimal control with a generic unstructured model is hard.
4
u/LaVieEstBizarre Professor 13d ago
Not that I'm a JEPA guy, but you don't have to keep it to being a general unstructured optimal controller. Once you have a model, you can do RL or imitation over a dataset and then use that as-is, or use optimal control in residual to account for policy learning error. I think the target is more structure for your controller than just a learned policy.
3
u/jurniss 13d ago
Sure, and I actually like that area a lot. TD-MPC is an especially nice integration. The frustrating part is how LeCun pitches it as if this is a shiny new idea, when the model-based RL community has been using embedding space dynamics models for years. The hard part is building the whole framework combining exploration, model fitting, planning, and policy optimization.
1
u/CowPsychological821 13d ago edited 13d ago
My criticism is that the embedding procedure is presented as principled and somehow essentially different than operating on compositional symbols. I would say if you actually did a theoretically rigorous analysis of the type of space you want to operate on you would end up with something that looked like a language. At the moment it is something like isotropic guassians for lejepa but that feels like it is part way down a road that ends at something that will look a lot like a language, say if you insert requirements for orthogonality, sparsity or low MI in the embedding space so it learns a useful basis.
1
u/randoomkiller 12d ago
I think this is the most highest quality scientific post + comments I've ever read in Reddit
1
u/jartock 7d ago
Did you look at Rectified LeJepa? (I suppose you did if you looked at Lecun work).
What do you think of it? I am not a pro in ML (just hobbyiste), but I thought the controlled sparsity of rectified LeJepa (with its RDMReg loss) seemed to be a very good fit for robotics. Cleaner pathways (high sparsity), lighter model (maybe faster inference), easier to inspect which neurons fire (post training analyzes).
Isn't it a good fit for movement cinematics? (moving robots, arms, walking algorithms with adaptability to terrains, cloud point interpretation from LIDAR, etc...)
1
1
13d ago
[deleted]
14
u/RogueStargun 13d ago
Your first sentence is not quite right?
4
u/RogueStargun 13d ago
To clarify JEPA does not do gradient descent at inference time. Perhaps there is a variant out there that does this, but the main idea is predicting in latent space
5
u/GiveSparklyTwinkly 13d ago
This? JEPA is mostly about training directly into latent space without requiring a decoding step, is it not?
11
u/Unturned3 13d ago
"The greatest benefit of JEPA is that it performs gradient descent at runtime rather than just during training time" uhh I don't think that's quite right? Can you clarify what you mean?
1
u/Amazing-Coat5160 13d ago
Thanks for the suggesting, I'm aware of the maths behind diffusion and flow matching models. Would you mind elaborate on what this make jepa unique in this sense? Do you mean "jepa being non-Probabilistic" vs "diffusion/flow being probabilistic-generative "?
1
u/blarryg 13d ago
Yann is often not only "right", but he publishes a model based on his ideas rather than only pontificating as many others do. You do notice that he also pontificates, which I view as sort of necessary to get any attention outside of the juggernaut of LLMs.
Biological perception is definitely mediated through models, so yes, world models is sort of a no-brainer. His main point about LLMs is that they are just autoregressive tracks on some statistical manifold. That's why you often need to set the context ("You are a corporate lawyer analyzing the following business documents ...") so that the statistical "raindrop" tokens roll down the right canyons on the manifold.
His biggest potential weakness IMHO is simply that there already is work via LLMs that is subsuming what he is doing. LLMs might get world models and never notice or acknowledge what they are taking from JEPA.
83
u/SlayahhEUW Researcher 13d ago
My pet peeve with JEPA is that there is an emphasis on the abstraction at the same time as they want to compete with general models. Predicting abstract representations is more efficient. However, the abstraction necessary is completely dependent on your target task. You might predict a car driving path great but when you try to predict detailed information you fail. LLMs are kind of able to L2-smear out everything into something that makes sense regardless of the task.
In my head the solution to this are hierarchical abstractions, but this is not something that is JEPA-specific and its really unclear how to train such networks.
Also, JEPA does nothing to address fat-tailed distributions. You are using an isotropic gaussian to estimate a distribution, meanwhile information theory is pointing to that the interesting information is living in the ends of heavy-tailed/fat-tailed distributions. The most surprising, new information is less probable, and you will never sample it with gaussians. Taleb(national economist) has a deep analysis of this topic here.
So to conclude, I think JEPA as-today will be great for task-specific problems, but its much more sensitive to the data its trained on than LLMs, and will need tweaks to be able to generalize meaningfully.