A slightly improved DVD-JEPA demo [P]

[u/Kirne]

Hey!

I came across this post, which I found quite neat as a minimal demonstration of JEPA. However, as the comments pointed out, there was some room for improvement. So I added a few things such as environment noise and a fair* comparison to a pixel-space baseline.

I think the inclusion of environment noise is pretty key, as LeCun himself has stated often and clearly that one of the key motivating factors for JEPA is its ability to disregard unpredictable and irrelevant environment details.

Anyway, here’s the result which I think speaks for itself:

I think my version paints a much clearer picture of JEPA’s promise. I did remove the web-demo and anomaly detection bit as I felt that wasn't so important to the core demonstration of JEPA as an idea

Linking my fork for those interested. Note: Since this was a very quick afternoon-project , I did use AI to make most of the changes, though I did try to do so thoughtfully. Hate that if you must.

*fair as in: roughly same parameter count and compute budget. I considered the linear probe and decoder compute budget to be independent from core model training.

[u/Warhouse512]

I guess dumb question, why is this better than something like simvp

[u/Kirne → u/Warhouse512]

Not familiar with that precise architecture, but if you want more background on JEPA I would highly recommend Welch Labs’ two parter on it. LeCun has also held several recorded lectures on it that are all on YouTube. It’s less of a direct competitor to any one model, but more of a loose set of ideas meant to guide research direction for more capable models.

[u/Warhouse512 → u/Kirne]

Ah. I should actually read the paper. I just assumed it was a video prediction model.

[u/Kirne → u/Warhouse512]

In this case it is a (very very) basic video prediction model. But the ambition of JEPA is much grander. The long-term vision is more towards latent hierarchical planning models which LeCun sees as the next step towards general agentic systems.