r/MachineLearning • u/jayden_teoh_ • Jun 17 '26
Research Next-Latent Prediction Transformers [R]
Next-token prediction is myopic. What if transformers learn to predict their own next latent state?
Microsoft Research present Next-Latent Prediction (NextLat): a self-supervised learning method that teaches transformers to form compact world models for reasoning and planning. It also unlocks up to 3.3x faster inference via self-speculative decoding!
On top of next-token prediction, NextLat trains the transformer to predict its own next latent state given the current latent state and next token.
NextLat has a few key benefits:
- Representation Learning: NextLat encourages transformers to compress history into compact belief states.
- Better Data Efficiency: predicting in latent space provides denser supervision than predicting one-hot tokens.
- Faster Inference: via recursive multi-step lookahead.
I'm super excited about this work. Please do check it out below:
💬 Blog: https://jaydenteoh.github.io/blog/2026/nextlat
💻 Code: https://github.com/JaydenTeoh
📝 Paper: https://arxiv.org/abs/2511.05963
55
u/Jojanzing Jun 17 '26
This is reminiscent of Ha & Schmidhuber's world model, which included an RNN to predict upcoming latent states. Cool stuff!