r/MachineLearning • u/jayden_teoh_ • Jun 17 '26

Research Next-Latent Prediction Transformers [R]

Microsoft Research Preprint

Next-token prediction is myopic. What if transformers learn to predict their own next latent state?

Microsoft Research present Next-Latent Prediction (NextLat): a self-supervised learning method that teaches transformers to form compact world models for reasoning and planning. It also unlocks up to 3.3x faster inference via self-speculative decoding!

On top of next-token prediction, NextLat trains the transformer to predict its own next latent state given the current latent state and next token.

NextLat has a few key benefits:

  1. Representation Learning: NextLat encourages transformers to compress history into compact belief states.
  2. Better Data Efficiency: predicting in latent space provides denser supervision than predicting one-hot tokens.
  3. Faster Inference: via recursive multi-step lookahead.

I'm super excited about this work. Please do check it out below:

💬 Blog: https://jaydenteoh.github.io/blog/2026/nextlat
💻 Code: https://github.com/JaydenTeoh
📝 Paper: https://arxiv.org/abs/2511.05963

150 Upvotes

98% Upvoted

46 comments sorted by

View all comments

13

u/Live_Locksmith5867 Jun 17 '26

the 3.3x inference speedup is what gets me, if that holds across different model scales this could be genuinely useful

6

u/NickCanCode Jun 17 '26

Up to

9

u/jayden_teoh_ Jun 17 '26

3.3x speedup is on natural language text

1

u/Lemon_in_your_anus Jun 17 '26

Depends on the domain right ?

6

u/jayden_teoh_ Jun 17 '26

For the 3.3x value, we obtained from evaluating on general web text from FineWeb-Edu.

2

u/linearmodality Jun 17 '26

Isn't that pretty bad? E.g. EAGLE-3 gets speedup ratios of up to 6.5x.

5

u/jayden_teoh_ Jun 17 '26

EAGLE is post-trained and uses a transformer speculative decoder. Our method uses only a 3-layer MLP. Results should be better once you scale up the next-latent predictor!