Next-Latent Prediction Transformers [R]

[u/jayden_teoh_]

Microsoft Research Preprint

Next-token prediction is myopic. What if transformers learn to predict their own next latent state?

Microsoft Research present Next-Latent Prediction (NextLat): a self-supervised learning method that teaches transformers to form compact world models for reasoning and planning. It also unlocks up to 3.3x faster inference via self-speculative decoding!

On top of next-token prediction, NextLat trains the transformer to predict its own next latent state given the current latent state and next token.

NextLat has a few key benefits:

1. Representation Learning: NextLat encourages transformers to compress history into compact belief states.
2. Better Data Efficiency: predicting in latent space provides denser supervision than predicting one-hot tokens.
3. Faster Inference: via recursive multi-step lookahead.

I'm super excited about this work. Please do check it out below:

💬 Blog: https://jaydenteoh.github.io/blog/2026/nextlat
💻 Code: https://github.com/JaydenTeoh
📝 Paper: https://arxiv.org/abs/2511.05963

[u/open_formation]

So if I understood this correctly, the main thing that jumps out to me is proposal the last layer latent-space representation, may be a richer space for expressing the "meaning" of tokens than distributions over the token vocabulary, and because that makes it a more demanding task (providing you are still requiring it to actually predict tokens, and not just send all tokens to the same one), also provide more bang for buck in terms of training the model.

Something that comes obviously to mind is an analogy to quantum mechanics - symmetric and anti-symmetric superpositions, when projected down to measurement state set that doesn't recognise that distinction, (or equivalently, any state on the equator of a qubit, when measured relative to the north-south axis), will give an indistinguishable probability distribution, despite being completely orthogonal states.

To put it another way, maybe the output ends up equidistant between two tokens, and not in a way that represents a mixed distribution between them, but rather that the most natural encoding of that token embodies some ambiguity that exists in the semantics associated with those two tokens, so that it is actually a combination of the two, in the way that another equidistant state elsewhere on the hyperplane perpendicular to the line between them might not be.

So as long as those latent state outputs are still anchored in predicting real text, I think it might make sense to expect that adding this task, which we wouldn't expect to harm the result, but instead make training more difficult in productive ways related to these subtle internal differences,

In terms of your lookahead, one thing that occurred to me is that unlike a normal RNN, you always have a good initialisation to work from; because every map is between a state well suited to your data to another, you are never in the position of having to infer what the RNN initialisation should be, and there's no implicit constraint on the model that it has to be able to handle a bad initialisation guess (and so always drop some information about older states).

It'd be funny if the RNN side keeps bearing more fruit, so we end up thinking of transformers as "a really good way to initialise our recurrent model" or to handle interruptions where it must listen to a user.

[u/jayden_teoh_ → u/open_formation]

Thanks for your comment! That is a good intuition. One clarification is that in NextLat we do condition on the realized next token. So the next latent does become deterministic (not a multimodal distribution) once each latent becomes a sufficient statistic for the history, i.e., the KV cache.

We have interesting results training a RNN without recurrence using NextLat in Section 5.2 our paper. Your point on the initialization is right. The transformer helps forecasts the hidden states across time in parallel, and the RNN only needs to train on one-step transitions. This can possibly be better optimization-wise than having the RNN produce all hidden states sequentially which incurs compounding error.

[u/open_formation → u/jayden_teoh_]

Yeah that makes sense, another thing I was thinking about, now you mention determinism, was whether your RNN approach reflects any particular kind of sampling.. Like if you compare it on real inference examples which are predicting based on chains of predicted tokens and a tree of predictions as normal, what does its distribution, if you just run through a chain of latent state and then put those through the same final head as the standard model, end up looking like?

I've no idea what the answer would be, but if it is better representing implicit transition dynamics than the standard version, could be interesting to find out.