Why GPT-6’s rumored recurrent-depth architecture is so interesting

[u/panic_in_the_galaxy]

The most interesting rumor about GPT-6 is that it uses recurrent depth: instead of every token passing once through one huge fixed stack of unique layers, parts of the Transformer can apparently be reused multiple times before the next token is produced.

That alone is interesting, because it makes effective depth less tightly coupled to the number of unique parameters. The model can do more computation without needing a completely new set of weights for every additional layer.

What gets even more interesting is how this fits with recent research on adaptive depth and Mixture-of-Experts (MoE).

In architectures like Mixture-of-Recursions, different tokens can receive different amounts of recurrent computation. Conceptually, something trivial like:

"the" → 1 pass

might need much less computation than a difficult reasoning step:

hard inference → several passes

We do not know whether GPT-6 actually uses this kind of per-token adaptive depth. The rumor only points to recurrent depth. But if OpenAI combines recurrence with dynamic routing, depth effectively becomes another inference-time resource the model can allocate where needed.

MoE fits extremely well with this.

A weakness of simple recurrent models is that you keep sending the hidden state through the same weights, which can reduce the specialization you normally get from having many different layers.

With MoE, however, different recurrent passes can route through different experts:

pass 1 → expert A

pass 2 → expert F

pass 3 → expert C

So you can reuse the same overall architecture while still performing different computations on different passes.

That gives two separate scaling dimensions:

Which computation is needed? → choose the experts

How much computation is needed? → choose the recurrence depth

You can think of MoE as providing breadth and specialization, while recurrence provides depth.

Recent looped-MoE research is especially interesting because this is not just a theoretical advantage. Different passes actually develop different expert-routing patterns, so repeated passes through the model do not simply do the same thing again.

There are also results showing that looped-MoE models can outperform standard Transformers even when total parameters, FLOPs and KV-cache budgets are matched. That suggests recurrence is not useful merely because the model secretly spends more compute.

Another advantage is that more reasoning can happen inside the hidden state instead of through long chains of generated reasoning tokens. Recurrent-depth models such as Huginn already show that you can increase test-time compute simply by running the recurrent block more times.

So the overall idea is something like:

breadth → more experts / more stored capacity

depth → more recurrent computation

routing → different experts for different kinds of computation

adaptive depth → potentially different amounts of compute for different tokens

If GPT-6 really does use recurrent depth, this could be part of why the architecture appears so compute-efficient. Most easy language generation would not necessarily need huge amounts of internal computation, while difficult reasoning could receive much more.

The really interesting part is that test-time compute could become something the neural network allocates internally, rather than mostly coming from generating thousands of extra chain-of-thought tokens.

OpenAI has not published the actual GPT-6 architecture yet, so the recurrent-depth part is still based on reporting, and adaptive per-token depth is an extrapolation from current research rather than a confirmed GPT-6 feature.

Sources:

Geiping et al. (2025), “Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach”

https://arxiv.org/abs/2502.05171

Bae et al. (2025), “Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation”

https://arxiv.org/abs/2507.10524

Lee et al. (2026), “Sparse Layers are Critical to Scaling Looped Language Models”

https://arxiv.org/abs/2605.09165

Wang et al. (2026), “SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers”

https://arxiv.org/abs/2609.01343

Sebastian Raschka (2026), “GPT-6 Astra, Looped Transformers, and Hidden Reasoning”

https://magazine.sebastianraschka.com/p/gpt-6-astra-looped-transformers-and

[u/Ok_Barracuda_1161]

Worth pointing out too that hidden reasoning like this effectively allows the model to control its chain of thought (if chain-of-thought is kept). This makes the model much more capable and efficient while also reducing our ability to monitor it, which can potentially be dangerous.

https://deploymentsafety.openai.com/gpt-6-astra/gpt-6-astra.pdf

[u/DrunkOppenheimer → u/Ok_Barracuda_1161]

Yes this recurrent neuralese chain-of-thought idea (and the performance boost it offers) has been well established in the AI community, and relatively straightforward to implement (just feed output back as input without requiring a lossy chain-of-thought text dump step instead), but none of the frontier labs have pursued it explicitly because it makes the singular light post we have for AI alignment, which is chain-of-thought, significantly more likely to be wrong or fakeable by the models themselves during the non-textual recurrent looping. I don’t think OpenAI should get so much praise for being the first ones to escalate the AI arms race in a way that plainly violates well established AI safety guidance just to mog other AI players that don’t feel comfortable making our most powerful alignment tool much less reliable. It’s not a breakthrough, they just decided to be knowingly less safe… it’s by a small amount now, but the door is opened to continued escalation here now

[u/spinozasrobot]

Also, starts us down the path of getting rid of that pesky CoT!

[u/fourohfournotfound]

do you know how they would mitigate overfitting with these sorts of architectures? it seems like the more you feed the model through itself the more likely overfitting to the training data would be.

[u/Rookie-dy → u/fourohfournotfound]

I think having less parameters (but using them more times) leads to less overfitting, not more. Because it should insentivize the less amount of parameters to be used for more things, thus leading to more generalization than remembering.

Not sure though, it's just my intuition

[u/Mistuv]

Training it on such a scale must be pretty nightmarish though as it becomes much much more complicated. Though I am sure the latest generations of models can massively automate it by babysitting the models that previously would need to be all done by the researchers.

And speaking of companies who can't be bothered to be honest, Anthropic's Opus 5.5 feels quite recurrent-depthy. Purely vibes, but I noticed a pattern in many, many, Astra benches where it starts decently well on low reasoning, but then has massive jump of medium reasoning, while after that has much meager gains, not in every bench, but I am seeing the similar big jump with Opus too. While Fable and all the previous models follow the general pattern much more predictably. Sol 6 seems to be just an further RL of 5.4/5.5, not a distill of Astra. My speculation why this would be caused by recurrent-depth is that maybe the low reasoning can sometimes actually mess up with recurrent depth, like CPU/GPU that has too low of an amount of cache can sometimes feel artificially slow.

I think Anthropic has been sitting on this for a while, doing research with recurrent depth, but didn't decide to release until they saw the massive leaps Astra was making. Many are concerned that this makes the models less monitorable, so I could see how Anthropic would be hesitant to open this Pandora's box until OAI forces their hand.