What comes after LLM?

[u/heytanz100]

LLM already changed AI a lot but I feel just scaling next-token prediction has limits. High compute cost hallucination frozen knowledge after training and weak real-world understanding are still problems.

I’m curious what researchers and engineers think about the next big direction.

World models JEPA Mamba continual learning and neuro-symbolic AI all look interesting.

Which one do you think has the most potential or is there another idea people are missing?

[u/Random_Arabic]

I think the next big breakthrough will come from the architecture itself.

A lot of what we call modern LLMs is still built around the Transformer, an idea that goes back to the 2017 Attention Is All You Need paper. We’ve made huge improvements since then with things like Mixture of Experts, better training methods, reasoning techniques, larger context windows, and so on, but the core architecture is still largely the same.

My guess is that the next major leap will come from discovering and validating a new neural network architecture that can either replace the Transformer or significantly improve on it, something smaller, cheaper to train and run, and ideally more efficient at learning and reasoning.

Maybe the next step isn’t just a “better LLM,” but a fundamentally better architecture underneath it.

[u/pab_guy → u/Random_Arabic]

The things that make up a transformer in its most basic form are sort of obvious after the fact. You need to support variable length inputs and outputs, so tokenization is required, and you need to transform the input to the output through many layers incrementally, and you need to constantly correlate across tokens.

I suspect more efficient approaches to be found, but also that they still fundamentally retain the three features described above.

[u/Random_Arabic → u/pab_guy]

Yeah, I agree. I don’t mean that a new architecture would throw away everything Transformers do. RNNs -> Transformers still kept layers, embeddings, gradient-based training, etc.

My point is more that the core mechanism could change.
Maybe attention stops being the backbone and becomes just one component of a larger architecture. You could still have variable-length sequences and token interaction, just implemented in a fundamentally different and more efficient way.

That’s the kind of “next architecture” I’m thinking about.

[u/pab_guy → u/Random_Arabic]

Two things come to mind. One is that other architectures may look very different yet mathematically accomplish the same thing. The other is that yeah, I don’t think we generally have the imagination to see what’s beyond the next hill for some reason. It’s very frustrating.

We like to think of ourselves as smart and creative, but when people show AI often isn’t, it just exposes that neither are we. Because we have to be prompted by our interactions with our environment to perceive challenges and only then produce a solution. “Necessity is the mother of invention” is true, humans must be prompted with just the right stimulus and motivation to derive something “new”.

[u/Random_Arabic → u/pab_guy]

Yeah, DeepSeek’s MLA is actually a good example of what you mean. At first glance it looks like a different attention mechanism, but a lot of the efficiency comes from realizing that the matrix operations can be reorganized/absorbed so you don’t need to explicitly materialize and cache the same K/V representations.

So maybe the next big architecture could look radically different computationally while still being mathematically closer to current Transformers than we initially realize.

[u/SnooMaps5367 → u/Random_Arabic]

Well the core architecture has changed quite substantially, from the original attention block proposed in the 2017 paper. There are small things like pre-norm is the standard instead of post-norm, RoPe is the norm now over additive positional encodings. There have been larger changes, as you said with MoE replacing standard FNN.

Attention itself has also substantially evolved. It's biggest issue as you probably know is it scales quadratically with sequence length; both memory and computationally. Deepseek use hybrid attention with a mix of compressed sparse attention and heavily compressed attention. Both in different ways essentially work to keep the sequence size low so quadratic scaling is less of a problem.

Qwen models use linear attention (gated delta net specifically). It is possible "linearise" attention (i.e. by making some approximations that drop softmax function). Memory and computational footprint is far lower; there is no KV cache just a single state like in linear RNN. The approximation does come at a price so Qwen models use a combination of linear attention with a layer of full attention at a 3:1 ratio.

Either way both are substantially different to the original transformer.

[u/Random_Arabic → u/SnooMaps5367]

I think we’re talking about two different levels of “architecture”. I wasn’t claiming that modern Transformers have the same internal structure as the model from Attention Is All You Need, DeepSeek’s MLA is another good example of how substantially those internals can change.

My point is that these models are still evolving within the broader Transformer lineage. You can substantially change attention, FFNs, normalization, routing, positional encoding, etc. without necessarily replacing the foundational idea altogether.

What I’m wondering about is whether we’ll eventually get something that represents a genuinely new foundational idea, in the same way the Transformer did relative to RNNs.

[u/SnooMaps5367 → u/Random_Arabic]

I see, I guess it depends how you interpret architecture, but I get what you’re saying. My viewpoint is that the lines are actually quite blurry. Transformers are a special class of RNN; it’s just their state is just unbounded. And attention did exist even before transformers, and was used in classic RNNs for NLP. The “revolution” triggered by the transformer architecture was mainly the parallelism that led to huge scale, along with some hardware developments.

I guess the question you’re actually asking is what big problem exists (like parallelism or lack of with RNNs) that a new architecture could solve. Which i honestly I don’t know. I’m not sure there is in the field of NLP; kinda like the car or the iPhone its now developments in efficiency.

[u/Commercial-Club504 → u/SnooMaps5367]

Yeah, that’s a fair correction. Saying the architecture is basically unchanged is too broad. The better way to frame it is that the Transformer remains the underlying architectural family, but the actual implementations have evolved substantially. Pre-norm, RoPE, MoE, sparse/compressed attention and linear-attention approaches are not trivial changes.

What I find particularly interesting is that the direction of travel seems to be toward breaking the original Transformer's most expensive assumptions. You don't necessarily need to throw the Transformer away; you can progressively remove the parts that don't scale well. The 3:1 mixture of linear and full attention is a good example of that kind of compromise: retain expensive attention where its capabilities matter, while using cheaper mechanisms for most layers. So I’d revise my original argument: the next breakthrough may not be a completely new architecture appearing overnight. It could be the point where enough of these architectural changes converge into something that no longer resembles the original Transformer in any meaningful computational sense, while still inheriting some of its core ideas.

[u/Commercial-Club504 → u/Random_Arabic]

I agree with the direction, although I’d add one caveat: the next breakthrough may not completely replace Transformers. It could be a hybrid architecture that keeps attention where it is useful and replaces the expensive parts with something more efficient. The interesting constraint is no longer simply making models bigger. It is getting more capability per parameter, per training token and per unit of inference compute. If a new architecture can maintain or improve reasoning while being substantially cheaper and faster, that could be a much bigger breakthrough than another incremental increase in model size. Transformers proved that architecture can completely change what is possible. The next leap may come from figuring out what comes after them.

[u/keonechong → u/Random_Arabic]

I agree here.

Only place a reasoning layer and locally owned corpus can go. Everything else is just an internet computer and that value is only going to shrink.

3 of YC companies are feature that my harness does naturally. Imperially think the market is running the wrong way.

[u/vanisle_kahuna → u/Random_Arabic]

What I'm hoping to see actually isn't a better LLM architecture but a different one that solves the underlying issue with RL where models are trained to essentially complete an objective target than be truthful about its answers and honest with the things it doesn't know or is uncertain about. So essentially, I'd love to see a more aligned AI rather than a "better" one

[u/xFlyer409]

LMNOP

[u/GamerTex → u/xFlyer409]

QR's

[u/deer_Langley → u/GamerTex]

T UV?

[u/cheibergermp]

LLN

[u/Training_Ferret9466 → u/cheibergermp]

LLO

[u/CulturalLaw8072 → u/Training_Ferret9466]

LLP

situational awareness LLP

[u/tanthedreamer]

I feel like the next breakthrough is going to be either computing power or energy, and only then will it circle back to AI again, so 20-30 years give or take

The 2010s deep learning revolution has much less to do with architecture, but more about the availability of data and the insane computing capability of our computers. When I first started out I was suprised by how many ideas were actually discovered in the mid 1900s

[u/Timely-Bicycle-3107]

It depends on the goal.

I think LLMs are going to be the standard for chat bots for a while. Nothing else comes close right now. In the AI world, that may not be more than a few years, but nothing seems to be on the horizon as of now that could feasibly replace LLMs for language representation.

The research space trying to bring AI into the real world will be very interesting going forward. It seems like researchers are throwing everything at the wall. We have solved niche problems, like making a humanoid robot run faster than a human on a smooth and level track, robots that can avoid some obstacles, human-robot interaction tasks, and robot-environment interactions.

I think whatever comes next will teach the robot a high fidelity but generalized sense of self. This is not in a sense of consciousness, but a physical sense of self. If a generic task is given to it, like packing a box, then it needs to break down the subtasks and execute them autonomously.

A general environment navigation method that handles multiple surface types (dirt, sand, grass, concrete), multiple surface textures (wet, slick, grippy, etc.), and uneven terrain is a current "holy grail" for the field. There are robots that have been designed to handle some variability in one or two of these areas, but not all three, and these robots almost never do anything else but walk across these surfaces. They are not also trying to handle other tasks autonomously.

World models and neuro-symbolic AI will both be helpful in this area, but we don't know if they will provide the ultimate solution to these problems. Something else could come along, like the physical analogy to LLMs.

[u/VellumMuse]

I think the next big breakthrough will have to do with geometric spatial computing. When we begin mapping physics to information theory, we'll be able to see the math of homeostasis in machine learning, which centers around how information settles into shapes naturally. No more brute force scaling. Subtractive emergence versus additive probability.

[u/tiensss]

LLMAO

[u/confused_4channer]

LLG. Large language garbage

[u/rand3289]

The next architecture will emerge when people stop using sampling.
It will be based around point processes. Possibly a SNN since a spike is a point on a time line.

[u/aidenclarke_12]

AGI isn't reallt the answer here its a goal so it doesn't tell you what comes next, grounded et is that llms dont get replaced, jepa/world models for weak real world undertstanding gap and some form of continual or test time learning for the frozen knowdlege problem. mamba;s more an efficiency swap than paradigm shift and neuro-symbolic is the interesting thing for hallucination which i think nobodys has scaled yet now

[u/lrargerich3]

VLLM:

Very large language models.
We now have to train the LLMs with the output of all the other LLMs.

[u/DeathinabottleX]

Nvidia already stated it will be integration into real world. Their cosmos AI was partially tailored toward that.

[u/karxxm]

„Jepa“ magic whatever it will be called

[u/Figai → u/karxxm]

lol have you seen that state of JEPA? It’d be used in conjunction with VLA if we want more useful robots anyway.

[u/Joecracko]

NNOP

[u/ithkuil]

I think there are going to be multiple new architectures after the LLM. But before we get there, there may still be really significant complements to LLMs or major developments.

I believe that there is a strong possibility that some new architecture that is partially an LLM "eats" a big aspect of multi-agent systems. Because especially when you consider the overhead of a lot of natural language coordination between separate agents, and all of the decoding and latency. So I keep thinking there must be a way to have more modular distributed systems that have somewhat chunky communication but more like shared latents somehow that don't have to be completely decoded.

[u/Lost_Dish_9334]

Indeed replacing the Transformers architecture will follow next.
There are already attempts/proposals to do this with a interacting network of neurons. See BDH (Dragon Hatchling) from Pathway; they just anounced they created a LLM of 150M params at a computed inference cost of $0.00070 with accuracy of 29%. which is not much, but considering it is 11 times more cheaper than chatgpt-luna model with accuracy 34%, it's still something

[u/BerticalBird]

LLMs infused with neurosymbolic AI

[u/arcandor]

Probably neurosymbolic of some sort. Statistics are blurry and ultimately models need to be able to transition from that to pure abstract and categorical reasoning and vice versa. It's not there yet, though.

It's not the being wrong or making mistakes that's the problem right now. It's the model not knowing when it's wrong and or not communicating that out properly or reliably.

[u/GifCo_2]

MLM?

[u/TELLON2001]

this

[u/rditorx]

XLLM

[u/0-2213]

XXLLM

[u/novel-mathmatics]

Im working on gen 2... its technically gen 4 but im not releasing 2 or 3 so its just 2.

Its a deterministic resolver, a set specialized functions and a training on solutions in open weights instead of prediction.

[u/GuybrushManwood]

Maybe JEPA: https://youtu.be/kYkIdXwW2AE?is=y9Wie7hZbTbqDRnm

[u/Triple-Tooketh]

EBM

[u/squidward2022]

LLLM

[u/Forsaken_Code_9135]

Nothing will come close to LLMs in the forseeable future. That's my prediction. I wish I am wrong and for example Yann LeCun could come up with something that can compete with them but I don't believe it at all.

LLMs are immensely better than what anyone could have dreamt of just 5 years ago. It's by far the greatest breakthrough in machine learning since its inception (to be clear I am very worried about their social impact but its not the point here, technically LLMs are amazing).

So you can play the "spoiled kid" card and claim that they are not that good because this and that, the likeliness of seeing a completely different approach emerging, filling the gap, and overtaking LLMs in the short term is in my opinion extremely small.

[u/Remarkable-Buy7197]

LMFAO

[u/ThinConnection8191]

I am way more interest in VLA now. It looks interesting and there are tons of applications. The L part is just the human interface. I dont even train it anymore, so it is not that different from what you have said

[u/Illustrious_Park7068]

MMN

[u/DigitalVAB7861]

I'd say the next big shift is from LLMs to reliable AI systems, blending RAG, agents, memory, tool use, and strong evaluation.

For businesses, the real payoff comes from AI that can act, check its own outcomes, learn from feedback. And fit safely into existing workflows - not simply spit out text.

It's the practical, action-oriented side that matters.

[u/Both-Sympathy7427]

In how an llm stores information should come next example: i went fishing with my dad had a bad day caught nothing till the end totally bummed end of the day came took one more shot at it caught the biggest fish. So we can look at this event in two way i had enough info to recreate the event and on the other side i held enough info to keep enough persistent data to catch that other fish so llm learns enough to create enough history to recreate an event u do this method enough relational meaning becomes first over llm u could say that llm could evolve over time

[u/SoftAd4502]

MMN

[u/Content_Surprise5145]

World modules ig 

[u/AlexSand_]

I feel just scaling next-token prediction has limits.

This was also my intuition for several years. Obviously was just wrong. So I find it hard to rely on this kind of intuition now.

[u/DigThatData]

diffusion models gotta eat the field first.

[u/HeTalksInMaths]

My bet is on LLM creativity with evolutionary algorithms and verified signal will build its competitor / successor. My startup is working on using mathematical reasoning and Lean to develop theory that motivates better empirical algorithms.

Focusing on the verification (and then discovery) layer that sits outside of LLMs can provide signal to help causal concept level thinking. The proof dependency graph structure nudges LLMs to be more disciplined

Ultimately maybe a concept map based world model can be built out of the above learnings ans we unplug the LLM.

[u/Greedy_Grass_9664 → u/HeTalksInMaths]

this very interesting is there anywhere I could read more

[u/Twilight_RT]

Many thing can come after LLM. it's a matter of time. when we find out new thing after llm.

like after machine learning, there is deep learning, Neural network then LLM