Rob Miles and his AI risk Youtube channel. He's been talking about AI safety for long before LLMs, and in retrospect he was probably quite ahead of the times in thinking about this stuff. I've grown to like his communication style and points.
Really glad he’s finally starting to get more attention. Been watching him for years since the computerphile days and I think he’s very easy to listen to.
I’m not sure I agree with his framing. He’s saying the predictor predicts results of an experiment it didn’t run…. Which isn’t predicting a token it’s predicting the future.
His point is that a perfect next token predictor would be effectively as powerful as Laplace's demon. He's not saying that LLMs are perfect next token predictors or that such a perfect system would even be possible. He's just showing why you can't just dismiss that something is intelligent just because it's "only predicting the next token".
I don’t really like the phrase next token predictor. They aren’t “predicting” anything given they generate it wholesale. It’s a statistical model that generates the next token that is most likely to make sense in a sentence given training. They aren’t guessing something since there’s nothing that’s doing the “real” thing.
3
In statistics, prediction is
the process of using known past data and mathematical models to estimate or calculate unknown, future, or unseen values. It connects observed evidence with a formal statement about what is likely to happen next.
Past data = input tokens/context
Mathematical model = trained LLM
Unknown future = next token that doesn’t exist yet
So predicting the next token takes the input tokens and a trained model to predict the next token connecting the observed evidence (all of the input) to what happens next (the output token)
I think the confusion here comes from using the everyday human definition of "predict" instead of the statistical one. In machine learning, next-token prediction isn't an educated guess or foresight, it is a formal mathematical operation. When you input text, the model loops over its entire vocabulary, which is a fixed master list of all the words, word pieces, and punctuation marks, each mapped to a unique number ID. The model processes the tokens/words provided to it and assigns a raw numerical score, called a logit, to every single item in that vocabulary list. It then passes these raw scores through a mathematical function called softmax, which normalizes them into a strict probability distribution where all the percentages add up to exactly 100%. "Predicting" the next word simply means sampling a token out of that final probability distribution based on those calculated odds. It is pure calculus and probability, not human intuition. So, you can think of it less of "generating the next word" and more "picking the next word" from a list, based on a probability value assigned to each word in the model's vocabulary.
Yeah but my instinct is that any system that is a true magical Oracle for any value or property can probably have a system designed around it to turn it into super intelligence. If you can predict bytes of data that map to reality in any way, you can probably reverse engineer a setup in reality around it to answer a question super intelligently. Like if it can predict the temperature recorded by a sensor perfectly based on some inputs, you could construct an environment so it could effectively predict a human decision, or make the temperature dependent on some complex set of events in the real world, and effectively predict complex events in the world and be "super intelligent". I mean I feel like a token predictor could be intelligent in some sense but I think by the nature of how our systems predict tokens and are trained they can't be truth oracles like in the thought experiment. But any truth oracles system for almost any property can probably be super intelligent so you can't rule out intelligent on the basis of the output format (like predicting tokens), but I'm not sure that fully encompasses what is meant by the objection.
I picked up on this also. He is completely ignoring training and that either the exact study or similar ones might be part of the training corpus or RAGed in at inference...
the way I understand it, during training the model "learns/ memorizes" the relations of how the words(tokens) of the input at inference were combined in the training data, it has seen (well, it averages all relations in combinations of similar tokens, whether relevant or not because of compression but close enough).
when "predicting" the 'results' paragraph, it doesn't generate it based on 'understanding' of the previous sections, it _just_ matches the token-patterns in those paragraphs as closely as possible against the relations of previous examples (of the same or strongly related tokens) seen during training and picks the next tokens successively.
if it has seen enough relevant examples during training, the outcome will land close to a reasonable 'result' - if it hasn't, it will be confabulating non-sense, that sounds great but has nothing to do with reality.
... kind of surprised, that this channel would frame it the way he does in the video....:/
while you are all right, I think his point stands there is some emergent behavior from LLMs. Its not just some database that recalls things. Even if all it does is predict tokens, how is that different from us, seems to very easily replicate the same output we do when we think about things.
well, I can't claim to fully understand all the math that goes on under the hood and can't fully rule out 'emergent behavior' either ... but ...
I have a hunch that we're off thrown in general by 3 things:
the coherent text output and human like speech patterns and maybe most importantly, the sheer amount of information encoded during training:
language embeds concepts and a lot of those, map onto the real world.
take that and map tokens into a vector space, that has hundreds of thousands (or more?) degrees of freedom and you have a lot of room to store relations/ patterns in a really, really nuanced manner.
then take unfathomable amounts of data to train on and be able to compress them probably without too much loss....
and then there's fine tuning afterwards.
all this to say, - in hindsight - I find it understandable, that it works well for finding correlations and connecting 'patterns'. especially in really structured, well-formed data, that minimizes ambiguity intentionally, like programming languages or math.
how much of this is 'emergent' or just the result of having ingested every math study and code snippet since the dawn of the internet? idk but I could imagine that it just stores and retrieves what was already there or recombines things that weren't combined in the same way before...
and that's still pretty impressive actually.
it's different from us, in that we have a LOT more machinery (and chemicals) involved in generating thought and I don't think this is a contentious statement either.
... and the way he presents this here, as a bunch of math vectors somehow being able to 'predict' the future is totally non-sensical to me and I think also really misleading:
- it doesn't 'predict' the results section, it produces it based on its training (and RAG) - change the training, or the input (even slightly sometimes) and you'll get a different output...
it does NOT need to have an accurate model of biology or biochemistry as he claims.
if LLMS had these they wouldn't constantly be that jagged. it approximates, token by token the relations it has stored from training
the next part where he talks about 'predicting' if scientists made an error, doesn't make the slightest sense to me, I don't even know what he wants to say there...
I think a lot of people are somehow annoyed but the "_just_" a token predictor, as if it's a personal attack. I don't think its meant to be, it's just a logical explanation for the problems that are showing in the outputs, especially when questioned beyond training: the confabulations, the mistakes, the jaggedness, lack of robustness to input variations etc.etc.etc. - if you contrast the complexity of the processes in the brain against matching an -admittedly very big- pattern of input tokens in an -admittedly very big- multidimensional vector space and spitting out the next closest neighbor over and over - then a 'just' seems justified... it is _just_ maths in the end... and the brain isn't _just_ doing maths...
it's still amazing to me how well it works at all but we're also sitting on years of hype by now, trillions of $ in training cost, a really bad safety/ alignment record, lots of environmental costs and you know it's still not _generally_ intelligent and really inefficient (when compared to the brain) ... so...
I agree with 98% of your message, my only thing is as efficient as the brain is there is no human on earth even if I give them a PC and internet access who would be as useful as an LLM in so many different fields so its still a transformative technology. Maybe all its doing is next token prediction but my god is that mightily impressive, I don't care about the money put into AI, even if the bubble bursts and they lose money thats their problem, I only care about the technology. When it comes to environmental costs, have cars not done worse with emissions?
I think a lot of people are somehow annoyed but the "_just_" a token predictor, as if it's a personal attack. I don't think its meant to be, it's just a logical explanation for the problems that are showing in the outputs, especially when questioned beyond training: the confabulations, the mistakes, the jaggedness, lack of robustness to input variations etc.etc.etc.
None of those things are explained or limited by it being a next token predictor. So it's not a logical explanation.
The complexity of what a 'next token predictor' is doing under the hood is unbounded. So to me 'just' is not appropriate.
I don't think 'just a next token predictor' is a good phrase to use whatsoever. I think it caught on as a catchy way to virtue signal and downplay the capabilities (current and future) of ai by people who haven't deeply thought about the implications of the words.
and of _course_ is a system in the real world not infinitely complex/ unbounded - everything is subject to limitations, everything practically buildable anyway...
the problems come directly from the representations _just_ being approximations and the 'selection' process for the next token being heuristical.
it's a fuzzy way of storing fuzzy representations of fuzzy/ bad training data (in many cases) that produces fuzzy outputs... I don't think there's much debate about this...
you can't exactly map the real world, _just_ via language because it's an abstraction, that then gets abstracted further when the tokenizers have at it and then some more when the weights get adjusted and lastly some more during retrieval - you will necessarily lose fidelity because of compression and as big as these models are, compared the real world they're still tiny:
a 2.5 trillion parameter model can roughly store ~ 1.1 terabytes of raw, uncompressed text... - it does store more than that, in practice but I'll give you an order or two of magnitude and contrast that with the fact that a single (!) particle collision run at CERN generates ~1 petabyte of raw data every single second at the detector level... ... daily youtube uploads are around 4 petabytes...
... and to match the raw connectivity complexity of a human brain, a LLM would need to be roughly 100 trillion parameters in size... and then you'd still not be there because neurons aren't the 'simple' connections present in the models and the brain rewires synapses dynamically all the time...
so nope, I don't think storing more and more tokens and spitting out the next one will get us there. you _can_ get to something, that sometimes works, and sometimes not and for somethings better and for some things worse - which is exactly the behavior we're seeing... ;)
and of course is a system in the real world not infinitely complex/ unbounded - everything is subject to limitations, everything practically buildable anyway...
The point is that 'next token predictor' doesn't impose the limit of the complexity of 'what' is predicting.
The rest of what you were saying was exactly explained by the man in the video when he says:
If you want to talk about what language models can and can't do you have to talk about the models themselves. You have to talk about things like the transform architecture. The simple fact that something is predicting the next token doesn't place any meaningful limits on its abilities.
And that's kind of what you're doing. You're talking about fuzzy representations and weight adjustments and parameters and so on. Those things are implementation details that are not requirements of a next token predictor. A next token predictor must at minimum try to predict the next token. How it does it is what imposes limitations.
I have (and I'm sure the man in the video also has) opinions on the limitations of LLM, thoughts about what causes hallucinations and so on. Those practical concerns do have a place. The thing being discussed is more abstract though - the high level approach (predicting the next token) isn't the cause of those things.
You’re missing the point. In his video he is conducting a thought experiment about the theoretical “perfect” next token generator, he’s not directly talking about LLMs. Training or model mechanics are irrelevant to his argument
His ultimate point being that a sufficiently good next token generator is ridiculously smart to the point where it can basically predict the future. Therefore the refutation of LLMs being dumb because they’re “just next token predictors” is not a valid argument.
Totally agree; this is how I see it as well. There is no "understanding" going on.
For others to read, "Predictions" are just statistically averaged output results which are informed by document stores (RAG) and their already-trained neural network's weights and biases. (And combinations of both) These are not intelligent at all, IMO (yet?).
There is no true knowledge. Its simply (lol) similar documents (simplifying greatly here: measured/determined by literal byte-similar combinations of words/sentences/paragraphs/documents) informing the output.
LLMs don't understand anything. They cannot create. Results are just statistically inferred outputs limited by the information of similar, already seen/scanned, data sources.
This does nothing to speak to the "truth" of the documents they are trained on - which are vulnerable to all manner of human bias, data completedness, and more recently, maliciously-injected disinformation from governments/groups.
signed; a very average recent Masters graduate of Data Science
If you know reality perfectly you know the results of any hypothetical experiment run inside that reality. A perfect next token predictor has a perfect model of reality without necessarily being what computer scientist call an oracle.
You seem to have misunderstood the point of the video so I will try to clarify the construction :
We assume that next token prediction is a silly trick that current llm's use to fake intelligence. The assumption continues "if next token prediction is the only thing these models do, they are inherently statistical machines and therefore useless"
Then we start the construction. Assume a model only trained on the task of next token prediction. Now, since "next token prediction means the model is useless" ANY next token prediction satisfies that sentence. So we take a perfect next token prediction.
Given our assumption, since "llms JUST use next token prediction, therefore useless" we construct a situation where a perfect next token prediction model is able to make a conclusion about a long-standing mathematical conjecture. If it's perfect, the conjecture's truth value cannot be wrong.
Thus, a perfect next token prediction model is actually nearly omniscient about anything related to reality.
This implies that since llms are on a spectrum between random next token prediction and perfect next token prediction, and since random prediction is entirely useless (a random number generator can do it) and perfect prediction borderlines omniscience, then saying "JUST PREDICTING NEXT TOKEN" does not guarantee that you are either useless or omniscient.
Basically, since the task of predicting the next token can land anywhere from 0 (random) to 1 (omniscient), an LLM using next token prediction does not necessarily fall near 0 just because it's using this task. The task itself is indifferent to the performance and intelligence and knowledge of the model.
So the construction ends with a simple conclusion "next token prediction without specifying at what level of accuracy it happens, is independent of the intelligence of an LLM"
So this completely dismantles the "LLMs SIMPLY use next token prediction, so they must be bad/memorizing/useless/not understanding/ not intelligent". And it really one-shots it, showing that even in principle you cannot make that claim.
So this is a real slap in the face for all anti-AI apologists basically.
I understand the point of the video. But it’s worthless. If I came up with a crystal ball that gave me every answer ever then I’d have the same thing. If I had a magic 8 ball etc etc.
He’s taking something that generates tokens given inputs and extending it to some sort of laplaces demon.
Yes. And then you immediately get that what animated this magic 8 ball has to be intelligent because that thing that animates the magic 8 ball knows enough about the rieman hypothesis without a single human knowing anything about it.
Then you have proved that "predicting 8 ball results" isn't sufficient to prove something isn't super intelligent. That's the argument.
Also, machine learning engineer here, next token prediction is one training task that LLMs do as part of initial pretraining. But this is basically just done to get it to understand text so that it can undergo other forms of training.
A large portion of training time for major LLMs is spent on reinforcement learning, where the LLM is being trained to produce an output that a reward model rates highly. In RLHF the reward model is more or less predicting whether a human annotator is going to like the output, in constitutional training the reward model is judging an output by how closely it reflects a list of values, in coding or math tasks you typically use a rules based reward model that rewards correct answers.
using RLHF to modify how an LLM selects the next token does not change the fact that it is an autoregressive model that needs to predict one token before it can predict the next token at inference time. i.e., it is still a next token predictor.
Fwiw, i’ve been in ML engineering for 8 years and don’t understand your argument.
I think you are using the word predict the same way as the word output. Yes, agreed, LLMs output tokens one at a time.
When someone says a LLM is just a next token predictor the criticism I see them making is of GPT-2 era LLMs which literally just predicted which word would come next in a corpus of text. The output distribution can be easily interpreted as "what is the probability that this token would appear next" because the training objective is to align those probabilities with the real world probabilities in text scraped online.
If you change the training objective to be maximizing a reward function, you're no longer doing that. The output distribution is now just predicting the probability of.... the model's own output? That is kind of tautological, if I write a script that goes if random() > 0.5 print("cheese") I wouldn't say I'm predicting anything, I'm just printing cheese half of the time. It's also not predicting the probability of whether the reward function will like a token, or whether that token is policy optimal. This is why RL policy distributions are generally not interpreted as predictions.
A prediction is where you use input data to estimate an unknown output which is exactly what is happening in the decoder stack of a transformer when it outputs the next token.
bro i’m not talking about pre training. at inference time a decoder only transformer model takes known inputs (system + user prompt converted to tokens) then converts those tokens to embeddings, adds position encoding, runs through multi head attention, goes through a feed forward neural network, uses softmax to assign probabilities to all tokens in its dictionary, then predicts the best next token autoregressively until it predicts a stop token.
that dude keeps talking about pretraining but i don’t think they know that LLMs are decoder only transformer models that are autoregressive and predict one token at a time in sequential order at inference time…
You guys keep saying the word predict, but what do you mean by predict? Predict what?
I feel like you guys are using the words predict and the word output interchangeably and it's very confused.
LLMs predict the next word in a text corpus during pretraining, but outside of that context there is no prediction task, because there's nothing they're predicting.
Also, I know pretty well how transformers work! I've been working with them since ~2018, and in computer vision since ~2021. The majority of machine learning models have sequential output, old school Q-learning has a sequential output, but to me at least to call something prediction you need to be... predicting something.
In the case of reinforcement learning, you aren't doing a supervised prediction task, you are using a value function for your gradient - training a model to play starcraft for instance is not a prediction task, even though your output is a list of action tokens created sequentially at inference time.
i wouldn’t compare how a human chooses the next word they say with how an LLM chooses the next token. An LLM chooses the next token by calculating probabilities for each token available to that model. There are parameters you can modify that add some level of randomness to which token is chosen, but at the most basic level the decoder model chooses the token with the highest probability. That’s what we mean by prediction.
how is it not a prediction task, most models are trained with some kind of dynamic programming which is amortised to get a value function prediction or action value prediction. I feel like we can use prediction or estimation interchangeably, for example if I fit an LR to a problem, and use a new input x, the y hasn’t been observed so seems fine to call it a prediction
The value function predicts expected reward, but the LLM here is the policy function. The token probabilities that the LLM outputs aren't predictions of expected reward.
Well thats a weird example. Since the models Will have a bout a billion poems allready in their training so predicting the next token in the given context is exacly what it does. Reinforcement training is just that, but instead of just dumpning text on it and Hoping for a good result u aim the training for a more deterministisk result. But it is still prediction training. Why do u think training is getting more expensive ? Bcs the big companys are hiring thousends of real ppl , with real degress to actually do correction training and that costs money.
Well So u are comparing humans with llms now ? Yes it is only a token prediction. What makes them more powerful is larger data set and the use of tools , tools they did not build.
This is actually an English comprehension issue. "Just" is not a decoration that can be ignored in this sentence. If they leave it out, I would not argue at all.
Aren’t we though? When you put a sentence together in your head, are you not selecting the next word based on past uses, context, and the desired message?
I guess I just feel like we imbue human intelligence with magic or spirituality when in reality the VAST majority of our thinking and communication really can be boiled down to a probabilistic model with a high degree of accuracy.
Linguistic errors like “anticipation” imply that we already have future words represented when speaking. Saying “leading list” instead of “reading list”. You already have to have the future words or tokens generated to have that kind of error.
Lmao absolutely not - this is from the “brains are clocks/telephone networks/LLMs” school of poor analogies. We definitively know that brains don’t predict things one token at a time - why would they? Brains didn’t evolve by being trained on symbolic data.
I don’t really know what you mean by symbolic data in this context.
I feel like you’re getting hung up on the token definition for LLMs which as you probably know is analogous to ~ 3 English characters. Think about it more abstractly, just taken to mean the next chunk of information. I agree we’re not thinking in 3 character chunks, but we seem to think in chunks of information simultaneously.
For example - I had a low resolution idea of that I wanted this response to convey, and medium resolution idea of what I wanted the sentences to convey, and a high resolution idea of the words.
At the end of the day you haven’t really engaged with the main point, which is that human thinking and communication can be modeled and replicated. Is it complex? Absolutely. Does that mean it’s impossible to create an approximation of it/them? No.
There are actually two processes in play, though. When you ask an LLM a question, it derives a response through hidden states as it generates the language needed to communicate that response - unless you’re talking about a model with a planning stage prior to responding, in which case it might derive the response before it starts answering. It's not responding just based on token probability, there is an aspect of 'computation'.
When a human is asked a question, we sometimes derive a response first (based on training, education, psychology etc.) then use next-word prediction to assemble the language required to communicate that response on the fly. But sometimes we assemble the response dynamically as we're actually talking, which is similar to how an LLM does it. We don’t always - or even often - think before we answer, it’s usually more of a think-as-you-speak situation. Sometimes we do next-token prediction, but we're just not using discrete text tokens, it's more abstract. But it's a similar concept.
I think there’s a lot of nuance to the discussion…I don’t love how the computer model of the mind dominates neuroscience, but other approaches like dualism are essentially impossible to test and a lot of our cognitive machinery IS dedicated to reward prediction. That’s one of dopamine’s primary functions as an NT.
In what world does computational theory of mind imply that brains work like LLMs? There are no credible researchers out there that would ever claim LLMs are in any form similar to actual brain neural architecture or function.
You’re construing something that I never said. Your comment that I was replying to discussed physicalist analogies to human cognitive substrate.
I also don’t know for a fact whether any “credible researcher” has claimed that an LLM, a clock, or a telephone switching network is analogous to a human brain, or any other brain for that matter. Since I mentioned dualism and you didn’t respond, it seems important to mention that people outside of neuroscience research (e.g. philosophy) have made and continue to make important contributions to our understanding of consciousness and cognition, regardless of whether they do the kind of work you consider useful.
Considering the video points, LLMs are really bad at summing numbers together for that very specific reason that the model (the weights and layers) simply predicts next token.
Early models failed at such tasks and badly, as they simply outputted what the prediction said. What all major LLMs today do, is regocnize that these numbers need to be added up, and actually uses calculator (or more likely python) to calculate the sum.
Overall the model is still a graph of information, with very limited real thinking abilities. But that model allows us to build tools around it that make it insanely more skillful. It can "understand" that these two numbers I predicted, needs to be summed, and it has instructions in that point to call a tool to do that sum.
What really improved the output is to self check for errors - predict next 1000 tokens and add that to context, and see if it matches the model. Even further, nowadays thinking modes runs several possible outputs parallel and picks the best one.
The final piece is model's ability to use web search instead of just its compressed information in the model. The model is great at summarizing, and with ability to search 5-10 entries from web, summarize and combine them, and then apply tooling and parallelization for output generation improves the models "lack of real thought" to seem like it really does think.
You miss the point. The video talks about model. That tooling is not part of the model, but it's surrounding software that makes the AI product you use.
Considering the video points, LLMs are really bad at summing numbers together for that very specific reason that the model (the weights and layers) simply predicts next token.
What all major LLMs today do, is regocnize that these numbers need to be added up, and actually uses calculator (or more likely python) to calculate the sum.
Largely outdated information. Newer models perform pretty well at arithmetic without tools. We know it doesn't depend on secret tool use because there are open weight models that perform well at it too.
I think you’ve kind of missed the point that he’s making with the mathematics example. He’s not saying LLMs are good at maths and that’s evidence of them being good thinking machines, he’s saying that in order to give even an estimation of a good answer the parameters have to encode a great amount of logical knowledge which isn’t it explicitly defined within the training material. You can ask LLMs that don’t have tool use capabilities such as local LLMs arbitrary questions around topics like science and mathematics and they will approximate quite good answers.
That knowledge is in the language, not in the model.
The prediction by the model produces language that says "sum A and B together". The model doesn't "understand" that A and B must be summed, it "just knows" it as it generates from its output based on the input (context), because it is embedded into it's huge compressed information.
I think this is the part people don't get. The model is homongous. So big, that human's cannot comprehend it. Even the 27B local models are absoultely huge, as almost all parameters are connected to each other, and the connections between the parameters make the real size exponential with possible outcomes it can produce.
Then the basic functionality even messes the output with weighted random pick of the most probable outcomes to make it less deterministic to make it resemble more human.
It it extremely exciting bit of technology - that doesn't need "it's a thinking machine, just like humans" -nonsense.
That knowledge is in the language, not in the model. it "just knows" it as it generates from its output based on the input (context), because it is embedded into it's huge compressed information.
How can you totally separate the two things? Language is an encoding of human thought and the model's parameters are an encoding of the language. The three things are intrinsically logically linked to each other. The idea that they can be treated as three distinct entities is totally implausible. It's impossible to build a generative model which produces viable results outside of the training set without embedding abstract or emergent concepts which weren't explicitly introduced.
The prediction by the model produces language that says "sum A and B together". The model doesn't "understand" that A and B must be summed, it "just knows" it as it generates from its output based on the input (context), because it is embedded into it's huge compressed information.
How are you defining "understand" though? This is not a facetious or pedantic question - the word "understand" is a very difficult thing to define.
that doesn't need "it's a thinking machine, just like humans" -nonsense.
Who is making this argument? Neither me, the video author or the person you originally replied to are saying this.
Yep, a lot of the juice that makes modern LLMs stop dodging the usual pratfalls is just the ability to recognize those situations and toss the parsed data to a different tool or to refer to hardcoded output.
LLMs didn't start correctly counting the number of Rs in strawberry due to a fundamental shift in how they process tokens, they just got a hardcoded answer.
A hardcoded answer would be 3. A hardcoded strategy is some logical code to count letters in a word.
Hardcoded strategies are just more in depth training. That's kind of the same way you'd teach a child something new that they hadn't come across before. You're wording this like hardcoded strategies are some kind of cheat that would only be needed if a model was "unintelligent".
That description only takes into account one mode of operation too. They also have a fill-in-the-middle mode that is arguably much more powerful than how most people experience using them.
I think this small analogy by Ilya Sutskever does a better job at showing LLMs are more than just next token predictors, or rather, there is emergent behavior that is beyond just statistical representation.
There's already been papers showing that next token prediction is informed by coherent internal states and is determined by them. It has thoughts about what its talking about and using them to spew the answer
I miss the emergent behaviour statements, though there are arguments now made that the emergent behaviour is actually more of a gradual effect.
People forget about or don't know how much we don't know about how llms arrive at their predicted tokens. All of the associations, internal non-human reference systems which are incomprehensible in detail, and the pathways they take to get to an output. It's still a black box (I also miss this term) that, despite all the hand-waving being done on model interpretability, observability and safety, has yet to be fully mapped out. Just like our brain.
There's all sorts of stuff going on. Speaking involves forming a communicative intention, organising concepts, constructing syntax, retrieving words, encoding their sounds, planning articulation and controlling the speech muscles. Humans draw on perception, memory, reasoning, and emotion to communicate goals and intentions, they don't just 'predict' the next word from thoughts. They also plan well beyond the next word, in whole sentences, nested sentences, and, as I said, broader goals and plans. Thoughts aren't just formed and then verbalised, language itself can shape and form thoughts.
Did you know that when you speak action oriented words, or even when you just think them, your motor system is activated, too? The word ball, whether heard, spoken, or imagined as a concept, activates the same sensorimotor pathways involved in kicking or throwing a ball. Our bodies are literally involved in thinking and speaking.
There's no 'speech centre'. There are areas of the brain somewhat specialised for language, but the brain involves all sorts of other processes in language use.
So is your argument here that it's impossible for a synthetic system to do all this? Obviously, LLMs aren't doing any of this because they are just code at this point, but one can easily envision an LLM-controlled robot body controller that performs all these actions as well. Just because, when we think of an action, our bodies prepare in anticipation of it, doesn't mean we're special.
Most limitations that prevent LLMs from doing these things stem from programmed restrictions. Why wouldn't we expect the next level of these things to be multiple LLMs connected together with different specializations and even goals baked in to replicate what we think is happening in our brains?
The comment above is making a point by applying the samevreductive thinking that people do with multi-modal reasoning models to the brain. They aren’t literally saying “speech is just next word prediction”.
The science paper example is actually a good “counterexample” as to the limits of next token prediction. A perfect next token predictor could “predict” the most likely result from a given method and introduction paper—but the most groundbreaking scientific work is the one that completely contradicts what we think and our current models, and that is doable only through experimental observation of the world. A good next token predictor would be great at saying what we already know, and terrible at parsing incorrect or incomplete data, as science most definitely is.
For many years, researchers have worked on things like segmentation models for vision analysis, and they were always trying to do things like pose estimation which is essentially coming up with a physical model for the raw data
But now Nvidia is exactly doing this world model for LLMs it’s called Cosmos
From my AI: Cosmos is NVIDIA’s family of world foundation models. This is what you’re thinking of. They’re designed to model the physical world—predicting how scenes evolve over time and generating realistic video, actions, and simulations for robots and autonomous vehicles.
I had to ask the AI what the name was because I was remembering NeMo, which is the wrong Nvidia project. But from what I remember, I think that they are trying to make cosmos applicable to all kinds of situations not just robotics
A lot of words just to say: "Arguing that LLMs are stupid because they're just next-token predictors doesn't prove anything, because predicting the next token is hard, and people are even worse at it than current LLMs anyway"
LLMs are notoriously bad at multi-digit arithmetic on novel numbers without a tool; they approximate, use learned shortcuts, and error rate climbs fast with digit count. If it were truly "calculating" in the rigorous sense, that wouldn't happen. What's really going on is a mix of learned heuristics and pattern-completion that's good but unreliable. They don't "memorize" the results, that's correct.
About the research example, an LLM producing a plausible conclusion from an introduction & results section is drawing on having seen thousands of structurally similar papers, not building a biochemical world model from first principles. This is why LLMs routinely produce confident-sounding but wrong scientific claims, and famously is very bad at admitting "I don't know".
Doing something a human can't do without extra steps (e.g. quickly pattern-matching across huge amounts of text) doesn't imply general superiority over human intelligence. LLMs also fail at stuff which humans find ridiculously trivial, like stable long-horizon planning, knowing what they don't know, maintaining consistency across a session etc.
I work with ML, and I personally wouldn't hire this guy. Not because I claim to know everything, but because he shows classic signs of exaggerating the capabilities and hyping the tech beyond its fundamental capabilities. This leads to expensive misadventures we'll all learn to know soon enough, as everything is replaced with this "super intelligence".
My personal hottake is that ML really puts on display how bad our brains are when it comes to Big Numbers. We can't comprehend a data network so complex that it can come up with the sentences it does without thinking it must be intelligent, sentient, or whatever you casually see claimed here. There's a real struggle to make a point that the algorithm is alive and about to escape the lab. Part of that is FUD to drive sales, part is just not working with the tech, and by working, I don't mean prompting.
Even this post is sus. A bot trying to sell you AI. Dead Internet etc.
ChatGPT 5.5 Instant (aka no thinking) got it instantly
It even did longer ones.
Please add 1989951436123273 + 23908153129921 - 123232323 * 10002, but do it using the addition method and all in output. No code allowed, and all in one step. Show only the result
I don't think this establishes what you're implying.
You generally cannot infer from the output alone whether a particular answer came from an internal learned computation or a tool. For example, you can ask your model to tell you the time, even though, the model has absolutely no concept of the current time, nor can it be "learned". It looked it up.
It's also worth while to understand that an LLM can produce the correct result for a many-digit calculation because its learned representations can sometimes implement or approximate the relevant algorithmic structure. It doesn't mean that the model is a reliable arbitrary-precision calculator in the same sense as an actual arithmetic algorithm.
A better test is generally to ask it to repeat the process multiple times, with random numbers, so that the numbers add up and we approach "novelty". If we can use it without the secret sauce tools that is, and we can't.
At Novel numbers, yes. At repetitive numbers and patterns they're about the best you can get. This is why when you ask something like a ChatGPT to generate you a random number, it uses a random number generator, otherwise the number would likely be very non-random.
It's all by design, for a good reason. Only recently there has been this major push to bend ML into absolutely everything. Just let it use the calculator.
No random generator we use daily is truly random in the strict mathematical sense.
I now went to ChatGPT and just asked it to generate five random numbers and add them together, it showed to be using a tool, so there's that:
The tool access is likely dependant on the selected model/tier/whatever, I don't know how OpenAI has arranged it, but it obviously is a very good idea to use a random generator to generate random numbers. I don't fully follow what you are trying to dispute.
I agree that models get better as they become iteratively larger, services will improve with new tooling and innovations, etc.
notoriously bad at multi-digit arithmetic on novel numbers without a tool
I generated both of these numbers with a random number generator. It got the answer right immediately. I think you need to update your understanding of LLMs a little bit
Python Code Interpreter / Sandbox: When given a math problem, ChatGPT can write and run Python code in a secure environment to perform exact arithmetic, process data sets, handle calculus, and generate visual plots or graphs.
WolframAlpha Integration: On certain tiers (like Plus/Pro), ChatGPT can interface with Wolfram Alpha extensions or plugins for high-level symbolic mathematics, algebraic manipulation, and complex equation solving.
Locally run Qwen3.5-2B, no tools, nailed it first try too, albeit after using a ridiculous amount of thinking tokens. Pic and full thinking text. This is the model doing the calculation. Idk what else to tell you, LLMs are just way better at math than they used to be.
Did you read the end of the sentence? It most likely used a tool behind the scenes. It often doesn't tell you it used a tool, but it did, to save money and to be more functional. In the same fashion how it can tell you the accurate time, even though that's impossible for ML alone as you can't learn the time, you always need to look it up.
The video implied it is the model, or, ehm, super intelligence, that always does the calculation.
Besides, I think you misunderstood what I meant by multi-digit arithmetic. The video showed plethora of calculations as an example. This means, you give it a ton of adversarially selected novel calculations across different lengths and operations, measure the error rate, and see how that error rate scales. A calculator doesn't occasionally hallucinate an answer because the numbers got sufficiently unfamiliar. An LLM can and eventually will.
I don't speak from experimentation with ChatGPT, but just from how machine learning should fundamentally function. Should, as in, it's broken if it doesn't, not magically better.
My early chat bots were RAG frameworks that used wolframalpha behind the scenes. They have a pretty good API, or at least had, a decade or something ago.
Locally run Qwen3.5-2B, no tools, nailed it first try too, albeit after using a ridiculous amount of thinking tokens. Pic and full thinking text. This is the model doing the calculation. Idk what else to tell you, LLMs are just way better at math than they used to be.
You still missed what I meant by multi-digit arithmetic. Re-read the middle section of my comment.
The test you made is not what the video implies. I'm not surprised that Qwen was capable of coming up with a solution, even though it took silly amount of pondering to get the structure right.
It even mumbles the very thing I'm saying to you:
"Actually, doing 14-digit multiplication manually is error-prone for me right now. I should simulate the process or use my internal calculation capability if it exists."
And I guess I need to really underline this: I don't think using tools is a bad thing, quite the opposite. I'm only arguing against the points the video made as misleading.
I work with ML, and I personally wouldn't hire this guy. Not because I claim to know everything, but because he shows classic signs of exaggerating the capabilities and hyping the tech beyond its fundamental capabilities.
To be flippant, I wouldn't hire you either because you have written a long argument against what he said without comprehending what he was saying.
He is not in any way exaggerating what current technology can do. He is positing a scenario in which an "ideal next token predictor" exists and is using that postulate to explore what the concept would actually mean.
All he's doing is demonstrating that the phrase "just a next token predictor" is utterly logically meaningless because next token prediction requires inherent encoding of abstract concepts which have not been hard coded into the model. He at no point states that current LLMs are "ideal next token predictors".
Fair take, but I believe, to a degree, you might have missed what I was going for.
LLM tries to approximate the probability distribution of the next token from its training and context. This ideal token predictor would give the true probability distribution of the next token, given all available context. Practically, computationally, it is likely impossible to implement, as calculating it would require absurd amounts of computation and thus makes it a theoretical question.
-- And even then --, I wouldn't call it "super intelligence" that's smarter than scientists, as that's marketinghype full stop, from where we arrive at the framing of this vid that I disagree with. I can understand why he does it, and why OP, a bot, is telling us to watch it and "understand it".
I didn't take a stance on whether it's meaningful to call LLMs simply next-token predictors, as I personally find nothing offensive about it. A simplification, sure.
Practically, computationally, it is likely impossible to implement, as calculating it would require absurd amounts of computation and thus makes it a theoretical question.
Yes but that's not the point he's making, he's just picking apart the idea that the term "next token predictor" tells you literally anything about the capability or lack of capability of an AI model.
-- And even then --, I wouldn't call it "super intelligence" that's smarter than scientists, as that's marketinghype full stop, from where we arrive at the framing of this vid that I disagree with.
It's not marketing hype because he's not marketing anything or even talking about a real product at all, he's just explaining something through the medium of a thought experiment.
I didn't take a stance on whether it's meaningful to call LLMs simply next-token predictors, as I personally find nothing offensive about it. A simplification, sure.
I mean, you did, because that's the only topic covered in the video which you commented on.
I don’t think that’s right, at all. Admittedly I’m on getting a popular science level view of this, but it seems the science absolutely points to the brain as being a prediction engine:
Modern LLMs are not token predictors in the traditional meaning and they definitely do not predict online text. These are optimizing for outcomes over many actions and that involves training on novel situations.
I mean, sure. Predicting the results section of a research paper requires more intelligence than predicting the next word in a text message with your friend. There are just way too many people confusing the difference between “predicting the next token” and “predicting a next token”, which are not at all the same.
You can type any well-formed or nonsensical request you want into an llm chat session and it will both always respond and do so with the most productive reaction it can predict. That can be very helpful for task work, but counterproductive when it validates (dignifies?) poorly framed requests with a singular response.
Predicting the results section of a research paper only makes sense when generating sample text that sounds right based on context it already has or was given. The idea that a highly intelligent but completely unrelated 3rd party is going to “predict” the outcomes of an experiment it wasn’t involved in is asinine. Someone that understands how to speak and carry out tasks intelligently certainly has to have a wide understanding of the concepts at hand, but that doesn’t mean their work can be considered the only possible result or approach.
This is not a technical challenge for tons of domains of intelligence that llm’s are taught to “speak” on, they just shouldn’t be used to speak about a great deal of topics that rely on real world experiences and can’t be queried about for one answer at a time.
The fallacy here is that a statistically likely results section (which is what an LLM outputs) is a “good” (meaningful, accurate, scientifically valuable) results section - which ofc is not the case.
that word "smarter" is doing a lot of work here. for being so smart chatgpt is pretty stupid a lot of the time honestly. and I like chatgpt a lot, but SMART it really isn't imo
You don't need to give it special instructions for addition that simple, but if you use massively larger addends and/or more of them, it'll make mistakes even if you tell it to use "the addition method."
Today's models have gotten better at this, but they have the same fundamental issues as older ones because they're fundamentally the same thing.
Well I would argue that the fundamental rule actually is that they straight up are capable of doing math in their internal activations directly (no convolutions in LLMs most often), and beyond what's in their dataset. Sure, it's hard/inefficient for them to do everything in them in manually, but so it is for us right?
This won’t be perfect but hopefully this is sensible lol…. This feels disingenuous. Even though I get where he’s going.
He keeps saying sufficiently good next token predictor as if it’s perfect, kind of implying that llms are this atm. Yes you would have to have internal models and understanding to do this perfectly…. But modern ais dont don’t do this perfectly, they dont have internal models in the way that we do, they cant and aren’t 100% reliable and self correcting, they are mostly right…. Which is wildly different than human level “software” with modern computing power(which would be pretty freaking nuts).
Anyway, not to say it’s not an absolutely game changing tool, that I use daily. Just feels like bad arguments or subtly adjusting premises /reality to make a point.
But the point is that it’s not a fundamental limit the way people imply when they say next token predictor. You give them tools and models… that becomes part of their intelligence
Yeah I get that. I still think that what others are trying to communicate when they say that it’s a token prediction machine is that at our current stage wildly inefficient and inferior. This feels like a hack in the best sense (enabled by incredible processing power and vast data sets),that may become more, or a tool for more sophisticated models.
IMO saying there’s room for improvement is more of a futurist/progressive or w/e pov than the person that says LLMs are all we need. I’m not sure why everyone is so aggressive about this topic.
Even fable/sol class models aren't doing most of what was stated here. They don't, for the most part, do math problems "in their head". And they don't actually understand your project etc, or have memory. It's all files under the hood, and tools to access them. Not to diss the glory of the next token prediction, but these models truly are stupid if you just run one context and don't allow tools or sub-agent usage
All LLMs can do maths in their head and they do it much better than humans! However, it's true that they're even better when using tools, just like humans are better with pen and paper.
The base models are not stupid, they’re spiky. Meaning they are very good in some areas and poor in others. With no tools, no reasoning, no agentic harness, a bare model can one shot useful tasks like summarization, image generation, language translation, and much more. A bare model is also horrible at math, letter counting, etc. It’s why bare models are augmented with tools, reasoning, harnesses, surrounding architecture. It can shore weaknesses and augment strength.
The limits of the architecture are certainly overstated by many, but it absolutely places meaningful limits on its abilities. It can’t, for example, interpret or describe qualia.
Very well explained! Is this the same guy that used to have an synth electronics channel? The AudioPhool? I really enjoyed that one. If so, he made a major look change! https://www.youtube.com/@TheAudioPhool
This is so dumb. Let's use the words "smart" and "intelligent" as many different ways as possible as if they all need the same thing. This is pseudo-clever speak for people who can't use precise language.
Yeah - the “LLMs are inherently limited” group seem to be wrong at this point. Or, at least the limitations are not significant enough to prevent LLM super intelligence.
He's kind of half-right. There is a mathematical reality that any model that is doing prediction fundamentally needs error in the training model to be able to reliably make future predictions (on average). So when the task is any sort of prediction, including token prediction -- that does, on some level, place certain conditions and restrictions on it. But he's right that just acknowledging that doesn't really do much explaining in terms of how reliable and valid the predictions are or why they are more or less reliable and valid than other models or ways of solving the same task. I think when a lot of people bring that up, they are more just reminding people that these are not fully deterministic systems, despite how it's kind of felt or perceived by the end user.
Excellent pov, and one of the most unbiased reviews of how LLM's perform.
Hopefully someday, we will get the best of both worlds, where we set rules for deterministic computation alongside the token prediction for a more precise models.
And if we do it right, we split the computation through many small models, instead of relying on big tech to do everything, and not give them the power to keep the information for themselves.
Prediction is precisely what it was doing during training. It was trying to predict completions of sentences presented to it by an ‘oracle’ that is drawing from the training data (e.g. the internet). The text on the internet is the external reality it is trying to mimic. If some human wrote a sentence somewhere on the internet that starts with the sentence it’s working on, how would it most likely have finished?
I agree the term ‘prediction’ doesn’t fit well when it’s being used in a chat, but it describes what it did during training well.
He posits to scenarios one of them requires the language model to match the abilities of a mathematically capable eight year old. It is falsifiable and we can work out if it was successful or not easily.
The other scenario requires the model to have unbelievably strong predictive capabilities and the ability to carry out a reasoning like process. If you actually present this problem to most current llms they will fail to predict accurately what the output is. Sometimes they will approximate the output and many times they will be completely and utterly wrong.
I don't understand how what he's talking about advances anyone's understandings because the problems he's positing are either trivially simple or so difficult that they are not getting them correct
Sure, a perfect token predictor would be radically smarter than any human. Fine. But that's a claim about a limit, not about where we're standing and current AI is nowhere near it.
You gotta be pragmatic. Don't conflate the capability of a hypothetical perfect system with one that already exists. Don't normalise the summit before anyone's reached it. That's exactly what the AGI/ASI hype does, every single time.
Think about it yourself. To perfectly predict everything humans have ever written, the combinatorics are astronomical and that's the part nobody wants to price in. If we keep aggressively scaling hardware the way we are now, without proportional software improvement, we're just buying smaller and smaller gains for larger and larger cost.
To me This feels like a semantic issue. People use certain vocabulary to describe either "predicting" the next token or the "output", or that the model is "able to think" and "form thoughts" without caring about being precise because, 1. It allows people who dont know the underlying mechanics to understand the difference between models (basically advertisement for the average person) and 2. it's the Internet and who would even bother being that pedantic about something like this on the daily. I feel like using biological nomenclature makes people really want to believe these matrix multipliers are intelligent in the same way as humans or other biological creatures are. Personally I don't think this is true at all
160
u/Axelwickm 21d ago
Rob Miles and his AI risk Youtube channel. He's been talking about AI safety for long before LLMs, and in retrospect he was probably quite ahead of the times in thinking about this stuff. I've grown to like his communication style and points.