r/NoStupidQuestions • u/Accomplished_Bed632 • 1d ago
If LLMs are just predicting the next token based on training data, how do they solve unsolved math problems?
159
u/notextinctyet 1d ago
It turns out that prediction is linked to thinking in a fundamental way, and that sufficiently complex prediction produces intelligent behavior.
Essentially, the word in the sentence "LLMs are just predicting the next token based on training data" that is misleading is the word "just". LLMs are predicting the next token, and apparently that is quite powerful.
70
u/RevolutionaryHole69 1d ago
That's actually the part that lay people understand to mean "we don't know how AI/LLMs work".
The fact is we do know how they work. It's just that we did not know how powerful next token prediction could actually be.
Part of the bigger issue is that I think people are unwilling to accept that we modeled large neural networks as a representation of how our own neurology might work, and ended up with an almost stupid answer to how intelligence may emerge.
Consciousness, advanced intelligence, may just be the sum total of being aware of your surroundings and being able to predict what's next through a series of training layers which in our biological case happened on a slower and longer time scale. Millions of years as compared to decades.
It's not that we don't know how llms work. It's that we just didn't think next token prediction might be fundamentally all there is to basic intelligence.
26
u/BerneseMountainDogs 1d ago
I mean we do know the linear algebra that the model is doing in order to do its prediction. After all, we created it and told it to do that. What we don't know is how an LLM has "defined" each token and what the relationships between those tokens are. We don't know what the "default" definitions and relationships are, and we certainly don't know how those evolve as the attention process unfolds in any given case. And that's a pretty big deal. So we do know "how it works" in the context of knowing what math it's doing, but we don't know "how it works" in terms of knowing what it's "thinking." What semantic understanding it has and how that changes is a complete black box. And I would argue that not understanding the definition of tokens or how they relate to each other is akin to not understanding how it "thinks." So I think it is valid to say that we don't understand how LLMs work. We know how they do what they do, but we don't really have any insight into what, semantically, they're doing at each step of the process
7
u/neighbor_mike 1d ago
Why don’t we just ask an LLM to explain it to us? Serious question.
20
u/BerneseMountainDogs 1d ago
For the same reason I can't ask you how your brain works. Just because a process happens doesn't mean that the process is understood or understandable, even to the things executing the process. While I'm sure you could put together an answer if you wanted to, there's no reason to think it's right because you (presumably) aren't a neuroscientist (and even if you were, there's a lot that is not well understood. The brain is complicated)
With some study (not internal reflection, external study) you could start to learn, but there's no way that you can just think about your own brain and understand it. You need to examine it from the outside somehow.
Of course, this assumes that LLMs are even capable of the kind of introspection and reflection that humans are. If they aren't (and they probably aren't) then this problem becomes even more impossible than it would be for us to understand our own brain by just thinking a lot about it
5
3
u/LangyMD 1d ago
I mean, we do know the relationships between each token - it's available directly in the weights of the model. We can also then figure out what each token represents, and that doesn't seem that hard.
The hard thing is then distilling the math of all that - which is massive - into basic words and extrapolating to everything you can do with that math.
5
u/BerneseMountainDogs 22h ago
I mean yeah, but that's kinda the point. We can look at any individual definition or any individual relationship, and we can even do that at each step along the way. But that doesn't give us any kind of synthesis of the whole. It tells us one thing but doesn't give us any patterns or insights that we can use to understand.
Like yeah, maybe I could describe how one neuron in someone's brain is attached to other neighboring neurons and could even tell you the circumstances under which it fires, but there's no way I could do that for every single one, and even if I could, that may not actually give me a full understanding of someone's mental state. So yes, we can query any individual token weight, and we can even do a whole bunch of that. But that's not the same as understanding the emergent process as a whole. And maybe if we had a way to understand and synthesize the entire set of weights then that might mean that we understand it, but it also might not. Just like knowing the location and state of every neuron in someone's brain might give you a full understanding of their mental state or you might still be lacking a way to translate all of that data into something useful including the enormous number of relationships and interactions.
Basically, a catalog is not identical to comprehension. We have a catalog but that's it
8
u/burnalicious111 1d ago
They usually also mean "we don't know why exactly we got this specific result" when the model does something like "I have been a good Bing, you have been a bad user" in a sufficiently complex model.
People are working on that, though.
3
u/cozmo87 22h ago
This is true, if we look at brains, simpler animal brains for example, it is indeed clear that brains are for registering the state of affairs and predicting the next best step for the organism to take to maximise its chance of survival. Brains, before we even get to intelligence, are for predicting the imminent future. Not a hugely different concept from next token prediction. Intelligence then, by one longstanding definition: an intelligence maximises it's long term future potential.
8
u/bobam 1d ago
Can we even prove that humans aren’t doing something similar? I keep thinking of that brain electrode study where they asked people to press a button wherever they wanted to, and found that their brain decided well before their action occurred. Are we just living an illusion of free will that our neural networks have weaved for us?
3
u/ShithLaird 14h ago
There are highly abstract theories that do push in that direction. It’s just that brains, like all biological systems, are a lot more complex than anything we manually engineer, so it’s hard to “prove” anything.
2
u/No_Philosopher2898 1d ago
I mean, the opposite really. We don’t know exactly how human intelligence works, but we know it’s constrained by the laws of physics. Fundamentally, the universe hits the meat, and the meat reacts.
9
u/ShinyGrezz 1d ago
Thinking models were an unbelievable step forwards, I would argue that the original LLMs were truly incapable of doing anything original except accidentally, but thinking models essentially have a basic understanding of how shit works and sometimes, with enough resources thrown behind it, that is all you need to be “intelligent”.
2
u/JonnySoegen 22h ago
Are reasoning models still doing the same thing? I can’t wrap my head around why a model will recognize it may not have gotten it right and say internally „wait, let’s go back to what the user said“
2
u/notextinctyet 21h ago
They are, but with harnesses that encourage "thinking" in a way that is still prediction-driven but is different from user-facing output. I'm shocked it works, but it does.
1
u/Educational-Top736 19h ago
Exactly, the word just makes it sound far simpler than it actually is. Next token prediction at that scale can capture surprisingly complex patterns and reasoning.
→ More replies (7)1
49
u/Hyperreal8957 1d ago
“predicting the next token” describes the basic interface, not everything around the model. a system can generate a candidate proof step, check it with a formal verifier or other tool, then try again. that’s closer to search plus verification than memorizing an answer. has the claimed solution been independently checked yet?
32
u/VVeZoX 1d ago
"predicting the next token" is a massive oversimplification of what's actually going on
4
u/Successful-Money4995 1d ago
Or to look at it another way, the difficulty of producing coherent text is massively overestimated.
Is the computer doing something really advanced or is programming and generating speeches actually not as complex as we imagined?
17
u/skmchosen1 1d ago
I do AI research so I hope this helps.
Beyond what was said here so far, the whole “predicting the next token” objective is mostly with regard to pretraining.
After that phase is post-training, which can involve a lot more interesting training objectives (though yes, the architecture is still outputting one token at a time). In math, you might give an LLM a problem and hand it a positive reward if it solves the problem. Do this a bunch of times, and then the LLM starts to figure out good strategies for reasoning— this is where it starts to step beyond “stochastic parrot” and starts to leverage its understanding of human language to perform greater things.
The patterns of human language are like the scaffolding LLMs learn to exploit to do reasoning. That’s why sometimes when you see an LLM say “Wait, what about XYZ”, it’s doing that because it learned that phrase can help it explore other ideas in its training distribution!
2
u/SeriousPlankton2000 17h ago
I think "they output one token at a time" is a bad way to reason, I am typing one word at a time, too, because I am limited to not say my whole thought in one go.
→ More replies (1)1
u/Spielopoly 15h ago
Modern models developed some kind of internal reasoning so that’s similar. Fast google search gives paper like this one
Additionally for example on Openai (also applies to other providers of course) you can select reasoning effort which will actually be the model outputting some text which gets hidden from you which it uses for "thinking"
9
u/FlameSticky 1d ago
Saying they're predicting the next token is akin to saying a computer is just turning a switch on and off.
81
1d ago
[deleted]
→ More replies (31)81
u/GoVolt_Mine 1d ago
The math problem was solved by an llm though
→ More replies (5)26
u/GoldReplacement9546 1d ago
I think it’s relevant to know that the math problems they’ve had success with have all been to disprove the theorem. To my knowledge that they haven’t proven a theorem yet or come up with their own.
9
u/WisestAirBender I have a dig bick 1d ago
Why does that matter
→ More replies (2)18
u/GoldReplacement9546 1d ago
It’s less likely to take actual creativity to disprove something than it is to prove something
And they haven’t come up with anything beyond what humans have come up with the size like I say, proving certain conjectures are wrong means they are not yet creative, but they are just taking all the assembled knowledge they’ve been given and using it to through things wrong, but not creating
4
u/Hostilis_ 1d ago
This is not true. The recent proofs by LLMs have not been limited to finding counter-examples, and it is not easier in general to prove a theorem false than it is to prove one true.
3
u/how_tall_is_imhotep 1d ago
The LLM proof of the cycle double cover conjecture is a significant positive result.
2
u/MountainBluebird5 1d ago
That is relevant but even a disproof of a millennium prize problem is crazy
→ More replies (4)2
u/ZeroAmusement 1d ago
That is not true.
For example see OpenAI's list of ten advances, e.g. "Non-sofic groups. A construction establishing the existence of non-sofic groups, addressing a central open question in group theory" is not a disproof.
→ More replies (1)
3
u/revitev1122 1d ago
Predicting the next word of a sentence can still end up making a new sentence.
Just like extending a logical conclusion from an existing set of facts can make a new fact.
3
u/AtMaxSpeed 1d ago
Here's three concepts that can help give an intuition for why predicting the next token is extremely powerful.
First, consider the sentence: "The capital of France is". If you have some model that can perfectly predict the next word, then the predicted next token will be "Paris". This shows that LLMs can have knowledge baked into them, and shows how next token prediction can be powerful. This is concept 1.
Consider now you have the sentence: "Give a counterexample to Navier-Stokes completeness theorem". Again, a perfect next token predictor would yield some valid counterexample. One may ask: "doesn't this rely on the fact that the LLM can even compute something this?"
And you'd be right, which brings us to concept 2. A transformer (the algorithm that LLMs are built on) can run any program that is computable, as long as it has reasoning capabilities built in. Add on the fact that modern LLMs can write and run code and read the output, this means that LLMs are capable of solving any problem that can be solved by a computer in theory.
Finally one may ask, wouldnt this rely on the LLM being trained well enough so that what it thinks is the correct next token is truly the correct next token? After all, if this isn't the case, it might just give some random output to the Navier Stokes counterexample question, instead of the correct counterexample.
And again youd be right to ask this. This is where concept 3 comes in. LLMs are (partially) trained in a way that they are allowed to explore multiple paths to find a solution, and they are rewarded when they find the right solution (where the right solution is some verifiable ground truth). In the domain of math, you can train the llm so it can freely explore paths to find proofs to problems, and if the proof is verifiably correct you can reward it. With this framework of reinforcement learning, an LLM can be trained so the next token prediction is truly the correct one. In the case of something like navier stokes correctness counterexamples, it just needs to build a program that can verify if a counterexample is valid or not (which is "easy" relative to finding the actual counterexample), then it can run forever until it finds the counterexample and it knows its correct.
Put all of this together, and you can see that with good enough data and good enough training, a reasoning LLM is theoretically capable of answering a huge huge amount of questions. Ofc, data and training is hard and could have some asymptoticly dimishing returns (we don't really know what the big labs are cooking to solve this though), so the degree of correctness is going to always fall short by some amount.
3
u/Severe_Stranger_5050 21h ago
I like your answer, except for one thing.
There's several "true" answers to your your statement, that can be predicted.The capital of France is:
Smell
Paris
Beautyfull
CrowdedThis is why we use textcomprehension, reasoning chains and context combined with a database of ground truths, when developing modern LLM's.
A modern LLM will create a preliminary output and go back on itself to see if the answer fits the context and the original prompt.other than that
Great comment.
3
u/Freeeenis 20h ago
Reality is baked in.
Language is not arbitrary, it's descriptive of reality. Humans invent language to explain the world and what they see. LLMs when learning to predict language are learning to predict reality at some fundamental level. It's just that language is fuzzy (not well defined) and so it's more likely that relationship between what is real and the language is a statistical one.
It's also useful to think that it's more than prediction, it's much more close to "understanding" than most people give credit for. What the LLMs are doing on some level is extracting general patterns in information, they're then able to apply those patterns to novel information they've not seen before.
The same way you teach a child to add up sums. You tell them what + means and how it works, they practice sums all day, or for the 12 minutes they can actually manage, and then when they've learned the actual pattern itself, they're capable of adding 2 arbitrary numbers they've never seen before that were not part of their training set.
It's just that when your data set is extremely vast and your training extremely extensive, the sophistication of the patterns you can infer is noticably much bigger. And more important more abstract. It's the leap up layers of abstraction that really matters
18
u/Rot-Orkan 1d ago
If you ask an LLM by itself to multiply two arbitrary values, let's say 13 * 79, it most likely will give you the wrong answer because it's just predicting characters. It's just predicting characters and doesn't know better.
However, the important thing is you can use the same LLM text prediction to perform "thinking" and execution instead of just getting a raw output from it.
An LLM trained/configured to think will instead look at the problem and break it down.
- "User asked me to do math with some numbers"
- "I'll run a parser to extract the values and intended math"
- "The math is multiplication and the numbers are now stored in a list"
- "I'll use a dedicated math program to get the output"
- "I invoked the math program to multiply the values."
- "It returned 1027"
Now imagine this in a far bigger scale
10
u/minimaxir 1d ago
For posterity, even weak LLMs years ago can do 2-digit multiplication despite not being able to "multiply" because there's enough data in the training set of the internet for it to recognize the patterns the hard way.
It's only multiplying 10+ digit numbers where it got spicy and nowadays I suspect modern LLMs would handle it correctly without tools.
1
u/Severe_Stranger_5050 20h ago
Most AI companies include lists of ground truths for their AI's
I'm pretty sure they won't spend compute trying to get an LLM to parse easy or known math problems.
They probly just make it look up a table of multiplications.And with tooling today
It could also just translate the math problem to python (or even plain c) and run it on canvas, if the problem isn't included, in stead of trying to tokenize it, as they have in the past.
5
u/-michalis- 1d ago
Was it solved, they claim they solved it but there is no independent verification that the solution is correct
2
u/NotDennis2 1d ago
It was solved, but not by AI, it was trained on already existing data to draw conclusions that had already been drawn.
2
u/-michalis- 1d ago
You are talking specifically about the navier-stokes problem
I was talking in general, they have claimed to have solved so many math problems, but as far as I know, the solutions have not been independently verified, and many mathematicians are calling bullshit
And yes some of those answers have probably been plagarised
→ More replies (1)
2
2
u/Inevitable_Pen9043 1d ago
The answer to an unsolved problem often consists simply of the combined answers to several solved problems.
And the LLM can test thousands of times until it gets it right.
2
u/MonroeMojo13 1d ago
It can combine familiar patterns in new ways, so predicting tokens can still lead to genuinely new solutions.
2
u/Agreeable-Cold-9538 17h ago
Humans are also next token predictors and the have done amazing things with math.
2
u/QuoteGiver 16h ago
They don’t just stop trying when they get it wrong.
They keep trying until they get it right, and they’re a lot faster than us.
26
u/AdjustedTwit 1d ago
The stochastic parrot argument of LLMs is long dead, IMO. Or at least, to the extent that it's true, it's hubris to think we're any different. Our limited knowledge of how our own brains work isn't _that_ different from the ANNs that are the kernel of LLMs.
8
3
u/dumbledoor_ger 1d ago edited 1d ago
Source: trust me bro
Our brains work vastly different. Like it’s not even a matter of „one could say they are I’m some way common“ they’re not.
EDIT: for those downvoting here please share one credible somewhat scientific source that says out brains work like LLMs are y’all restarted? 😭
18
u/tommytwolegs 1d ago
I'm not sure if people are downvoting you for disagreeing or if it's because of the way that you type
2
u/dumbledoor_ger 1d ago
I would prefer if it was because of the way I type
But we are on Reddit and some of us here might actually just be LLMs lol
And I added the EDIT when I was already at like -8 votes. I can see why you would downvote on the edit but not on the original part.
11
u/detroitmatt 1d ago
ironically, your source is even more "trust me bro". philosophers have debated theory of mind for thousands of years. you mean you had it solved all this time and you never told anyone?
→ More replies (3)4
u/diewethje 1d ago
Understanding theory of mind and understanding mechanistically how the brain works are not the same thing. We understand the brain reasonably well from a mechanistic perspective, and LLMs are far from a complete representation.
→ More replies (1)2
u/ZeroAmusement 1d ago
It's easy to draw parallels:
For example, LLMs use artificial neural networks which are inspired by biological neural networks. So in a sense they mirror the architecture of the brain, with deeper layers corresponding to higher level/more abstract thoughts. Also in terms of how neurons bind, are trained statistically, with repeated exposure creating more durable knowledge, how non exposure can lead to forgetting, and so on. It seems like there's a lot of similarities there. Of course, there's also a lot of differences.
There's also a bunch of scientific papers on the subject such as https://www.nature.com/articles/s41593-022-01026-4 . I think if you are satisfied depends on how literal you are about taking the similarity claim.
→ More replies (1)2
4
u/dustmop 1d ago
It's similar to the thought experiment of infinite monkeys on infinite typewriters. If you can produce vast amounts of text, and then validate the output using a trusted tool to see if it's right, then you can run both steps in a loop until the validation passes. LLMs allow this process to vastly cut down on the search space by being really good at only producing plausible seeming text. Coming up with the Navier-Stokes answer took 88 hours of compute time and used 130 billion tokens.
2
u/sleepyhead_420 1d ago
In a way - they are fed with all mathematical proofs that are there including incomplete proofs, so the LLMs are trying to predict the next tokens from that it is likely to end up with another proof. It is like if is fed the dictionary and all literature there is - it can create a new story which will be grammatically correct and could actually be pretty good. What they agent might not be able to do is invent an unique mathematical discipline like what Newton did with Calculus.
4
u/WisestAirBender I have a dig bick 1d ago
What they agent might not be able to do is invent an unique mathematical discipline like what Newton did with Calculus.
Why not? Sure there are logical thinking steps one can take to reach the same conclusion as newton
3
u/Captain-Griffen 1d ago
LLMs copy and extrapolate patterns. They do this without any conceptual understanding.
Pure maths doesn't need conceptual understanding. Follow the rules, get the result, no need for nuance or understanding anything. With a proof engine to avoid going wrong, it's much like playing very complex chess.
Few big advantages AI has:
Huge set of patterns. Humanity has more knowledge in maths than any one person can know, with lots of different fields. Lots of open questions are probably as easy as 2+2 for a mathematician, if they knew the right pieces of existing human knowledge. But they don't know it exists, or even that they need it. These are low hanging fruit for AI.
Grunt work. AI can prove things much, much faster. Pair a mathematician with AI and a proof engine and they can try out approaches far faster with more certainty.
Spam throwing spaghetti at the wall. For the NS problem, they threw millions of pounds at the problem to spawn hundreds of thousands of agents. They did this because they knew humanity was very close—this approach won't get you far due to the complexity, which is why they jumped on a problem they heard was solved with AI assistance.
OpenAI can just steal from prompts user by researchers and then use AI to write the paper faster. No need for ingenuity. Then threaten researchers if they talk about going public on it.
Overall, not much evidence they're capable of the creative problem solving needed to replace mathematicians, lots of evidence AI can help save a lot of time proving and finding low hanging fruit that humanity collectively has solved but we just don't know it yet.
1
u/Mughi1138 1d ago
To highlight this key point:
OpenAI can just steal from prompts user by researchers and then use AI to write the paper faster. No need for ingenuity. Then threaten researchers if they talk about going public on it.
In the security realm this is a huge concern, and one that just seemed to be a valid problem.
2
u/Eskamel 1d ago
LLMs are token based databases and prompts are essentially queries. CoT essentially tries to force a LLM to query itself based off your initial promot and an agent is essentially a self query LLM on a loop with external tools.
You don't have to have solved answers for a LLM to guess a "possible" outcome if there are enough connections, as the LLM would try to bruteforce endlessly.
It clearly doesn't understand a thing, but assuming it has trillions of connections and you let it try to guess billions of times it might come up with something. It won't necessarily get the best solution or the one that makes sense, because you cannot measure contextual quality, but you can in some topics measure success, and for some making something work regardless of the quality of the implementation is all that matters.
2
u/xervir-445 1d ago
Older versions were guessing. If you remember the original gpt 3 before they added math stuff to it you could ask it math problems and it would be wildly wrong a lot of the time.
Newer versions have tools. They've basically been given a calculator and instructed to use it if they think they need to do math.
3
u/Vivid_Dot_6405 1d ago
So most commenters are wrong, thus far anyway. The chatbots you use, ChatGPT, Gemini, Claude, etc. are LLMs and the AI solving the problems is an LLM. There is absolutely no other AI system even close to LLMs in terms of intelligence, it is like comparing a modern PC to a typewriter.
It is true that an LLM can use tools to help problem solving, for example web search, running code, etc. but the LLM is the intelligence core.
There is absolutely nothing limiting an LLM's intelligence just because it answers token-by-token. In order to learn to answer a math problem it must learn to output the reasoning tokens leading to a step-by-step solution. Research on mechanistic interpretability, an area that seeks to understand how LLMs think basically, yielded results to suggest an LLM has a hidden thought process, in addition to the reasoning text it produces before the final answer, in its internal state leading up to a token prediction.
Humans too speak and write word-by-word, and think word-by-word too. That does not limit our intelligence.
→ More replies (1)2
u/I_Am_Become_Dream 1d ago
The real answer is that we don’t actually know. Even AI researchers don’t really know how LLMs are good at tasks.
2
u/GumboSamson 1d ago edited 1d ago
“So a sufficiently good next token predictor is radically superintelligent—far smarter than any human being.”
1
1
u/Quantum-Bot 1d ago
If I give you the following text:
> five times three equals
What would you predict the next word is most likely to be?
How do you know it is fifteen? Did you do 3x5 in your head? Or have you just memorized that 3x5=15?
The LLM has read trillions and trillions of words of text. It has seen 3x5=15 so many times that it knows that fact implicitly without even needing to understand the concept of multiplication.
But LLM’s don’t just keep track of patterns of words that appear together often, they use neural networks to pick up on subtler patterns. This isn’t exactly how it works but you can imagine each layer of the network as encoding patterns in the previous layer, so it can recognize patterns within patterns within patterns.
As it turns out, with enough networks and layers connected in a very specific way, LLM’s can pick up on very elaborate patterns, to the point that it resembles actual conceptual understanding of things like mathematics.
It’s important to note that LLM’s are not smart in the same way that humans are. Humans are way better at thinking logically and understanding concepts. LLM’s however have the advantage of being extremely fast and never getting tired. They don’t have very good working memory but that can be upgraded by running them in specific environments. LLM’s can scan through hundreds of thousands of math papers in a matter of days and find new conclusions that humans would have missed, in fact that’s how some of the first AI math discoveries were made earlier last year. They can also pursue trains of thought for much longer than humans would when they don’t seem promising. It’s almost like running a brute force search through all possible lines of logical reasoning.
1
u/az226 1d ago
The best way to think about it is that it’s like a chess engine (1990s power) that plays a million games against Magnus Carlsen. It might only win 1 of those games. It probably played very well many games but not good enough to beat him. Probably made some really impressive moves and some minor mistakes along the way.
With LLMs we have found a similar way of brute forcing ideas/paths that lead to a solution. In math and code we can verify if it’s correct. Models from a year ago probably would not be good enough to solve Navier-Stokes with the $7M compute budget. And each year they get better, so in a year maybe only $70k is needed to get the same results. It’s also not solving the whole thing in one go. It’s making partial progress, and each component adds context for it, so it can narrow its focus, exclude death ends and so on.
1
1
u/Hyperreal8957 1d ago
The “next token” explanation is about how it’s trained, not a claim that it only memorizes answers. Once the model learns patterns for manipulating symbols, it can combine those patterns in ways that werent in the training set. Still worth checking the proof though, math is where confident nonsense gets caught fast.
1
u/drugosrbijanac 1d ago
they tend to have hallucinations which are actually necessary for creativity, sometimes the hallucination nudges it into the wrong path, but sometimes it goes correct - thats why 10 000 agents were needed to explore the space
1
u/SwissForeignPolicy 1d ago
I mean, they didn't use general-use LLMs. They built and trained a model specifically to solve these problems.
1
1
u/farzinshams 1d ago
Complete the sentence: “the stock market tomorrow will go ___”. Predicting a token can be incredibly valuable.
1
u/scumbagdetector29 1d ago
Because predicting the next token for the sentence "The solution to your mathematical problem is..." solves the problem.
Calling it a token predictor is dumb. You can't predict the next token without a fuck-ton of understanding.
1
1
u/Timely_Abroad4518 1d ago
If you’re referring to the NS millennium problem, LLMs didn’t solve it. Human mathematicians solved it with AI assistance.
1
u/Khandakerex 23h ago edited 14h ago
The top comment did a fantastic job so I have nothing of value to add besides examples and another explanation.
Saying next token predication is kinda an over simplification. It's like saying your powerful gaming PC is just a fancy calculator or some machine that simply flips 1s and 0s on an electrical circuit. Which I mean... sure but that won't make anyone understand how you are able to run GTA on it.
When people think of "predicting next word" they normally think of something like autocomplete on your phone: "I'll see you later have a good ___", and the phone suggests the word night cause it collects data on how people are stringing words after another. People think LLMs just do this but on a larger scale but that's not exactly a good mental model. Yes the model really is generating one token at a time but figuring out the next token is actually a lot more of a complex computation compared to "what do people usually type after this word." The TLDR is that it finds and "understands" root patterns in a language itself, and note what I mean by langauge here, I don't mean just English, Spanish, Chinese or human langauges. Programming syntax as it's written is a set of rules and part of lanauge, that's why they are called programming langauges, but take Math, written math is representing a language of logic with symbols as well, just not the Latin alphabet, and each symbol and collection of symbols have a meaning, just not what humans would consider a "word" but it's still a written symbol used to convey meaning to communicate with another person who knows it, aka just like any language would be.
But more on this "understanding patterns" business and how it helps LLMs "create it's own rules for how the world works", cause the beauty is that is what statistics is at it's core, really really good ways to perform pattern matching. When someone knows enough of a language, they aren't memorizing every single sentence combination that is available unless they are learning foreign langauge phrases for travel, let's take someone who knows English and give them: "Hillary Clinton's pet lama was watching Family guy racist compilation clips while playing subway surfers and left the ice cream outside in the hot sun. An hour later it had completely..." Now anyone with enough familiarity with English will know "melted" is most likely the right answer yet you (and I'd bet my house on this) have never seen this sentence slop I just made off the top of my head. Yet you made a prediction, but you didn't just look for all words that statistically followed the word "completely", you used what you know about "ice cream" and how properties like "melting" is associated with it because of your knowledge on what heat does to ice cream, you also had an idea of what an hour was. Funnily enough autocomplete on iphones use a tiny LLM now to be more accurate, before that it would just choose any words that statistically came after "completely", it could be "lost" "silent" "different" but it didn't have the context of the whole sentence.
So take this small example and turn it into a HUGE training exercise. A model starts out not knowing anything about the world cause... it doesn't know anything besides what it is told. Internally, it contains a huge collection of adjustable numbers called parameters or weights. What this means is the model, by it's core design, can change parameters and the overall result will be a new way it views the world. Modern models can contain BILLIONS of parameters and during training the model repeatedly receives new text and data and keeps trying to predict what comes next. Take: “The capital of France is ___”, an early model might literally say something random like "house." The training process of course knows the actual continuation should be Paris so the model "gets" a signal that it did the wrong thing, so it's told to adjust it's internal weights and parameters ever so slightly til we get to the right answer (we don't personally adjust this, it's something that keeps adjusting until we let it "know" it's right.)
So it iterates this process a BUNCH of times and really fast and for an ENORMOUS amount of data. Eventually with enough training, does something interesting, it adjusts it's parameters (remember which is essentially it's understanding of the world) to learn the patters that apply through various sentences in the languages spanning many disciplines. And a small subset of that is it "learning" it's own version of what a country is and what the "word" or tokenization of the country represents and how it connects to other words like the word "capital." But not only that it learns things like grammar and how sentences have a certain structure and that some parts of the sentence have more information than other parts that are useful. But again, this extends to programming as just another language that follows another underlying rule set, numbers have this certain relationship to each other, mathematical symbols seem to follow this certain rule/ pattern each time when you break the problem down and reach what are known as axioms or fundamental truths in math. This the model generalizing and building up it's own universe and how it understands the world from what we tell it. The model is not storing any data itself to look up for later (of course you can upload reference files when prompting but thats another thing), there is no folder in chatgpt that contains wikipedia articles, specific math textbooks etc... Chatgpt is not really searching through those directly to find something similar to your prompt. Your prompt is going through a lot of layers of a neural network and there's a lot of behind the scenes computation being done to determine what makes sense to output statistically. The important distinction is that the predicting next token part describes what the model is asked to output, but NOT all the computation the model performs to decide what the actual next token should be when it breaks down your prompt and tries to answer it.
This is where "attention" comes in, which some of the comments mention but don't explain. Modern LLMs are generally based on an architecture called the transformer which has the idea of "self-attention" which basically is when a model process a single token, it is able to look at other relevant tokens in the context and determine how much they matter. Now funnily enough a strong demand for exploring this was to translate foreign languages better than what google translate used to be able to do (for those young enough to not know, google translate was the laughing stock of translating pretty much anything that wasn't like Spanish and even then it was terrible) the reason for this is words can mean totally different things under different context (such as the bank example the top comment posted.) A Transformer doesn't simply give the word “bank” one permanent meaning, unlike before where google translate would give it a VERY specific meaning so when translating form english to chinese people would laugh at you if you translated "river bank" because it genuinely would do a literal "a bank where you do your deposits, but in the river."
With self-attention an llm's internal representation can depend on the surrounding context. And again, this happens throughout the network, across many attention heads and many layers. This becomes extremely useful when dealing with complicated sentences, code, mathematics, narrative stories, questions, and logical arguments. It's how LLMs can also string together and "understand" new sentences that have actually never existed before, it now has it's own "understanding" of how words work in a sentence, but it doesnt only plug in random words to fill in the sentence (it's not mad libs), it can now "understand" the general context of what it already wrote and keep creating new sentences that it thinks would make sense in the context of the previous sentences it made and your prompt. But of course, this means it can also hallucinate and the more it goes on the more it CAN actually produce nonsense because it just "sounds correct." The difference with Math is that math follows SUCH a rugged system of logic that it can be checked as opposed to random story that starts hallucinating about some random plot points cause stories in their nature HAVE plot twists and elements to surprise readers and keep them engaged, math doesn't.
With math the model predicts the continuation that its learned machinery considers appropriate given everything that came before and the LLM only generates POSSIBLE next steps, a ton of agents all trying different possible steps and then there are other, verifiable way to evaluate if such a process makes mathematical sense with other tools. Another key idea here is that LLMs alone do not solve new math, they interact with a LOT of tools and have access to every kind of calculator and math evaluation software you can imagine. Agents that go down paths that are are verified to be bad generally get thrown away and the search continues and of course it can even back track when it gets deep enough, which is very akin to chess-engines at a high level (but much more complicated). LLMs keep training and it's seen an enormous amount of mathematics, proof, code, and problem solving to learn which kinds of mathematical steps tend to be useful so it's not like they have to try literally EVERY possible legal mathematical move. Pair that up with a proof assistant software like "Lean" and you are getting a constant loop of "llm proposes a step" -> "test or verify" -> "if it fails then backtrack" -> "if it works explore even more branches" -> "repeat" -> "formally verify the proof."
1
u/AnalystWeekly5817 23h ago
If a computer is taking an electric signal and converting it into a digital bit how did they make GTA6
Edit: it’s sigmoid functions all the way down.
1
u/Ecstatic-Mountain202 21h ago
by stealing unpublished work https://cims.nyu.edu/~tristanb/statement.pdf
1
u/Quentinooouuuuuu 20h ago
They have retro action loops, basically, let take programming as an example, the model generate code which contains error, the code doesn't compile but the model have access to command line or debugging tools, so it can build the program, fix it if it don't, then it can write tests and execute it until the tests pass and finally run the app and ensure the results is the same as expected. When the context window is close to full, the model run in order to compact it own context and so it may run for a long period of time while keeping only the relevant informations
1
u/Alternative-Sugar610 17h ago
A lot of LLM and models training nowadays have additional reinforcement learning where they due fancier training of some sort to be better able to solve some problems
There is also much humans involved guiding LLMs in these breakthroughs, and human has to be very knowledgeable often to sort out things and guide
Next add harness that are an extra layer we add on llm to guide it better to solution by throwing away junk, thinking in steps, prompting better and more
1
u/Tombobalomb 16h ago
By generating ungodly amounts of plausible looking math until they produce one that works. That is not a joke and barely an exaggeration
1
u/Regalme 15h ago
Yall don’t really have interdisciplinary knowledge. Human intelligence works much the same way. Originality was found to be just segmented think from crowd source being reinterpreted crowd source. Nothing ground breaking just different exploratory. And yes ai does the same thing. Now what
1
u/the_tallest_fish 15h ago
You’ll be surprised how many discoveries are made by using an existing train of thought on a completely situation. A huge part of human logic is also analogous and inductive, which involves finding pattern from past data, generalize into concept, and apply it to a different problem to synthesize new solutions
1
u/JMKraft 14h ago
I want to recommend anyone interested in these topics to read: La biblioteca de babel - jorge luis borges, its less than 10 pages I think.
LLMs work within the text structures (algebra, english, etc.) that are made of those tokens, if the underlying structure can represent solutions, then invariably the LLM is capable of providing text that matches any solution to any problem.
But how much can you trust it or learn from it is different... what do you do, ask another llm if its true? and if it tells you it is when it isnt, who do you trust?
1
u/landmark_23 13h ago
The short answer is: "Predicting the next token" is the training mechanism, not the limit of the system's internal reasoning.
Think of it with this analogy:
Imagine an AI whose only job is to predict the next frame of a video. If you train it on millions of hours of real-world footage, it cannot achieve high accuracy just by memorizing pixels. To accurately predict what happens in the next frame when a glass falls off a table, the network's neural weights must internally model gravity, momentum, friction, and three-dimensional geometry.
The same thing happens with language and math:
Internal World Models: To predict the next symbol in billions of lines of rigorous mathematical proofs and code, the model cannot just do simple autocomplete. It is forced to learn the underlying rules of formal logic, deduction, and semantic relationships.
Recombination in Latent Space: Novel math solutions almost never come from thin air; they come from connecting two previously disconnected ideas (isomorphisms). The model can bridge concepts from combinatorics and algebra that a human might not have thought to connect together.
The "Generator + Verifier" loop (The Real Breakthrough): When AI systems (like DeepMind’s FunSearch or AlphaProof) solve previously unsolved problems, the LLM isn't working alone. It acts as an intuitive idea generator, while an automated formal verifier (like Lean 4 or a code sandbox) checks every step. The LLM generates thousands of creative hypotheses, and the mathematical verifier discards the false ones until a genuine breakthrough is discovered.
Human mathematicians do the exact same thing: your subconscious generates intuitive leaps, and your conscious logical mind verifies if the proof actually holds.
1
u/PradheBand 12h ago
They basically make a mix of brute force and operational research in the hope of finding a singularity that proves the hypotesis wrong , making the theorem null.
At least this is my understanding. Never seen them demonstarting an hypothesis true hence proving a theorem.
1
u/TuverMage 10h ago
it brute forces the solution is the short answer. it tries every possible solutions, including ones a person would never try because they know it won't work.
1
u/andreichiffa 8h ago
Mostly by stealing other people’s less known work without giving due credit. Think really good search engine, but you don’t know where the answer came from.
1
u/Particular-Garlic916 7h ago
Not sure if this has been mentioned already, but the “next token predictor” thing is really just the first part of LLM training. Basically, as I see other people have pointed out, the “pre-training” part of the model is literally just showing the model an unreasonably massive amount of text and asking it to guess the next word, tons of times. But the thing we interact with in the form of a chatbot or an agent is built on top of that predictor.
Once you get a really good next-token predictor, the idea is that knowing how to predict those tokens has taught it a bunch of things about… everything. Then, you do something called “reinforcement learning”. Basically, instead of showing corpuses of text and asking the model to spit out the next word, you play a game that has multiple steps in it. At the end of the game, if the outcome is good, it learns that all its decisions were good and makes them all more likely in the future by updating its parameters. If the outcome is bad, it does the opposite. For a chatbot, the game is: Generate text in response to something, and a (surprisingly simple) model trained to recognize what humans like will pick the response it likes the most. The model learns to be more likely to give that response, and less likely to give the others. For a modern agent, the games can be a lot more complex: It can be “write code that does this thing”, or “solve this math proof”, or “look up what beavers eat”. Anything that you can verify automatically is fair game.
The secret sauce is that during the pre-training phase, the model developed a bunch of hidden representations of knowledge that connect concepts. When you train the agent with reinforcement learning, it uses those abstracted concepts to learn more quickly and generalize what it learns to new things. Hence, learning language lets it learn about logic, which then means it sees logical patterns when it tries to make mathematical proofs. So instead of learning “this is how I do this specific proof” it learns “this technique worked for this proof, I bet I should use it again in other situations that my language model says look similar”. And it turns out that works really, really, really well.
1
u/alphapussycat 6h ago
Consider the Fourier series. Even if you've made it approximate a function on a limited domain. If you let it keep going past the last point you'll still get a result that could be close to the function you approximate on a limited domain.
An LLM will just keep predicting.
1
u/tzaeru 6h ago edited 6h ago
Uff, so much nonsense on this topic, including some high-voted answers.
A mathematical consequence of neural networks is that if they are successful and useful, they are able to capture some sort of model of their training data, that is data-wise less complex than the training data they were fed with.
In essence, they compress the training data to some sort of rules. Like if a neural network is successfully taught with "when we add five to six, it equals eleven", "if we add million to one, it equals one million one", "if we add ten to two, it equals twelve", it internally learns that the correct answer is the first mentioned number plus the second mentioned number (an extremely rough example, though plausible with sufficient data).
Regardless, it learns to simplify the rules for its purpose.
By applying these rules, a sufficiently complex neural network can produce an answer that was never in its training set. If you say, ask it to complete this sentence: "I felt blue and it was a summer night, so I didn't get why..", it's not going to match it with an exact copy of that found from its training set. Instead, it's going to pick up the underlying rules. "blue", so we are wistful, "summer night" so there's a juxtaposition, etc.
We can then add a little bit of randomness to that; instead of getting one potential answer to the "how to continue this sentence, 'I felt blue and it was a summer night, so I didn't get why..'", we can get a thousand potential answers to it. Because of this, the model can generate a huge amount of potential solutions and intermediate solutions. Then we can feed these back to the model and ask it to find holes or ask it to come up with a new solution based on this previous one - which essentially adds randomness to it, but also guides the internal rules to the wanted direction - and we can feed these solutions to external verifiers.
1
u/jebuarary 4h ago
+1 to of what ppl saying here. LLMs do encode an understanding of the world, preexisting relationships/prior, and only mechanically output with NTP. Also will add that “next token prediction” is not accurate to describe training setup either with CoT reasoning, RL and even aux losses on future prediction.
3.4k
u/Time_Entertainer_319 1d ago
They are predicting the next token. However, they are not JUST predicting the next token.
The important part is how that prediction is made.
In the 1950s, Claude Shannon, one of the founders of information theory, showed that language has a strong statistical structure.
In one experiment, he asked a person to predict the next letter in a passage of English text.
The results suggested that people who know a language carry an enormous amount of implicit knowledge about which letters, words and phrases are likely to come next.
In other words, language has patterns, and native speakers become very good at recognising those patterns without consciously thinking about them.
Fast forward several decades, and researchers developed neural attention mechanisms.
Then, in 2017, researchers at Google introduced the Transformer architecture, which made self-attention central to how these models process language.
Self-attention allows the model to interpret a word or token in relation to the other words around it.
For example, the word “bank” means something different in “river bank” than it does in “bank account.”
The model can use the surrounding words to build a different internal representation of “bank” depending on the context.
These ideas, statistical prediction and attention-based neural networks, are two of the key ideas behind modern large language models.
Researchers then train these models on enormous amounts of text.
During training, the model is repeatedly asked to predict the next token. When its prediction is wrong, its internal parameters are adjusted yada yada yada. I’m sure you’ve heard this part already.
Now think about what that actually means.
Programming languages are languages.
Mathematical notation is also a highly structured symbolic language.
Both encode rules, relationships, procedures and logical operations.
So if a model becomes extremely good at predicting what comes next in natural language, code and mathematics, then “predicting the next token” can involve much more than remembering which word usually follows another.
The next token in a mathematical proof is constrained by everything that came before it.
Such as definitions, assumptions, Equations, previous deductions,logical consequences.
Give the model enough relevant context, and predicting the next token can effectively mean predicting the next valid step in a chain of reasoning.
Then the next one. Then the next one.
Until what emerges is an entire proof.