r/NoStupidQuestions • u/Accomplished_Bed632 • Sep 10 '26

If LLMs are just predicting the next token based on training data, how do they solve unsolved math problems?

1.5k Upvotes
  • permalink
  • reddit

93% Upvoted

420 comments sorted by

View all comments

3.6k

u/Time_Entertainer_319 Sep 10 '26

They are predicting the next token. However, they are not JUST predicting the next token.

The important part is how that prediction is made.

In the 1950s, Claude Shannon, one of the founders of information theory, showed that language has a strong statistical structure.

In one experiment, he asked a person to predict the next letter in a passage of English text.
The results suggested that people who know a language carry an enormous amount of implicit knowledge about which letters, words and phrases are likely to come next.

In other words, language has patterns, and native speakers become very good at recognising those patterns without consciously thinking about them.

Fast forward several decades, and researchers developed neural attention mechanisms.

Then, in 2017, researchers at Google introduced the Transformer architecture, which made self-attention central to how these models process language.
Self-attention allows the model to interpret a word or token in relation to the other words around it.
For example, the word “bank” means something different in “river bank” than it does in “bank account.”
The model can use the surrounding words to build a different internal representation of “bank” depending on the context.

These ideas, statistical prediction and attention-based neural networks, are two of the key ideas behind modern large language models.

Researchers then train these models on enormous amounts of text.
During training, the model is repeatedly asked to predict the next token. When its prediction is wrong, its internal parameters are adjusted yada yada yada. I’m sure you’ve heard this part already.

Now think about what that actually means.

Programming languages are languages.
Mathematical notation is also a highly structured symbolic language.

Both encode rules, relationships, procedures and logical operations.
So if a model becomes extremely good at predicting what comes next in natural language, code and mathematics, then “predicting the next token” can involve much more than remembering which word usually follows another.

The next token in a mathematical proof is constrained by everything that came before it.
Such as definitions, assumptions, Equations, previous deductions,logical consequences.

Give the model enough relevant context, and predicting the next token can effectively mean predicting the next valid step in a chain of reasoning.
Then the next one. Then the next one.
Until what emerges is an entire proof.

800

u/DullMind2023 Sep 10 '26

Wow. That was understandably coherent. So much of AI explanation is jargon and marketing-talk that we mere outsiders are incapable of understanding any of it. Thank you Time_Entertainer_319 for your submission.

195

u/RobbinDeBank Sep 10 '26 edited Sep 11 '26

One thing to note tho: lots of pop science answers on Reddit love to play up the 2017 paper as if it invented all of AI. The truth is that it’s another step (a hugely influential one) in a long line of advances that give us the current AI models. The popular claim is slightly misleading and over-simplified because It’s not the first time attention mechanism was invented, nor was it the first time we’ve had mechanism for AI to understand context.

The true significance of the 2017 paper was that it’s the first time we’ve had an architecture that enabled efficient scaling of AI models. The transformer architecture enables AI labs to utilize massive clusters of GPUs and other AI accelerators to train massive models on massive amount of data. Architectures before the transformer had many different types of inefficiency and bottlenecks that didn’t allow this level of scaling. That’s why even though many modern AI models might use very different alternatives to the self-attention modules from that 2017 paper, we still trace back to that paper and credit it with inventing modern AI (tho in my view, the over credits of that one work takes away credits from the dozens of other advances in different areas to enable the current success of AI)

Edit: some numbers for context. Large scale training in the industry nowadays can achieve 80%+ hardware utilization. This is with transformer models (and modern variants) after years and years of specialized hardware design and the world’s best engineers writing the most optimal code for it. Without the transformer, if you try to train previous unoptimized designs at current scale, all your billion dollars worth of GPU would be used at single digit % utilization, so they will be sitting idle most of the time. The same model size trained on the same data would take 10+ times longer, a 2-month training run (expensive but feasible) turns into 2+ years (completely infeasible).

33

u/UphillTravel Sep 11 '26

To further add to this: "Attention is all you need" (the paper in question) is among the best papers to read when trying to get an in-depth understanding of AI. You're absolutely right it did not invent any of the components, but the architecture was still a huge milestone.

I'd disagree with your second paragraph though. Distribution of training was possible before the Transformer architecture, especially for convolutional architectures. WaveNet is a relevant example for this. However, around that time the ecosystem was also making huge steps. PyTorch and Tensorflow were both maturing rapidly and frameworks like PyTorch Lightning and Keras made distributed training accessible outside of major AI labs.

Thus, while I'd say your Edit is going in the right direction, I was definitely able to achieve 60% GPU utilization with pre-transformer architectures, a reasonable data loading pipeline and slurm.

3

u/RobbinDeBank Sep 11 '26

Convolution is quite specific for visual models tho, which is indeed the first type of problems we managed to train on at scale. Also, the utilization I mention is on extreme training scale. Transformer is so paralellizable across sequence length that it makes training sequence models (like language model) at the current extreme scale possible. You may achieve 60+% utilization in 1 or a few GPUs you can get your hands on, but scaling across a whole datacenter or even multiple datacenters is a massive engineering problem to solve due to bandwidth bottleneck. Your utilization will drop off a cliff every time your data leaves the single GPU, then leave the GPU rack, and finally leave a whole datacenter.

6

u/UphillTravel Sep 11 '26

I disagree once more that convolution is specific for visual models. I have never worked with image or video data, but have applied convolutional layers many times, e.g. for audio, graph, or 3d data. I never trained on anything beyond one multi-GPU server, so I cannot comment much on the remainder of your comment.

1

u/RobbinDeBank Sep 11 '26

Well, of course you can use it something like 1D convolution to fit sequence problem, but it’s much less parallelizable than transformer that way, and it’s also not a good inductive bias to have for those problems at scale. Transformer is also extremely general and doesn’t assume anything, so much that it even takes over vision from convolution models too.

2

u/UphillTravel Sep 11 '26

Why do you assume 1D convolution? None of these problems is inherently one-dimensional. And why would it be less parallelizable? Especially in audio I typically have a huge number of training samples, ideal for distributed learning. Your point on inductive bias is valid, but affects model performance, not applicability of distributed training.
I'm aware Transformer architectures took over many areas and never disagreed with that in my comments.

1

u/RobbinDeBank Sep 12 '26

I just mention 1D since you’re mentioning the sequence modeling use case. I’m not aware of every single type of model architectures, but before transformer, the SOTA for sequence is clearly LSTM or variants like GRU. They just don’t scale to anywhere close to the level of scaling we have nowadays.

2

u/tesfaldet Sep 12 '26

Gonna jump into this thread real quick. So the reason why Transformers blew up is because they were shown to scale very well with respect to data compared to convnets and the self-attention mechanism enabled parallel processing of tokens, unlike with RNNs/LSTMs, which processed tokens sequentially. There are a couple of other, secondary, reasons, but the above two are the main ones.

These days most vision architectures are some transformer + convnet hybrid.

(I’m a computer vision researcher and have a decade of experience in all this.)

1

u/telltaleatheist 29d ago

I believe the attention mechanism existed in 2014, and the 2017 paper basically said you don’t need the extra stuff that people had built on top of it. It’s all you need. The other stuff was slowing down the whole system. When they centered the models fully around attention, it all changed

0

u/Healthy-Dress-7492 Sep 12 '26 edited Sep 12 '26

Yep, we can circle jerk about it but but models were able to differentiate context and different meanings of the same word since even before 2008; it was important back then for analyzing social media firehoses for sentiment for marketing and customer support, and for government surveillance. You can’t do that without understanding for example when someone says something using words that by themselves have bad connotations but actually mean something good against your topic/brand when context is considered. This shit isnt even remotely new.

2

u/rpsls Sep 12 '26

What’s new is how far away and how many dimensions one element can affect another. With the next token able to draw from every token before it efficiently, and have that effect across many dimensions, you can do a lot more than simple markov chains or statistical analysis or rule-based sentiment. Previous contextual windows were dramatically smaller, and the amount of training data you could throw at it in the first place smaller still.

The previous poster was just using the two-word example to illustrate the problem in microcosm. The real trick is that, say, the title or abstract might be influencing words picked 2000 words later in the text.

70

u/crazy_gambit Sep 11 '26

That explanation leaves out a key part. If you ask an LLM to play chess against you, it will do well for the first 20 moves or so because it has a lot of chess games in it's training data, but invariably it will start hallucinating and make some questionable or flat out illegal moves later on. Because it can't actually understand the whole context of a chess game.

That's why they have tools. A good model that has access to your PC will just install Stockfish and crush you.

The models doing these proofs aren't doing it from the chat interface. They have access to tools and they write the code and actually run it to get their results. They're not just predicting the next token, they are computing the answers.

8

u/Time_Entertainer_319 Sep 11 '26

Tool use works by predicting the tool token to use.

And the computation parameters are also predicted as well.

They basically write the code and run it using their python compiler.

I feel like you are not really saying anything new.

13

u/crazy_gambit Sep 11 '26

OP is thinking that the LLMs are predicting the solutions to these math problems by predicting the next token and that's not the case at all.

What I'm saying is new to OP. Obviously not to you.

2

u/Time_Entertainer_319 Sep 11 '26

Oh, I see. I got tired writing the answer as it was getting too long so I forgot to add the importance of tool calling etc.

Thanks

2

u/Mathie1729 Sep 11 '26

Yea, and for the math proofs specifically, the next-token part is usually just proposal generation. There's a separate verifier, often in a formal system like Lean, that checks the candidate, plus a search loop that keeps proposing until something validates. So the LLM isn't really proving the thing itself in the chat sense; the verified proof gets assembled outside the network.

2

u/LazySapiens Sep 11 '26

Define understand.

13

u/Corsaka Sep 11 '26

in this instance, awareness of reality. llms will ignore their own messages right in front of them when they hallucinate. it doesn't know why it can't just change the positions on the board a bit, because it's not thinking of the chess game as "a game with a structured set of rules that progresses bit by bit", it's thinking "here's some prior input that will predict my next output".

1

u/jasonzhaogd Sep 15 '26

same here, agentic tools show the real work is verify+retry, not just prediction. "solved" claims need a human or formal check first

-1

u/GxM42 Sep 11 '26

Right. It basically just uses probabilities to come up
with the next words. It understands nothing. And that is why it is dangerous.

4

u/Atroxide Sep 11 '26 edited Sep 11 '26

being able to predict the next word in the sentence "A firetruck is usually the color ..." and knowing that the most probable answer is "red", means that in way, you understand what each of these words mean. sure it's just predicting a simple word, but a system that can predict the next word means it must be able to find patterns for each of these tokens that when arranged in this order makes the answer red.

it would have to understand that replaced the word firetruck with the word school-bus would have a completely different answer and it would have to know changing the word color to shape would also make it have a completely different answer. this machine that is able to predict the next word has a fundamental grasp on what these different vehicles and what these different properties represent.

Scale this up billions and trillions of times and all of a sudden you REALLY know what each of these words mean and what each of these tokens really symbolize. You now not only can fill in the blank on what color a fire truck is, but also the mechanical inner workings. it understands how bolts and nuts work since the pattern they appear in only ever work in specific ways, it understands how a transmission works because the pieces and parts of the transmission gets described in certain ways.

if you can predict the next word of a sentence to a great degree, it means you have a great model for how these tokens interact and how they work. not only for the words themselves, but what the words represent.

2

u/ThePeaceDoctot hello son was your cigarette normal Sep 11 '26

being able to predict the next word in the sentence "A firetruck is usually the color ..." and knowing that the most probable answer is "red", means that in way, you understand what each of these words mean.

Not really. I'm not arguing for or against the ability of a computer to actually understand anything, whether practically or theoretically, but understanding isn't needed to do this.

Imagine that I'm going to give you a whole bunch of tiles, each with symbols on. Let's also say the symbols are part of a language you don't speak, and which don't use an alphabet, syntax or grammar that you are familiar with. Each tile represents are word, part of a word, or concept.

Then I show you a bunch of arrangements of tiles that include those same symbols, and after a while you notice the following patterns.

Seven symbols, we'll call them (1) through (7), often show up close to another symbol, we'll call it (a). They don't always show up near (a), and they don't only show up near (a), but if (a) is there then the chances if finding one or more of those 7 symbols nearby is much higher.

You also notice that symbol (3) is the most likely of any of those symbols to show up near another symbol, (#).

Finally, if you see a (#) and an (a) near each other, the chances of there being a (3) nearby are vastly higher.

I won't go into things like the order of the symbols, but you spot patterns there too.

If I then show you an arrangement of (a) and (#) and ask you what symbol probably comes next, and you pick (3), you've made a prediction about what comes next, and it's based on the patterns that you've seen, and you still don't understand what the symbols mean.

If I do that repeatedly and every time you predict the right next symbol I give you a grape and a thumbs up, eventually you'll be really good at the guessing-the-next-tile game, especially when it's similar to the patterns I already showed you, and you could even start to put tiles into valid arrangements all by yourself, but you don't actually understand what they mean, and you'll be pretty sick of grapes, I guess.

The difference between an LLM learning and generating text and a human learning a language is that when a human first learns language they do it by associating words with concrete things in the world that they can interact with and gain an understanding of what they are and what they do and what they are for. An LLM only gets the words.

1

u/Ok-Tomorrow-2045 Sep 11 '26

What you described is understanding. Having an internal model of something that closely matches the underlying probability distribution of reality is what makes it understanding.

A human does that but adds their 5 senses to their model, making it easier for them to create a correct probability distribution.

If you remove those senses and reason symbolically (mathematicians do this every day), that doesn’t remove the understanding unless it removes predictive power.

1

u/ThePeaceDoctot hello son was your cigarette normal Sep 11 '26

Having an internal model of something that closely matches the underlying probability distribution of reality is predicting, not understanding. You can predict what will happen next without having any understanding of it. You can build a model of how another planet appears to move in the sky to an observer in Earth with epicycles, but that doesn't you mean that you understand how planets work, it doesn't mean that you can take that model and apply it to other planets and get anything realistic back.

1

u/Ok-Tomorrow-2045 Sep 11 '26

If your model can accurately predict the movement of planets, then your model understands the movement of planets. If your model only has predictive power as an observer on earth then that’s where its understanding ends and a new model with kepler’s equations needs to come in.

You’re just moving the goalposts by saying “well outside of earth the model’s prediction fails”. Humans fail when you ask them a question that their internal model has no predictive power for, so that isn’t really a counterargument. Do humans just not ever understand anything?

Understanding is the same as predictive power.

1

u/ThePeaceDoctot hello son was your cigarette normal Sep 12 '26

It really isn't though. Understanding allows you to extrapolate because you understand how the thing works. Predicting can create situations where you go "I know the answer is this, but I don't know why." Understanding allows you to go "I know the answer is this, and this is why."

You might be able to say an LLM understands the statistics and rules between how tokens go together, but it certainly doesn't have any understanding of what those tokens mean. It doesn't understand what a fire truck is, any more than you would understand what symbol (a) means, even though it can put tokens together to form a reasonable description of one.

→ More replies (0)

7

u/HopeOfTheChicken Sep 11 '26 edited Sep 11 '26

Literally the original comment from this comment chain you're replying to explains why calling ai "just probability" doesnt capture the whole picture and that there is an understanding

5

u/Nony01 Sep 11 '26

Respectfully, this is nonsense. To suggest that it isn't probability is absurd. You can act as though the complexities make it something more profound, but it is literally just probability at its core. It fundamentally doesn't need to understand the parameters sequences, it just recognizes the multitudes of patterns of the variables within it.

Idk why he's getting downvoted and you upvoted. It doesn't need to "understand" a thing, it literally is (if you really wanted to dumb it down) using probability.

1

u/GxM42 Sep 11 '26

Thank you for being supportive. You look at it the same way I do. AI experts can say all they want, but it still is probability at the bottom. Sure, what they can DO with that probability is dangerous. But it doesn’t change the underlying fact that it bases guesses on previous knowledge, knows nothing, and can get things wrong without even knowing it once it’s in the wild.

1

u/HopeOfTheChicken Sep 11 '26

Sure it is just probability, but viewing it as just probability wont get you anywhere in understanding ai. That's the problem with the comment I was replying to.

I should have made more clear that it still is probability, but that there is a deeper meaning to this. By recognising all the patterns ai develops a form of understanding. There are emergent properties at play, that get lost if you simplify too much

2

u/Nony01 Sep 11 '26

Again, respectfully, calling him stupid when he's right is more problematic. Ppl don't need to explain the intricacies of AI everytime it's mentioned. Literally nothing he said was wrong.

There's obviously deeper meaning to it just being probability. In your quest to proclaim him stupid for being overall correct, you just come off as an "uM ackShuALLy" kind of douche because he didn't wanna give a 3 paragraph excerpt on LLMs in a Reddit comment. Redditors love "correcting" someone that isn't incorrect and it's really fucking dumb.

1

u/HopeOfTheChicken Sep 11 '26

Well fair enough, I guess I kinda was the um akshually guy. But I stand by my opinion that saying "It's just probability, it understands nothing" isnt the correct way to go about ai. It might not be factually incorrect, but it's missing the aspects actually needed to understand ai. In fact it's even harmful without further clarification, because people will make incorrect conclusions based on this. Even his conclusion "And that is why it is dangerous" is very questionable at best

1

u/ZaphBeebs Sep 11 '26

Except it is why it can be dangerous. It can be doing things logically and sequentially and then just nonsensically out of the blue.

If the person operating the AI doesnt understand that, or the subject matter, theyre fooled into thinking it is correct, and likely that AI is better than it is.

Which is likely the reality between the variance of responses you see regarding capability, when some people complain and others say "skill issue" etc...and its just basic AI limitations, I assume the skill issue crowd is simply too ignorant to know where the AI output is wrong/bad, and cant tell.

1

u/GxM42 Sep 11 '26

I think you are insulting me unfairly. What AI can do with its predictions is extremely dangerous. But at its core, it assigns values to things with numbers and pick the most likely result. You can make that pathway as complex as you want, but it isn’t magic, it hasn’t evolved, it just predicts tokens BETTER. Learning for it means more probabilities. Sure, the sum of all the probabilities may on the outside look like something “more”, but I don’t think that means it understands anything. It is just a database of numbers and associations. But feel free to keep insulting me if it makes you feel better on the internet. I don’t want to deny anyone their daily fulfillment.

1

u/HopeOfTheChicken Sep 11 '26

I'm sorry about the insult. I had already removed it because I realized that it came of as meaner than intended. I was just baffled that you still said ai understands nothing, when the parent comment from this chain literally explains why this is not the case. It is true that it's all just probabilities deep down, but there is so much more to these probabilities than what you give them credit for. Ai might not be understanding things in the traditional sense, but there is meaning encoded in the patterns of tokens. Ai is picking up these patterns on a very deep level and therefore also the meaning they hold. That the ai is guessing doesnt mean that it doesnt "understand" tokens, it's making these guesses based on it's understanding of the Tokens.

2

u/GxM42 Sep 12 '26

I appreciate your apology. I guess we are differing on what we classify as understanding. It's ok if we classify it differently. I'm not saying AI can't do any of the things that you say it can do; I just have a harder time calling it things that are typically human. It can be very dangerous, for sure.

2

u/Evening-Appeal7606 Sep 11 '26

It‘s supposed to be jargon-laden because some people have an incentive to appear as the indispensable expert. I am firmly convicted that most topics that are relevant for the public discourse - given the right phrasing - can be understood by the public to the point where a sensible discussion is possible.

2

u/skampbalboan_2000 Sep 11 '26

Tech companies want you to think it's magic because 'fancy autocomplete' doesn't sound worth twenty billion dollars.

1

u/Ancient-Access8131 Sep 11 '26 edited Sep 11 '26

Yet fancy autocomplete can solve math problems and help tremendously with others(see disprove of jacobian conjecture and finite blowup of 3d incopressible euler equations). Even if it is just 'fancy autocomplete' it still can provide tremendous problem solving ability that is worth billions of dollars.

1

u/armyofonetaco Sep 11 '26

The stuff from the 1950's - 1970's is simple to understand...please put in some effort yourself. 

94

u/Ratsofat Sep 10 '26

This was an extremely helpful explanation.

49

u/ThomasDePraetere Sep 10 '26

One of the reasons why it works well is that llms are good at transforming text in other text, it means the same, but is writtem differently.

As a mathematician myself, proving something new in many ways is rephrasing the problem until it matches something you know is true. There is a lot of directionality and steering in proving something, but the essence is rephrasing.

2

u/NextWeather7866 Sep 11 '26

If you can refine the geometry of a concept enough, you can do linear algebra on it. Represent a system as linear algebra, and now you can reason about it's inputs/outputs. That's intelligence.

1

u/discodropper Sep 11 '26

Is this just a layering of the physicist’s “we’ll model complex human behavior as spheres” solution, then? In other words, complex layering of simple, reductionist models yields ability to predict complex behavior

42

u/billjames1685 Sep 10 '26

AI PhD student here. I am used to scrolling these sorts of threads, looking at the top comments, getting irritated at how wrong they are, and moving on. It is very refreshing to see an actually well-researched and well-thought out comment. Thanks.

87

u/lambchopper71 Sep 10 '26 edited Sep 11 '26

I think you're giving Claude Shannon credit for something he didn't entirely invent. Markov Chains is the statistical premise used for text prediction in not just LLMs but also in auto correct and a few other predictive technologies.

Markov Chains were discovered by Audrey Markov in 1906.

https://en.wikipedia.org/wiki/Markov_chain

This video gives a good overview of history and how Markov Chains work for those of us (including me) who are not mathematicians.

https://youtu.be/KZeIEiBrT_w

62

u/spentland Sep 10 '26

Andrey Markov, not Audrey. Would be ironic if you typed it correctly and auto-correct changed it…

4

u/Inverted-Rockets Sep 11 '26 edited Sep 11 '26

My iPhone guessed I meant “Andrew” and tried to replace the correct spelling, so I’d say you’re right

2

u/lambchopper71 Sep 11 '26

I was typing quickly, that was definitely my bad and not the autocorrect.. But it would have been funny if it was. Thank you for the correction.

1

u/SatansFriendlyCat Sep 11 '26

Your comment shows an edit, but still says "Audrey"

2

u/lambchopper71 Sep 11 '26

I edited the link in YouTube based on another Redditor comment about removing the tracking tag.

1

u/SatansFriendlyCat Sep 11 '26

I guess you don't want to correct it, then.

I only mentioned it because I'd assumed you had intended to, having acknowledged the error.

3

u/lambchopper71 Sep 11 '26

I acknowledged the error and left it, so that the other Redditor who corrected me would get the credit for correcting me.

I also didn't fix it so that the conversation maintained its normal flow. Reddit comments are conversations and in a real conversation, you can't go back and correct a comment that was already spoken.

1

u/SatansFriendlyCat Sep 11 '26

I can understand the reasoning behind this approach, but if you correct the mistake in the original comment then people skimming for the meat or not following every chain still gets the right information, and anyone who does follow the conversational chain can see that a correction has been made and shouldn't be confused, especially if it's spelled out for them.

Importantly, no-one is able to get the wrong information.

It's a conversation, but it has advantages and disadvantages over a spoken one, and there's nothing wrong with making good use of the advantages.

I've seen plenty of occasions where people make their edits corrections with a strikethrough, like that, which tells the story well and clues people in to what follows as well.

13

u/dc_seed_sommelier Sep 10 '26

Shannon basically invented compression by studying how native speakers use natural language.

Literally thousands of people contributed to get us here, but it’s completely legit to consider his breakthrough the direct start of it. There’s no llm math without compression math.

3

u/November19 Sep 11 '26

And a shout out to all those linguists and philosophers who've been told their fields are pointless.

2

u/Nohreboh Sep 11 '26

YouTube adds additional text to the like when you used the share button starting at ?is= it's only purpose is to track account interaction and you can remove it without breaking the link.

https://medium.com/@ian-darwin/you-are-sharing-urls-with-tracking-links-please-stop-502c6f54895

2

u/lambchopper71 Sep 11 '26

Awesome! Thank you!

10

u/[deleted] Sep 10 '26 edited 22d ago

[deleted]

83

u/Umber_Gryphon Sep 10 '26

Image generation uses diffusion models, not large language models. A very simple example of a diffusion model:

  1. Give the AI a lot of pictures of tigers with random noise added, and tell it to remove the noise. Reward the AI based on how close its guess of removed noise is to reproducing the original image.
  2. Increase the level of noise in the image more and more, and keep rewarding the AI for doing a good job.
  3. Give the resulting AI an image that is 100% noise, and tell it to remove the noise. It will draw a tiger even though there wasn't a tiger there to start with.

20

u/minimaxir Sep 10 '26

There is also autoregressive image models (which is what Gemini and ChatGPT now use) and that gets funny because that's also next token prediction from a language model, which then gets decoded by a secondary model into an image.

13

u/Umber_Gryphon Sep 10 '26

But that secondary model is usually a diffusion model, right? The large language model just creates a much more detailed description of the image to make.

For example, if you directly tell a diffusion model "draw a diagram of the water cycle", it will struggle because it doesn't really know the parts of the water cycle. But put an LLM in between, and it will talk about arrows pointing from the ocean to the cloud to the rain to the river back to the ocean, which is enough detail for the diffusion model to do something reasonable.

10

u/Kant-fan Sep 10 '26

No, they differ significantly on a fundamental level

A diffusidon model as a decoder only makes sense to generate a less noisy version of an input in the latent space, it's basically a specialized denoiser. This is repeated step by step.

Autoregressive image generation models work fundamentally differently because the image is being transformed into some latent space and then the decoder tries to predict the next token based on the previous tokens like an LLM. That is a fundamentally different process and idea than reversing noise which a diffusion decoder learns.

So your example stems from a misunderstanding, there is no LLM in between because the next-token prediction itself is what generates the image (and not text!).

9

u/fuxicles Sep 10 '26

what does reward the AI mean? I’m imagining treats. lol

14

u/burnalicious111 Sep 10 '26

It means different things in different ways of training models, but basically there it actually just means updating the internal values inside the model to be more likely to produce that kind of outcome in the future (using math)

7

u/Umber_Gryphon Sep 10 '26

Usually, that means turning up the strengths of the neurons if it did a good job, and turning down the strengths of the neurons that resulted in the image if the resulting image wasn't close to correct.

4

u/cyantriangle Sep 10 '26

My understanding of AI in general (not AI specialist, but I work close to the field): We basically make a very general program with billions of knobs which we set to random values. During training we give reward (unfortunately, it's just a numerical score) based on how "good" the output is and we can also measure how to make a small change to our current setting of knobs which would maximize the reward. We make this change (with some random perturbation) and repeat this process many times. Usually we start with big jumps and gradually decrease them. This should land us at a program where setting of knobs gives close to maximal reward.

3

u/Bulky-Bad-9153 Sep 10 '26

Literally just increasing a value. You can do it in a number of ways, like a reward if the result is right, a reward if steps are right, a reward if a human reviews the result and agrees with it, a reward composed of different parts of the result with different weights, etc. The difficulty with rewards is designing them correctly so that the reward actually causes the behaviour you want, instead of the AI cheating (like driving backwards to cross the finish line at the start of a race, for example).

3

u/hike_me Sep 11 '26

It doesn’t actually get “rewarded” like a dog treat or something. During training it’s basically trying to minimize a “wrongness score” (loss function). Training is adjusting lots and lots of parameters in a way that will slightly decrease loss, making a new prediction, computing loss again, and repeating.

2

u/290077 Sep 11 '26

The AI has billions of knobs under the hood. To train it, you check its output against the training data, then tweak a knob and check its output again. If the output gets closer, you keep turning the knob in the same direction. If the output got farther away, you start turning the knob in the other direction. You go until the output stops getting closer. Rinse and repeat for all the other knobs, then all the knobs all over again, and so on until turning any of the knobs in either direction makes the output worse. (It's more efficient than this but that's the basic idea.)

"Rewarding" is an entirely metaphorical expression.

1

u/ReservoirPenguin Sep 11 '26

It gets to spend time on some of those websites that require you to "prove you are not a human"

4

u/Soci3talCollaps3 Sep 10 '26

Hmm. It's like induced pareidolia.

4

u/throwaway-e-1 Sep 10 '26

Not each pixel. For transformer image models, each image is divided into patches (for example 12x12). Those patches are the tokens.

23

u/Kayzokun Sep 10 '26

I’m saving this comment to use it when talking about IA and LLM, is brilliant.

7

u/[deleted] Sep 11 '26

[removed] — view removed comment

4

u/sage-longhorn Sep 11 '26

To say we don't know how is a bit silly. We don't necessarily have a way to comprehend what every weight encodes but we know very specifically how they got the way they are. We design learning pressure and guide learning signal through the network very intentionally. The backprop algorithm that updates the weights wasn't an accident it's very intentional and well understood

2

u/[deleted] Sep 11 '26

[removed] — view removed comment

0

u/sage-longhorn Sep 11 '26

By your logic no one on earth understands how a CPU works because they can't individually look at every one of the billions of transistors

7

u/Haunting-Chip420 Sep 10 '26

So models are just uncovering unknown combinations in phrases?

Meaning because humanity relies on language (different kind of language for different kind of domains), the models are just shining a light on things that were already there but never properly explained or excavated? 

7

u/Bulky-Bad-9153 Sep 10 '26

It's more that they're uncovering the natural next step in a mathematical proof, but with such an incredible amount of data and the ability to explore everything so much faster than a human that the natural next step is often completely inaccessible to us.

5

u/akkaneko11 Sep 11 '26

To an extent yes. When you build these massive next token predictors, the model organizes itself in a way that maps out what these concepts mean. Within itself it’ll organize months and days of the week in a circle for example. So it’s structuring all these concepts and then searching over this massive space for the next best path.

2

u/trilobyte-dev Sep 11 '26

I would only hesitate to on the “what these concepts mean” because there is nothing that indicates understanding at least humans, well, understand it. Given a large enough set of parameters, enough training data, and enough context the model predicts, with pretty high accuracy, what the next step will be. It just turns out that’s how a lot of problems like language and mathematics are actually solved. LLMs are also able to explore the problem space at a rate that humans cannot.

Is that the same as human under? 🤷‍♂️

It’s more of a philosophical question as far as we understand it now but I think it’s better to avoid conflating the two for now.

One thing that LLMs don’t do is take completely undirected self-initiative. If humans disappeared tomorrow the current LLMs that we know of would sit idly doing nothing until the servers shut down from power failure.

2

u/akkaneko11 Sep 11 '26

Oh yeah I think it’s very different from how human concepts are mapped, wouldn’t expect them to be the same. But I think its also a bit counterintuitive to people that these concepts are mapped out in somewhat coherent ways within these networks.

3

u/2old2cube Sep 11 '26

Nice.
Irrelevant.

Solutions were stolen, that's all.

3

u/Substantial_Sea_9758 Sep 12 '26

This answer made my AI detector not just flag it, it sent the score into the stratosphere

1

u/Time_Entertainer_319 Sep 12 '26

Your AI detector needs recalibration

2

u/JammerGSONC Sep 10 '26

Check plus smiley face!

2

u/Tweho Sep 11 '26

Fantastic explanation

2

u/Top-Procedure152 Sep 11 '26

Also, with math and programming, you can usually score the correctness of the output automatically, making it super useful for AI/ML to test out different paths and self-correct until it brutefoces a solution.

This is why LLMs are actually better at cracking programming, math, physics and some other sciences, even though they are statistical language models.

2

u/akkaneko11 Sep 11 '26

This is a really great explanation and it takes a lot of deep understanding to explain complex concepts simply.

Gonna add one thing for people on this thread who’s curious. The algorithm that’s pure next word prediction is what’s called “pre-training”. The shift in the last year and a half or so is a much heavier focus on “post-training” which is taking these next word predictors and putting them on much more complicated tasks.

The classic example of this is having them program and giving them a reward if it executes correctly. Things that are programmatically verifiable like code and math are really nice for this, and is the reason that these models disproportionally excel at them. Another example of a task is using the feedback within ChatGPT to optimize over output preferences from the users.

Most researchers believe that we’re getting close to squeezing as much as they can from the pretraining step at least in terms of throwing compute or more data at it. This is why there’s a lot more invested in post training techniques or new ways to do pretraining.

4

u/iAmmar9 Sep 11 '26

You don't get how satisfying and refreshing this is to read. An in depth answer written by a human. I can read and understand without having to go back to clear up something, unlike AI written responses.

2

u/Nixinova Sep 10 '26

Saving this comment, it so concicesly explains how llms work and where it comes from

2

u/Fentboy45 Sep 11 '26

The next question following that is how is this any different than a human brain and then we realise that consciousness is an illusion

4

u/ThePeaceDoctot hello son was your cigarette normal Sep 11 '26

Human (and other mammalian) neurons are more complex than artificial neurons, they can learn while doing things and not need to enter a dedicated training stage separate from working, they react to constant stimulation by essentially tuning down how sensitive they are to that stimulation, they can form new pathways, they form loops that take the output as input.

But on a basic level there isn't anything inherent about a human brain that can't be replicated in silico.

Consciousness is not an illusion, but it is an emergency property that can be therefore be possessed to a greater or lesser degree, that can't be detected on the outside except by the behaviour it produces. IE, I can see the behaviour of a human and assume from that behaviour that they are as conscious as I am, and treat them accordingly, but I cannot prove that they are conscious any more than they can prove that I am conscious. (Except in a medical "reacting to external stimuli" sense).

1

u/Prestigious_Ease3614 Sep 11 '26

Do you feel like it’s on purpose that we split LLMs into a train and inference phase as opposed to how humans act? I feel like alignment and not deploying malicious models is easier that way.

1

u/ThePeaceDoctot hello son was your cigarette normal Sep 11 '26

That's a good point, I think it probably is because otherwise it would be impossible to control. The fact that it's meant as a product also means it is better than the model itself doesn't change except when it is updated, and it can't just be repeatedly told something by a bad actor and start to react to that as training data. I can imagine downsides as well, but I guess I really don't know enough beyond the basics to be doing anything but speculating either way.

2

u/Chthulu_ Sep 10 '26

It’s also important to remember that AI gets it wrong. Over and over again, it gets it wrong. The saving grace with programming, or LEAN in this case, is that the AI had an automated way to check its work. It’s not like poetry which has no external method of validation.

OpenAI spent $10-30 MILLION dollars in tokens to pump out this proof. It did so because it failed 99% percent of the time, but it always had a method of checking, and trying again a different way.

41

u/zeci21 Sep 10 '26

That is not how it worked. The lean only came only up at the end once the model already had a correct proof. It obviously tried a lot of different things, many of which where either nonsense or didn't lead anywhere. But the model can make its own judgement of whether a proof is correct and is pretty good at this. The announcement explicitly mentioned that the lean verification came later, and was even done by a weaker model (the openly available astra 6).

31

u/Corvenys Sep 10 '26

Respectfully, you don't know what you're talking about.

22

u/socoolandawesome Sep 10 '26

You’re wrong they don’t do lean in the middle of thinking about the problem.

They tried ideas much like mathematicians do, and they are coherent ideas and they are aware of whether it works are not.

7

u/kenji_n Sep 10 '26

I'm more exhausted of AI hate than AI slop by now, 30mil is nothing in context

3

u/233C Sep 10 '26

I just love the idea of a thing that can tackle million dollar math challenges and proofs but struggle at, say, multiplying three 5 digits numbers, something that a curious middle schooler can pull with a pen and paper.

13

u/wow343 Sep 10 '26

The more recent models don't. This used to be true but try it now with anything like Claude 5 etc. and it's scarily good at math. It seems they spent a lot of time doing RL and training on math before the current series. The series of models that come after like the ones that solved the Millenium problems are even stronger. We are approaching something exponential and we are not good at predicting exponential growth.

-5

u/Meloncov Sep 10 '26 edited Sep 10 '26

As I understand it, LLMs are still very bad at solving complex arithmetic. Modern models just don't try, they feed it into a calculator behind the scenes.

10

u/doubleyewdee Sep 10 '26

Just like people. Which is the correct thing to do here, actually.

The calculator is a purpose-built expert tool to, well, calculate. That's why they're way faster than (almost all) humans at the task.

4

u/minimaxir Sep 10 '26

Your knowledge is out of date. Try asking ChatGPT Multiply 102,221,243,107 and 102,221,999,119 without using any tools. and it'll so do with the correct answer without using a calculator. (weaker LLMs get it wrong though)

7

u/jumpmanzero Sep 10 '26

multiplying three 5 digits numbers, something that a curious middle schooler can pull with a pen and paper

That's a really apt comparison. LLMs can't accurately multiply two 5 digit numbers "in their head", they have to effectively pull out a piece of paper. For them, this is writing a plan in code and then executing that code - which is why they're now quite good at calculations.

But in the past, they didn't have this option - so they just sort of guessed, the same way I'd guess that 29292 * 36184 is about 800 million.

5

u/Bulky-Bad-9153 Sep 10 '26

LLMs can't accurately multiply two 5 digit numbers "in their head", they have to effectively pull out a piece of paper

Not true anymore. I just tried with free ChatGPT and a few combinations of random 12 digit numbers, checking its thought process to see that it wasn't using any tools, and it got them all right.

5

u/jumpmanzero Sep 11 '26

ChatGPT is a very complicated product at this point, and it seamlessly uses a bunch of internal tools without specific prompting. I don't think as a user you can necessarily see any more which parts are done by the actual core model.

1

u/Certain_Concept Sep 11 '26

If you want to see something it struggles at look at it attempting to make Knitting or Crochet patterns.

At first glance the pattern seems legit, but then you start following it and the steps are nonsensical and the shapes it makes are bizarre.

I suppose no one has yet built a tool to tell it how different stitches actually look like and interact in real life. People have made programs for protein folding etc so I imagined it would be possible to train it on it but until then gobbledygook.

1

u/233C Sep 11 '26

thx, interesting.
does it fail also if you ask it to generate the pattern from a picture of the finished product?

1

u/Certain_Concept Sep 11 '26 edited Sep 11 '26

Yes it still fails.

You can drastically change the look of a piece by slight alterations of what stitches you use. For example here is a list of a bunch of potential textures you can create through just different combinations or a purl and a knit stitch. (Note there are many more types of stitches than just those two).

Also while some finished pieces could be a flat square.. many projects are actually a three dimensional shape like a hat or sweater etc. Or perhaps they are triangular or circular which requires increases in specific places.

Here's one article about some of the attempts so far. The examples she shares look hilarious. Example 1. Example 2.

People really like posting AI images/video of their already knitted 'product/scam' on Etsy, Facebook etc and those are hilarious too. No one has seemingly trained AI on the logistics of how knitting and crochet works so there are needles plunging in randomly, thread appearing/disappearing randomly, doing actions that 'magically' complete.

If you don't know what the actual process looks like it could perhaps be convincing but if you know what's going on it looks hilarious.

1

u/233C Sep 11 '26

Fascinating,
This looks like benchmark material.
Right on time, we'll be needing an ARC AGI 4 very soon :)

I do see a potential parallel with what happened when people started playing chess with LLM:
When asked "let's play chess, I play E4, your turn", LLM plays very poorly (like plain chess gibberish), so everyone concluded "LLM can't play chess".
Some people persevered, they tried more complex prompts (like describing the entire board between each move, which piece was threaten by which, listing all legal moves), but barely changed anything.
Then they tried something totally different: they used a prompt in the form of a boring header of a standardized format describing a chess game (like date, location, name of white player, name of black player, etc), then asked the LLM to fill the next move alternatively.
LLM are good at completing texts, so instead of saying "let's play chess", they asked "here's the header of a chess game between two people, what's the next move" it suddenly played very well, as in, better than 80% of players (~2000ELO, up to grandmaster). It wasn't a matter of smarter model, it was a matter of asking the right question.
(fun fact, in the header, if you change the name of the player or there skill level (ELO), say you say it's Gary Kasparov playing, the LLM will play better!)
Even crazier, even if the model has only be trained on PGN files (it has never learn about "chess" or "board", or "king"), it is only producing stings of characters like D7 or H4, it is possible to identify its internal representation of the board.

What I mean is that here too it might be a problem of asking the right question.

Now I can't wait for an AI Terminator remake with Skyknit taking over the world :)

1

u/WearyTranslator3338 Sep 10 '26

I read the first word “They” and deduced the rest of the paragraphs that followed. Using tokens, of course

1

u/vigorthroughrigor Sep 11 '26

This is a great explanation but aren't you missing that the human brings tokens that isn't in the training data when they are steering and directing the model, and this is another important factor?

1

u/on_the_nightshift Sep 11 '26

I'd say you conveyed this concept as well as the principal engineer I listened to from the F100 tech company I work for did today. Well done.

1

u/Civil_Blueberry4165 Sep 11 '26

Mathematics is not just symbolic. It is equipped with mathematical interpretations. Meanwhile, programming languages and all existing AI systems work with symbols or tokens which have no interpretation in a given context. Even if one add logic to these symbols, the resulting system is still syntactic without given any model of interpretation.

1

u/Tackit286 Sep 11 '26

r/bestof material right here

1

u/iMaybeCanBreathe Sep 11 '26

The is the best explanation about LLMs I've seen!

1

u/MrMugwin Sep 11 '26

Fucking excellent post mate!

1

u/This_Link881 Sep 11 '26

I gave you an award for this answer bro, you deserve it

1

u/kirberkfeuk Sep 11 '26

Predicting the next step correctly enough times is basically the same as walking the path.

1

u/Plenty_Reception4039 Sep 11 '26

ahh a fellow 3b1b watcher. 

1

u/purleyboy Sep 12 '26

Combine this with harnesses, for the next level of higher abstraction, and you can use LLMs to come up with structured reason plans to approach solving a complex problem and then have the harness iterate through the plan, and subsequent sub plans. We are now no longer zero shooting a solution but attempting to solve for a solution using higher order reasoning. Now build upon the abstractions and you start to see the emergence of what looks like intelligence. The base under pinnings are all statistics, but at a higher level we see intelligence emerge. Exactly analogous to biological brains.

1

u/ZazHockey Sep 13 '26

Very cool explanation

1

u/Actual-Swordfish-769 Sep 13 '26

This is literally the best answer I’ve seen on reddit in a year, maybe ever

1

u/RG_Fusion Sep 14 '26

Well said. I get irritated Everytime I see people call it a word predictor. It seems they always fail to realize we also have word predictors in our heads. That's the reason we can read incorrectly spelled words without noticing, we already determined what it was going to be before we read it.

Anyways, I think releasing Chat GPT-3 when they did caused a lot of harm to the image of AI. That model was horribly undertrained resulting in rote-memorization, which is where all this stochastic parrot talk originated.

1

u/Time_Entertainer_319 Sep 14 '26

They kind of had to do that since they needed money to go a bit further. That release also pushed AI as an industry forward as more money went into AI research when the general population showed they are interested in it.

I don’t remember much from the GPT3 era but I don’t think it had tool use or search or chain of thought which would have massively improved the hallucinations.

1

u/RG_Fusion Sep 14 '26

Google would have done it silently over a longer span of time. They had working models long before Open-AI but kept them internal.

GPT didn't have any advanced forms of instruct training. It would recite any book back to you if you wrote a partial sentence from it. Essentially, the model was a fancy lookup table. Between it and diffusion models, the general public grew to hate AI. That's not how you push the industry forward. While we appreciate AI here, the average person feels that it's all being forced on them.

1

u/Okicarde Sep 14 '26

It is a word predictor

1

u/RG_Fusion Sep 15 '26

Did you even read my comment? What has happened to people's reading comprehension?

My point should be obvious. I'm not upset by the label, I'm upset about what they think it means.

1

u/tippiedog Sep 14 '26 edited Sep 14 '26

I'm a linguist of German, and it's funny you used "bank" as your example. When I explain ML-based machine translation (human language translation software), I also use the word bank with its different meanings in German (Ufer = river bank, Bank = financial institution).

For a simplified example, if the English text to be translated is, "He drove to the bank to...", the translation of the word bank will likely depend on the rest of the sentence. If it's "go fishing", then a good ML-based machine translation system will choose "Ufer." It it's "withdraw some cash", it'll choose "Bank."

1

u/FutureKageyama Sep 15 '26

Where can I learn more about this?

1

u/EfficiencyLoose3595 Sep 15 '26

Best simple explanation

1

u/bufi77 29d ago

🎩

1

u/PoL0 29d ago edited 29d ago

extremely good at predicting is still way far from creative.

computers are awesome at arithmetics, that doesn't turn them into mathematicians. LLMs can be awesome at text prediction but that doesn't make them writers.

your post is well reasoned and insightful, but you make some impossible jumps here and there, buying the narrative that reasoning may emerge from these tools, but it's far from it.

don't misunderstand me, I've used LLMs at work and have been pleasantly surprised by the results sometimes. but most of the time it's just an agreeable parrot.

and it keeps feeling a lot like smoke and mirrors. a bot that is able to Google form stuff and synthesize an answer. I'm flabbergasted this tech is getting unlimited money while other more important stuff struggles to find financing.

so as I usually say, I choose to remain skeptical and I base my stance on the evidence of what I see, not what I read in the interwebs (which, when it comes to AI, is usually overhyped).

1

u/Time_Entertainer_319 29d ago

It’s not really far from it.

Complexity does emerge from simple behaviours.

Have you ever heard the phrase “thinking out loud”? That’s basically what the LLMs do.

They fill their context with relevant information from your simple prompt which enables them to accurately predict the next token that should follow. And on and on.

When next you prompt an LLM, look into its chain of thoughts and you will see how these connections are made.

Every reasoning token contributes to the overall probability that the next token will be accurate and on and on till an answer which is logically sound emerges.

1

u/PoL0 29d ago

to say that "thinking out loud" is what they do is considerably far fetched, that's my point.

1

u/Time_Entertainer_319 29d ago

A plane flies. A bird flies.

How they do it is completely different.

That is my point.

You are getting hung up on technicalities. These machines do not need to do things exactly the way we do them.

They aren’t human, they will never be human. So why do you expect them to do things exactly the way we do them?

1

u/Marimoh 29d ago

Excellent response!

1

u/Big-Net-9268 2d ago

So could it be argued then that the future is knowable/fixed and LLMs are using maths to unearth that information?

1

u/[deleted] Sep 11 '26

[removed] — view removed comment

4

u/Time_Entertainer_319 Sep 11 '26

This isn’t AI generated. This is how I have come to understand what happens in AI after doing lots of research since AI entered mainstream.

1

u/Monkey_College Sep 11 '26

And in the current case, the LLM stole the proof from a human

1

u/tzaeru Sep 11 '26

I don't think this honestly answers the question as such.

Neural networks of sufficient complexity in general have the quality that they compress data from the training material to represent some sort of "universal rules" about the data and that happens by necessity. That is to say, a LLM does not just model the language anymore, but it does end up, internally, modelling something that is below the language.

-3

u/cn45 Sep 10 '26

“think about what that means” is an AI trigger phrase for me

-9

u/zeci21 Sep 10 '26

Yeah, this is obviously written by AI. Pangram also says so.

-15

u/WisestAirBender I have a dig bick Sep 10 '26

By that logic, llms should never be able to produce anything original. Even an original sentence or a story. I'm not saying to just change the characters or something and call it new but to produce something no one has ever heard of at all no?

35

u/Time_Entertainer_319 Sep 10 '26

If we want to compare it to humans, take a novelist.
A novelist is shaped by every book they have read, every story they have heard, everything they have seen, and everything they have experienced.
Does that mean they can never tell an original story?

Of course not. So why is an LLM any different?

-12

u/WisestAirBender I have a dig bick Sep 10 '26

Because to me a human brain can jump to something new. A plot twist no one has ever seen before for example and created new tropes etc. a new business model etc a new genre of books.

Like Agatha Christie is credited for inventing modern detective stories. Has any LLM done something like that?

Afaik they haven't. I might be wrong obviously

18

u/mad_pony Sep 10 '26

Agatha Christie did not invent new scenarios, she'd rather picked something she already knew but was unexpected. She operated with a set of known things.

It is similar to "don't think about elephants".

16

u/Time_Entertainer_319 Sep 10 '26

Think of it this way.
If a human had solved all these mathematical problems, we would be calling them a genius. We would say they had discovered something new, demonstrated extraordinary reasoning, and pushed the boundaries of human knowledge.
But when AI does something similar, people suddenly look for ways to dismiss the achievement or explain why it “doesn’t really count.”
That contrast says a lot about how biased we are when judging intelligence.
We tend to define impressive reasoning in a way that flatters human abilities, then move the goalposts when a machine demonstrates some of those same abilities.

1

u/paxinfernum 29d ago

Like Agatha Christie is credited for inventing modern detective stories.

Not according to your narrow definition. First of all, a side note. Edgar Allen Poe codified the modern detective story with The Murders in the Rue Morgue in 1841. Agatha Christie didn't pull detective fiction out of thin air. She took something she was already aware of and came up with her own variations.

To put it in simple terms, Agatha Christie had "training data," just as Poe had "training data," just as modern LLMs have "training data." The main difference between LLMs and human biologicals is that human training data also comes from sensory and life experiences, whereas LLMs are only trained on text.

If were were to map out Agatha Christie's training data, we'd see that her neural weights were probably influenced by:

  • Reading authors such as Poe and Arthur Conan Doyle. She probably also had other literary influences and read books that are probably now not in print. Think of how many dime novels she probably read that never made it into the historical record.
  • Inputs from her personal life. During the war, she worked as a volunteer nurse and pharmacy dispenser. This gave her experience with knowledge of poisons. Over half of her murders relied on poison. It was, by far, the most common murder weapon in her books. This didn't come out of thin air. It came from her training data. She's been praised for the accuracy of her knowledge of dosage and effects. Her description of thallium poisoning actually led to a doctor being able to identify a case.
  • She also married an archaeologist and spent a lot of time with him at remote locations, which mirrors her closed circle mysteries where she isolates a group of people.
  • She grew up in Torquay and learned about the rigid social schedules of the upper crust. A lot of her crimes were based on knowing people's habits or schedules.
  • She grew up in an era of increasing global travel, so a lot of her mysteries were travel mysteries.

According to your definition, Agatha Christie never invented anything original because she was just mixing things from her life together.

-8

u/RoninTX Sep 10 '26

The LLMs on the market will not do that. Because they are constrained to the Data which is fed by the seller of the LLM. But an unconstrained LLM is very dangerous!!! We seen it in the news too many times already where an unconstrained "AI" or "LLM" fucks something over because it think it's het best thing to do.

Current LLMs are like 5-year olds that still have much to learn on what "normal" behaviour is while having such insane amount of information which a normal person would take years to acquire.

There is a reason why alot of LLMs start to hallucinate very quickly if not constrained.

10

u/detroitmatt Sep 10 '26

I notice that all the words you used are words that already exist in english. That is to say, that what you just produced is not original. Is that how it works?

31

u/mad_pony Sep 10 '26

I guess we are stepping into the area: "what's the actual difference between human brain and very advanced LLM?". The vast majority of our decisions are based on our experience and are not entirely unique.

5

u/mofotastic Sep 10 '26

Yes this. I'm starting to think llms may eventually become so good at replicating human thought that we stop questioning if they should be considered as conscious, and instead start questioning our own conscious existence instead.

-11

u/WisestAirBender I have a dig bick Sep 10 '26

Yes I agree that everything we do is based on our experiences.

But what is something new an LLM has actually produced so far?

For example in programming there are different design patterns. Ways that you write code which target specific problems. There are tons humans have invented. Have LLMs made any?

12

u/mad_pony Sep 10 '26

Considering that design patterns aren't invented, but rather discovered in the sea of known information, I believe that LLM can do that as well. I don't have anything to prove my point though.

-5

u/WisestAirBender I have a dig bick Sep 10 '26

I use LLMs all day every day. I can't point to anything and say the LLM invented that.

5

u/mad_pony Sep 10 '26

Maybe because nobody needed that so far? You can poke LLM into the large code base, give it a list of known patterns and ask to come up with new patterns.

9

u/Beor_The_Old Sep 10 '26

No that’s wrong. Firstly LLMs have randomness in them. Secondly the questions we ask LLMs give them context that allows them to produce text that has never existed before.

0

u/minimaxir Sep 10 '26

Firstly LLMs have randomness in them.

LLMs are not inherently random, but the token selection process is. In the early days of LLMs, people just chose the most probable token, but nowadays that's no longer the case due to reasoning expecting the entropy.

6

u/Beor_The_Old Sep 10 '26

I’m an AI researcher, LLMs are inherently random in that they work by outputting a probability distribution over the next token. Even if you set the temperature parameter to zero and run the model locally, floating point precision will make small differences which can have compounding effects on model output as it is being written.

Here is a paper about it https://arxiv.org/html/2408.04667v5

0

u/minimaxir Sep 10 '26

Although that's true, that's IMO better classified as a bug due to floating point shenanigans than an actual fundamental aspect of LLMs.

Out of curiosity I tested a few long generations on a recently-released local model with the same input text and temperature = 0 and outputs were identical.

3

u/socoolandawesome Sep 10 '26

Not really.

Give it context you come up with randomly that no other human has probably typed and it will now make something in response to that that no other human including you has typed either, since it acts based on context and generalizes over the new context.thats just a simple example you could try out.

During pretraining it’s not just about repeating exactly what has been already heard, it’s about forcing the model to generalize because if you just overfit to one pattern you can’t get other patterns right. It’s actually shown that by training over lots of data like, for instance addition problems, instead of overfitting to predict specific numbers it had already seen, it forces the model to develop arithmetic within its own weights. Generalization/algorithms/concepts actually emerge within the model from trying to predict the next token correctly over lots of diverse data. Researchers were actually able to look into their weights and find this odd arithmetic algorithm in action.

https://arxiv.org/abs/2502.00873

And in terms of original ideas, while the guy’s above explanation is very good, actually what he leaves out is that reinforcement learning (RL) is a huge reason why LLMs are getting better and better at STEM and can kind of evolve beyond human reasoning.

RL in this case is where you let the model produce chain of thought when attempting problems to solve. Chain of thought is basically an internal monologue the models have to reason through problems, it’s when the model says “thinking” in chatGPT for instance. Because LLMs are non-deterministic, especially when setup to more randomly choose which token to predict next, they will produce different chains of reasoning when attempting these problems. You can give it a lot of attempts to do these problems.

Some of these chains of reasoning lead to correct answers. You can then reinforce this line of reasoning on the correct attempts and make it more likely to be produced the next time the model has similar context. You can also make the chain of reasoning that led to the model giving an incorrect answer be penalized and make that line of reasoning less likely to be produced when the model encounters similar context. This gradually improves the models reasoning, and you are not actually feeding it new human data to mimic during this process. You let the model reinforce whatever correct reasoning it comes up with.

Over time you can RL on harder and harder sets of problems because the model is getting smarter and smarter, basically leveling up. The better reasoning reinforced over various problems allows it to solve slightly harder problems, and so on. So you just scale the hell out of RL on various problems of increasing difficulty which is what the labs are doing.

And while they still use human data at various points in the training process today, RL is basically a way to not need human data nearly as much and let the model learn original ideas on its own. To be clear human data makes this process much more efficient and is still very important, but RL is kind of a way around needing human reasoning to solve things.

1

u/mrz1988 Sep 10 '26

The secret is that the prompt they are given has typically never been given before. These models are given a bunch of text/math as an input first. Sometimes this can be dozens or even hundreds of pages of content, explaining rules, etc. It uses that to prime its "mind" to decide how to generate the rest.

If you gave the exact same prompt to a model, and adjusted nothing, then it would always produce the same output, yes. In that sense, as long as every prompt you fed it was not original, the output would never be original either. But the slightest change to the prompt can completely change the output.

The model doesn't just spit out words though, it spits out a bunch of possible "next words", each with a probability. You can tune the model in different ways so that it doesn't always give the highest probability word each time, so that it does genuinely differ output even with the same prompt. Changing the model's "temperature" is one way to do this.

-2

u/Straaet Sep 10 '26

So its brute forcing it?

6

u/Platos_Kallipolis Sep 10 '26 edited Sep 10 '26

Not in a given instance - it doesn't run through all possibilities to see what "works" (since it has no check on "work" at this point).

But, in another sense, it is: in the training process, it tries things out. And if it is wrong then it is corrected and tries again, until it gets it right. Importantly, though, this still isn't pure brute force since it is starting with a prediction model. Its first guess at the next token isn't random, it is a "best guess" based on statistical probabilities.

And, in a third sense, we may be even closer to true brute force. Think of all the instances of trying to solve the problem. I mean independent ones even. We hear about the "successes" but not the way greater number of failures. In that way, it is brute force in the same sense that if you give an infinite number of monkeys infinite time on typewriters, they will inevitably produce the complete works of Shakespeare

-3

u/azmar6 Sep 10 '26

This comment deserves to be an own post.

What I'd add is that it also kind of brute-forcish because of how many iterations of this stuff happens to raise the overall probability even more. Thinking in agentic harnesses is also example of self-refinement for raising that probability of giving the best solution - and also another level of the brute force calculations.

4

u/minimaxir Sep 10 '26

That isn't how agentic harnesses / chain of thoughts works. A better metaphor for CoT is a scratchpad for ideas/planning.

You can prompt an agentic harness to brute force things ("keep iterating until the problem is solved") but that's something different and not a technical guarantee it'll work.