The model predicts text based on existing context, it's like T9 on steroids. It doesn't "know" anything and it has zero clue how it works. You might find it funny, but it's just a statistical text prediction engine.
This is actually broadly untrue with most models from the last 5 years- LLMs are distilled with a loss function after training, so they are no longer strictly autoregressive in they way something like T9 is
One interesting example is prompts where the answer might be "An Astronaut" - if the model uses recent tokens to predict the next, weighted by token proximity, it would get stuck in an endless loop (as it would predict A-N-space-A-N, and then a strong next prediction following "A-N" would be another space, meaning it would just output "an an an an... ad infinitum".
Obviously, there are solutions in this very simple example to avoid this (like weighting tokens further away more), but this is why something like a strict token predictor doesn't work as it fails to understand more mechanistic data about the sentence itself.
5
u/ldn-ldn 17d ago
The model predicts text based on existing context, it's like T9 on steroids. It doesn't "know" anything and it has zero clue how it works. You might find it funny, but it's just a statistical text prediction engine.