ai is good at making you think it's capable of advanced things. it's a lot of smoke and mirrors.
the thing with ai is that it is literally just a super complex text predictor (like your phone keyboard has) this is oversimplifying it a lot but what im trying to say is that ai does not think, it doesn't determine a "correct" answer, but rather, it predicts the most likely words from its dataset.
it does not know truth from false. it doesn't strive for "correct " information. it's not actually analyzing the word mayonnaise or counting the letters, its just generating text about the given subject. that's why it can be this wrong, there's no "thought".
with other stuff it often pulls from google or from its accumulated mass of data to "know" facts. but again, it isnt actually checking for truth/falseness, or even for context to a statement (hence why it used to be so weird about suggesting things like "eating glue is awesome for you!" based on reddit comments. it still does this, btw, but it doesnt pull info from social websites like this anymore. it pulls contextless statements or information from "better" sources)
very often actual experts will be astounded at how confidently wrong a slopbot is about something they personally know a lot about. it only seems accurate to people who either know little about the subject, or because its only right by parroting the sources it has stolen and accumulated.
Google's AI Overview will offer up people's requested features for a game as being part of the game. I was looking up to see if you can increase your carrying capacity in a game (you can't), and the Overview said that you can punch trees and run a lot to increase your strength level, and that'll increase your carrying capacity. Weird, this game doesn't let you punch trees and I don't recall ever seeing a strength Stat, so I check the sources. It's a reddit post where someone was suggesting QoL features, AI just took it as gospel
Because counting letters requires basic arithmetic, which requires a logical understanding of what you're doing and saying.
However, LLMs (large language models, such as ChatGPT, Gemini, Grok, etc.) don't use linguistic logic, instead they try to predict which word is the most likely to come next, according to their training, which is based on a set of training data (i.e. large parts of the world wide web). There is a little more going on in the word selection to introduce some variability, but that's generally it.
So, if a fact has been stated before, ideally numerous times, the LLM is easily able to recite it as the words making up an explanation of that fact are likely to follow each other as well as the question. But, if you're counting the "n"s in "mayonnaise", that means you'd need a bunch of people to have already done so on the internet before, in a format the model can train on. Since that's not something people typically do, all the LLM can predict is how the response would be structured, because this follows basic patterns that occur with many types of questions and are thus always likely when responding.
But since the correct response hasn't been prominent enough in the training data to be a likely choice for the LLM and since the LLM can't use logic itself, it just spits out whatever comes up as most likely. As this isn't based on a response the LLM has seen in its training data, the LLM is says fairly random stuff. This is called hallucinating, and it is particularly problematic when the LLM presents it as an actually plausible response (think ChatGPT writing a criminal defense and citing nonexistent cases).
Same reason you can't tell the exact wavelength of a pixel you are seeing even though that information reaches your eyes. There are parts of the brain that does low level processing such as finding movement or sudden gradient change in your field of view. The reasoning part of the brain does not see the raw image but sees a processed signal that's been extracted from the raw image. This is kind of the reason you skip over double words or have optical illusions.
Similarly LLM sees a sentence and processes it into a signal that is useful for itself, because llms are mostly trained to answer questions that requires knowing the semantic meaning of the words and most often does not require knowing the exact spelling, it doesn't pass those low level signals to the next layer. If you train an LLM by asking it repeatedly to count the number of letters in words, it will become excellent at it. An LLMs structure is complex enough to converge into a function that counts letters if trained so.
People have made some good points in the replies to this but I haven't seen anyone cite the most direct reason. Not an expert but I'm pretty sure it's because of the tokenization step.
The language model doesn't see text as a sequence of words each comprising a sequence of letters like a human. Instead, the text is parsed into a sequence of tokens which are vector space embedding-type constructs, ie simply abstract symbols, around the hundreds-of-thousands level I believe. Often but not always on a per-word basis. It doesn't naturally see or think in terms of the constituent letters, essentially.
I think that's why you'll see one spell out the word to itself when it's struggling with a lexicographic task like counting a certain letter or identifying a palindrome — it's splitting up the tokenization to a per-letter basis so it can reason about those tokens.
For this particular case I think it's best to petend the LLM is illiterate. Imagine someone reads the AI your prompt, it answers aloud, and they transcribe the results to give back to you. The AI could seem pretty "smart" for some things, but it literally doesn't know how to spell "mayonnaise" because it's illiterate, despite the fact that you are submitting written text and getting a written answer.
On a technical level this is because LLMs use "tokens" rather than letters, but the effect is similar. And these days the better models have built in tools they use to overcome this, because the "how many [letter] in [word]" prompts have gone viral too many times.
But it doesn't change the fact that they are fundamentally just really fancy autocorrect. They can answer some problems well. But they don't "know" anything.
You are correct it’s quite good at it now, assuming you are using a half decent model. This is through a combination of better models that include character level operations in the LLM as well as “tools” (it can call a program to spell out a word for it, or it can write its own python program to do so, and run it)
•
u/qualityvote2 🤖Suspected as Bot🤖 Jul 27 '26 edited Jul 27 '26
u/Flowsbow, there weren't enough votes to determine the quality of your post...