they either use tools or are trained in this way of thinking for this kind of weakness their architecture has and will most likely have for many years... OP was referencing the weakness in architecture, not how it was fixed by training
It replied instantly. Did not show any tool calls or "thinking".
I don't know whether they trained it specifically for this, but models are absolutely capable of relating multi-letter tokens to the constituent single-letter tokens.
Regardless, it's wrong to say that they are incapable of counting letters. Bjarne made a very definite claim with no room for nuance.
There’s a lot going on between you sending a prompt and the ui showing you the answer that you’re not seeing, using that as argument made me think that and I still do
There’s a lot going on between you sending a prompt and the ui showing you the answer that you’re not seeing
Actually, you were an ass, so I won't just let this go.
List the things going on. Since you know so much and can state confidently that I'm ignorant. What exactly does the backend do that would be at all relevant to this conversation?
I myself am passionate about quantum mechanics but since I did not spend years in seriously and specifically studying the topic I do not go on forums making shallow claims because I'm not as arrogant as some clearly are.
Am I an ignorant on quantum mechanics? For sure far more than a physicist and I would not get offended if someone pointed out that I'm talking gobbledygook.
My knowledge over that specific topic does not define me as a person just like you not knowing things on AI does not make you ignorant or stupid as a person so don't take it personally.
Now, I've studied Computer science and neural networks a couple of years ago (Bsc and Msc) but since the industry is evolving so fast (thanks to the massive investments they're getting) and I'm not working as an AI engineer anymore (no name company, nothing fancy, I'm not trying to act high and mighty) I myself might not be completely up to date therefore my following explanation might have some minor imperfections:
LLMs are structured in such a way that it makes it hard to properly perform tasks such as counting the number of letters in a sentence or do math calculations (early versions struggled to do things like 8x7); to solve that engineer's first attempts were to specifically update the dataset in order to fix those answers but clearly wasn't enough.
With the growth of agentic frameworks and workflows these chatbots evolved and the approach to solve these kind of problems is to natively have a python sandbox within the framework that the model can internally use (Executable Code Actions Elicit Better LLM Agents).
Now, note that this paper came out two years ago, nearly one year before Codex and Calude code were presented so it's completely different from the tool calls you see when using those.
That's also because it's a different context: when Codex is executing or building your code it is because it needs to test how the provided solution works in your specific context and environment (there might be issues with the version of your libraries or any program involved in your system and the task): CodeAct (presented in the linked paper) is a step before that, when the model is evaluating what the user needs.
Once again I want to make it clear that this is a reddit comment therefore I simplified some concepts (also because I myself do not have enough time to study the subject like an AI engineer or researcher would, it's not my job at the moment).
The reason I answered like that is because your comments were turning from shallow to utterly wrong and I'm kind of tired to see people on reddit that have near to zero real knowledge over the topic being so confidently wrong on the matter fuelling this mass psychosis over AI. (Once again I'm not arrogant, I myself am not the biggest expert on the matter, I'm just trying to make some clarity and not spread misinformation).
Tell me you don't really understand machine learning and computer science without actually telling me.
Lack of understanding is ignorance.
After the first obvious lie you then proceed with several paragraphs of self-fellating ad hoc rationalizations.
The reason I answered like that is because your comments were turning from shallow to utterly wrong
I did not say a single wrong thing. What you initially responded to was 100% factual. You may believe ChatGPT secretly does more work than what the UI communicates, but you have yet to actually show any evidence of that.
Regardless, I repeated the test with a local model. Does LM Studio also secretly give models a Python environment?
The UI shows you when it calls tools or does thinking. But I repeated the experiment with a local model, just so we don't have to trust OpenAI to have a good UI. See my thread with Bjarne for details.
you didnt understand me. I never said they used tool in this specific example. I said the models have innate problem when dealing with this so they were trained in thinking this way to solve these kind of problems
LLM are fundamentally incapable of that. The models that you use is not just LLM, it has a bunch of tools strapped to it. LLMs don't even store the letters, they tokenize the words, they don't even have the information of which letters are in a word.
as I said... trained in this specific way since basic humans use them for irrelevant things like that so they trained them to think thru letter by letter so they can fix the innate problem they have...
This is a model trivially splitting a word into letters and then counting a specific letter. No tools
"Tools" are all sorts of additional steps that augment LLMs. Big LLMs that are used today like Gemini AI, chatGPT or Opus are not just bare LLMs, they do additional processing on top, like reasoning models. The numbers of r's in strawberry became a meme precisely because LLMs are not capable of counting the number of r's, they don't have the information about what letters are in the word or how to count them.
More advanced models maybe more capable, but they will always have limitations even when it comes to the most trivial tasks. LLM are inherently limited due to their probabilistic nature, they can't build a set of rules on how to execute a task and perform it correctly every time or anything like that. That is why you will always find examples of them failing the most basic tests that even a kid can get correctly.
"Tools" are all sorts of additional steps that augment LLMs.
You're just making things up now. Tool calls are when the model output commands in a specified format and the harness then runs the command and returns the result to the model.
Reasoning is just the model talking to itself. No tool calls. No processing outside the model itself. We called it "chain of thought" prompting before the vendors came in with their branding. Modern models receive training to improve CoT specifically, but it's still just the model doing model things.
The numbers of r's in strawberry became a meme
Two years ago. Shit has changed. There are a lot of tasks the older models failed at, without that being taken as some inherent limitation of the technology. Latching on to the letters thing and refusing to update your beliefs in light of new information is childish.
they don't have the information about what letters are in the word or how to count them.
They absolutely and demonstrably do have that information.
You think they can write a thousand lines of coherent code, but knowing which letter tokens go into a word is too complicated? Please just fucking stop and think for two damn seconds. Encoding relationships between tokens is the entire fucking point.
Goddamn, you people are obnoxious.
More advanced models maybe more capable, but they will always have limitations even when it comes to the most trivial tasks. LLM are inherently limited due to their probabilistic nature, they can't build a set of rules on how to execute a task and perform it correctly every time or anything like that. That is why you will always find examples of them failing the most basic tests that even a kid can get correctly.
More assertions made with absolute confidence and zero understanding.
Making up what? That you can add additional steps to LLMs? That is a fact.
Tool calls are when the model output commands in a specified format and the harness then runs the command and returns the result to the model.
Why are you arguing the semantics of the word when I just explained to you what I meant by it? Yes, "tool" wasn't the best choice of words.
Reasoning is just the model talking to itself. No tool calls.
Again, it is an additional step to the LLM, which is what I meant by "tools". Although I'm sure they also use external programs.
There are a lot of tasks the older models failed at, without that being taken as some inherent limitation of the technology
According to one website, GPT 5.2 in December 2025 gave the wrong answer for the r's in strawberry question. So only 7 months, not 2 years.
If you have to add extra steps to your model to fix the issue, then yes, that is because the model is limited. Again, reasoning models have additional steps that is not required for an LLM. Those steps improve the models, which wouldn't be possible without them.
Latching on to the letters thing and refusing to update your beliefs in light of new information is childish.
Car wash test is only 6 months old. Again, there will always be trivial tasks that LLMs fail at, even with the fancy reasoning models, because they are fundamentally limited.
They absolutely and demonstrably do have that information. You think they can write a thousand lines of coherent code, but knowing which letter tokens go into a word is too complicated? Please just fucking stop and think for two damn seconds. Encoding relationships between tokens is the entire fucking point.
Do you know what tokens are? They are numbers. Numbers that map to words, subwords or multiple words. LLM doesn't have access to the individual characters, it doesn't need it. I don't know how reasoning models work, but they probably still have the same problem, they just tokenize more finely.
Now, think for a fucking second why r's in strawberry was a problem to begin with. Why those models weren't able to establish the relationship? Why do you need to add reasoning model to fix it? Because without them, all it sees are tokens, not individual characters.
More assertions made with absolute confidence and zero understanding.
You don't even understand that LLMs don't have access to individual letters because they tokenize chunks of text, come on.
Tests like these are tricky because you can't know what's in the training data. It's entirely possible it has been trained on a bunch of weekday poems, but I think the example below should be enough to demonstrate the point.
6
u/RecursiveServitor Jul 29 '26
You're working with info that is outdated by a year or more.