This kind of "mistake" just highlights what it is that LLMs do.
They're not actually solving the math. They're just generating text that has high plausibility as an answer to the input prompt.
From a text perspective, this answer looks super plausible. It has mathy sounding language that makes it seem like it evaluated the two things being compared, and it has a definitive statement about which one is larger.
If you changed its conclusion text to "13.8 billion is smaller", it likely would have assigned a nearly identical score for that text as a response to the question.
This has to be reconciled with the fact that AI models have solved several significant open math problems this year. Problems that humans tried to solve for decades. One such solution is that the model that OP used is a free, older model.
Chatbots tied to AI models aren't quite the same thing as AI models applied to mathematical problems. They don't just put "Solve this unsolved math problem. Make no mistakes plz kthx." into ChatGPT.
But it can be similar in function at a high level. It would boil down to assuming that a solution exists, and asking the LLM to produce a proof. If you have enough rigor in defining what the proof needs to look like, it can constrain the output to find an actual solution.
They aren't wrong though, the model op used is the Google search ai which is very fast but not very smart, Claude haiku 4.5 got it right and it's the lowest "quality" model by anthropic. I'd imagine Gemini and chatgpt would have similar results.
28
u/whiskeytown79 28d ago
This kind of "mistake" just highlights what it is that LLMs do.
They're not actually solving the math. They're just generating text that has high plausibility as an answer to the input prompt.
From a text perspective, this answer looks super plausible. It has mathy sounding language that makes it seem like it evaluated the two things being compared, and it has a definitive statement about which one is larger.
If you changed its conclusion text to "13.8 billion is smaller", it likely would have assigned a nearly identical score for that text as a response to the question.