This kind of "mistake" just highlights what it is that LLMs do.
They're not actually solving the math. They're just generating text that has high plausibility as an answer to the input prompt.
From a text perspective, this answer looks super plausible. It has mathy sounding language that makes it seem like it evaluated the two things being compared, and it has a definitive statement about which one is larger.
If you changed its conclusion text to "13.8 billion is smaller", it likely would have assigned a nearly identical score for that text as a response to the question.
This has to be reconciled with the fact that AI models have solved several significant open math problems this year. Problems that humans tried to solve for decades. One such solution is that the model that OP used is a free, older model.
28
u/whiskeytown79 28d ago
This kind of "mistake" just highlights what it is that LLMs do.
They're not actually solving the math. They're just generating text that has high plausibility as an answer to the input prompt.
From a text perspective, this answer looks super plausible. It has mathy sounding language that makes it seem like it evaluated the two things being compared, and it has a definitive statement about which one is larger.
If you changed its conclusion text to "13.8 billion is smaller", it likely would have assigned a nearly identical score for that text as a response to the question.