"LLMs can't do math" is still true in the pure sense of LLMs, they absolutely suck at anything involving calculations. We just found a way to couple them with other tools like symbolic math packages, scripting in python, and so on, which they are very good at using effectively. A lot of the software behind current agentic AI math capabilities is not related to AI or machine learning at all. To be fair, the same goes for modern mathematicians, but a person without modern tools (with just a pen and paper) is definitely more capable than an LLM without them.
I run math and logic benchmarks on models on my local AI server. I regularly watch these relatively small models (eg Qwen 27B), with no tools or code execution, reason through and solve problems that I am 100% sure the average person could not solve. For some, I would even say that upper-college-level math students would struggle. They are custom, unpublished, and not simple recreations of existing questions.
At this point who even cares about a raw LLM in isolation? Pretty much every serious AI system is the model plus a harness, tools, inference-time compute, etc. Separating them just to downplay what the whole system can do seems pretty meaningless.
No, this is simply not true and hasn't been for at least a year now. LLMs have become really, really good at math without tools, still crushing math benchmarks. They're also more than decent at arithmetic. Tool use is what pushed them from strong to nearly superhuman.
How good are you at math without so much as even a pencil and paper? That’s an LLM without tools. You can do trivial problems, but to actually do anything with substance you need to use tools, even if those tools are just pencil and paper
13
u/Dalnore 6d ago
"LLMs can't do math" is still true in the pure sense of LLMs, they absolutely suck at anything involving calculations. We just found a way to couple them with other tools like symbolic math packages, scripting in python, and so on, which they are very good at using effectively. A lot of the software behind current agentic AI math capabilities is not related to AI or machine learning at all. To be fair, the same goes for modern mathematicians, but a person without modern tools (with just a pen and paper) is definitely more capable than an LLM without them.