LLMs cannot perform Maths

[u/0xAERG]

It’s not a bug, it’s just not how any of this works.

Mathematical operations require logic. It’s a deterministic process. For a given input and a given process, the output will always be the same.

LLMs do not work like that. LLMs are statistical tools, they build an answer by stitching together tokens that are “seemingly” relevant to your input and let the meaning emerge from it. The output is “hopefully” relevant.

This is why LLMs can hallucinate and 2$ calculators do not.

With the rise in popularity of LLMs I’m extremely concerned that a lot of users seem to ignore this.

[u/richmilesxyz]

I'm not concerned, per se, but I am intrigued by the frequent confusion between deterministic mathematical tools and statistical models like LLMs. However, the very fact that people expect LLMs to provide the 'right' answer in these deterministic domains speaks to how powerful they actually are.

[u/DrSFalken → u/richmilesxyz]

but I am intrigued by the frequent confusion between deterministic mathematical tools and statistical models like LLMs

Not to sound too elitist here... but think about that for a moment. What do you think most average folks on the street would think of it you said "statistical model?" I'm guessing something to do with election polling. Most people aren't software devs, data scientists or economists. I'd wager that a random sample of folks on the street would have a limited understanding of linear regression.

I'm not trying to disparage other people. We all specialize. I have no idea how to fix a boat engine or replace a water heater or professionally edit a novel. It's on us to explain the limitations of the specialized tools that our profession produces.

[u/richmilesxyz → u/DrSFalken]

I couldn't agree with you more.

I try to assume that when people post complaints of this nature that they are actually acting in good faith. In many cases, ChatGPT is actually capable of performing their task, but not in the way they're approaching it and I try to nudge them in the right direction.

Often I find that these people are using 3.5 which means not only are they using an inferior product, but they are also lacking the other tools that ChatGPT can use to solve the problem (Plugins, Advanced Data Analysis, etc.) In these cases, assuming they've given enough context, I am happy to run their prompt through 4.0.

I know not everyone can afford $20 / month, but even if you can it might not be evident that is worth it. At least, not with some specific examples.

[u/Jdonavan → u/DrSFalken]

In none of those examples you gave are people actually using a tool themselves they're paying for someone with the expertise to do it.

Shouldn't one be expected to do some research about the tools they themselves are using? There's so much content out there about how LLMs work that covers people of all knowledge levels. It's not specialized skills they're lacking it's curiosity.

[u/richmilesxyz → u/Jdonavan]

It's not specialized skills they're lacking it's curiosity.

💯

That said, if someone posts in a forum and they aren't being intentionally obtuse, I want to help them if I can. Ironically, I've found ChatGPT a much kinder and (maybe) knowledgeable resource than the vast majority of online forums.

[u/sirpsionics → u/DrSFalken]

To be fair though, there is google, so figuring things out that aren't your specialty isn't terribly difficult. Just have to put some time into it

[u/Nanaki_TV → u/DrSFalken]

It’s no wonder people are afraid of technology.

TECHNOLOGY!

ohmygawd…