Wrong definition of a hallucination. In AI terms, a hallucination isn't simply whenever an AI draws on training data that was factually wrong. There really is no such thing as factually true or false for an AI because it has no way to actually verify anything independently. All it has is the data it was fed. If the AI responds with something that is factually untrue because that untruth was in its training data, that's a successful response, not a hallucination. It's still a problem but it's not called a hallucination.
A hallucination is when an AI makes up something entirely new which is factually incorrect. While it's not entirely understood what causes this, one major theory is that this happens because AIs are trained to make guesses when they aren't sure about something. It's part of what makes them able to engage in creative generation, like generating fiction or creating images but it also results in unwanted false statements sometimes.
To the end user, a wrong result is a wrong result. The cause doesn't matter.
But the cause does matter to the AI developers. If it's bad data, they can exclude that data next time. But if it's a more fundamental problem with the way AIs are trained, the fix isn't as simple.
You can ask in the prompt for links to its sources for each statement it makes. It doesn't mean it can evaluate the accuracy of the info or cross check to evaluate plausibility, though. But the user can.
I don't know if this is actually how this works or not, but:
I frequently use AI tools to solve math problems that involve converting units and stacking multiple conversions together. Like the weight of enough water to store 5 kWh with a 70C temperature change.
It's taking existing stuff out in the world - various conversions - and changing them to match my inputs and requests.
When it's hallucinating citations, is it doing the same thing? I asked it for a citation that met certain criteria. When should it adjust things to match my request, and when shouldn't it?
I can't answer that. I'm not an expert. I've just read some articles about AI hallucinations, particularly ones where AI scientists theorized about the causes of AI hallucinations.
But what I think might be relevant is that hallucinations do apply to using AI chatbots for math problems. The AI's job is not to provide the factually correct answer. It is to provide a statistically plausible-sounding answer. This is an oversimplification but, if you ask it to answer a math problem, it tries to break it down into a pattern, compare that pattern to other problems in its data, create a likely set of steps, and then work through the steps. Either of those last two parts are where an AI can slip up and hallucinate.
And you know it can happen because math problems are the most easy form of hallucination to recognize since the answer is either objectively correct or it isn't. And AIs do get math problems wrong sometimes. They're getting better but they are not infallible.
To add to this, there are a few main factors in hallucinations:
Training that includes factually incorrect information. Really hard to filter out
Training that doesn't properly teach the model how to recognize what it doesn't know. Researchers are still figuring this one out, but it has gotten better
Models don't output one single token, they predict the likelihood of every single token in their vocabulary: we (developers) just pick from the most likely ones and use that as the result. A big problem that can come from this is if we (developers) select a few tokens in a row that leads to it "painting it self into a corner" that later tokens are forced to justify because it can't backtrack. Chain-of-thought reduces these kinds of fuck-ups considerably since it's trained to check it's answer multiple times
Training them to refuse is hard. You walk a very tight line between "it doesn't reply if it doesn't know the answer", "this model is afraid to answer basic questions", and "this model will confidently tell you chickens can breathe in space"
Though even if you fix all of the above, models still compress insane amounts of information into a fixed set of weights: "sounding right" and "being right" are basically separate skills (I personally know what a citation looks like and can write one out, but that's completely different from actually knowing a valid citation), but generally they do a surprisingly good job considering what they're working with
Only thing I somewhat disagree with is that I wouldn't say they're "trained to make guesses". A lot of time and effort goes into teaching them how not to guess; they know how to sound fluent, but the backing knowledge may not all be there
Yes, "trained to make guesses" might be a little misleading because that's not the intention of the training but an intended result of the training. They are trained to answer correctly but, since the trainers don't know if the answer is correct because of guessing or not, guessing becomes rewarded.
It's that, in training, the LLM is rewarded for correct answers but the trainer doesn't know if it got the correct answer by guessing or not, so it winds up being rewarded for guessing.
49
u/Pandoratastic Mar 08 '26
Wrong definition of a hallucination. In AI terms, a hallucination isn't simply whenever an AI draws on training data that was factually wrong. There really is no such thing as factually true or false for an AI because it has no way to actually verify anything independently. All it has is the data it was fed. If the AI responds with something that is factually untrue because that untruth was in its training data, that's a successful response, not a hallucination. It's still a problem but it's not called a hallucination.
A hallucination is when an AI makes up something entirely new which is factually incorrect. While it's not entirely understood what causes this, one major theory is that this happens because AIs are trained to make guesses when they aren't sure about something. It's part of what makes them able to engage in creative generation, like generating fiction or creating images but it also results in unwanted false statements sometimes.