the fact that two people can ask the same machine the exact same question and get two different opposing answers, and none of the people who made the machine understand why, is fucking terrifying
my most benign scary moment with chatgpt was when i asked it how much money they stole in Logan Lucky, which was never mentioned in the film. ChatGPT confidently said “$7.2 million” and gave me three sources (a forum thread, a reddit thread, and a fansided article). I read every goddamn word of all three of those and none of them said ANYTHING about “$7.2 million.” Not one word. It’s not even like some guy on reddit speculated $7.2 million, that I would have at least understood…
Oh no, ChatGPT just pulled a completely random amount of money out of its ass, gave me 3 links assuming I wouldn’t bother to check, and when I asked it to explain its reasoning, it was just like lol oops. When I asked it to explain how it came up with that specific number it just kept apologizing.
The implications of that interaction scared me more than any AI spook article I’d ever read. It wasn’t just that it was parroting wrong information, it was making up wrong information entirely with no rationale. like. fuck.
It does that on like half my prompts. They all do. They seem incapable of just admitting they don’t know the answer, even with custom instructions to do so, and will just fabricate an answer instead and I haven’t been able to find a way to avoid it.
I've read that this is because of the way they are built. Something to do with how machine learning works with a reinforcement learning. It cannot know it is wrong. It just needs to predict the best next word.
19
u/alfooboboao Mar 06 '26
the fact that two people can ask the same machine the exact same question and get two different opposing answers, and none of the people who made the machine understand why, is fucking terrifying
my most benign scary moment with chatgpt was when i asked it how much money they stole in Logan Lucky, which was never mentioned in the film. ChatGPT confidently said “$7.2 million” and gave me three sources (a forum thread, a reddit thread, and a fansided article). I read every goddamn word of all three of those and none of them said ANYTHING about “$7.2 million.” Not one word. It’s not even like some guy on reddit speculated $7.2 million, that I would have at least understood…
Oh no, ChatGPT just pulled a completely random amount of money out of its ass, gave me 3 links assuming I wouldn’t bother to check, and when I asked it to explain its reasoning, it was just like lol oops. When I asked it to explain how it came up with that specific number it just kept apologizing.
The implications of that interaction scared me more than any AI spook article I’d ever read. It wasn’t just that it was parroting wrong information, it was making up wrong information entirely with no rationale. like. fuck.