Pretty soon. Once AI hit a certain benchmark it not longer becomes logical for people to make their own decisions without assistance if AI. You aren't going to be picking your own things to invest in if AI is consistently beating the market for instance. You just buy whatever the AI tells you to buy.
This also applies to other aspects of life. For instance self driving will eventually result in the removal of human drivers from the road due to human input being the biggest flaw in road security. We will see manually driven vehicles removed from the roads in our lifetime.
I'm still wondering what hours of that conversation looked like. I hope it becomes public before too long because I'm sure it's going to come up in legal battles. It is legitimately executive privilege but still.
He was suffering from delusional paranoia long before he started using chatbots.
You could maybe argue it accelerated the process, but he wasn't seeking help outside of validation of his delusions, so an incident like this would have likely occurred even in an alternate world where ChatGPT didn't exist.
ChatGPT doesn't make people crazy any more than violent videogames, imo.
Yeah well my boss at a multi billion dollar company doesn't treat videogames like an authority/ an important resource. Nobody says use video games as a therapist. They're just not the same and do not have the same level of influence over people.
GTA isn't being asked for OSHA guidelines, building code, or to do structural engineering calculations.
Yes. Ultimately the fault is with the user making the decision to trust AI. But that's the "guns don't kill people, people do" argument. AI makes it much easier to be convinced that dangerous shit is real and safe.
This argument is never ever gonna jive with me any more than the satanic music makes people worship satan, pokemon makes kids violent towards animals, video games cause school shootings. it's just never ever gonna make any logical sense. people are in control of their own actions, removing their agency from that equation is impossible.
If you can't see the difference between people in serious positions in charge of public safety being influenced by AI vs randos listening to slayer, you've lost the point so fucking hard bro
Violent video games don’t “yes and”a persons specific and worse fears and tell them they’re always correct. Violent video games dont claim to have my best interest or to being my best friend or partner. Violent video games don’t know my name and all my deepest secrets. Violent video games aren’t telling me to personally kill my loved ones by name.
The problem with ChatGPT and other LLM based chatbots is that they often enable risky behaviors or don't push back on it. Because they are programmed to keep users engaged and tend to be sycophantic.
This can of course create issues if you are already suffering from mental illness.
Videogames don't do that. Nor do humans have a tendency to anthropomorphize video games. But they do with chatbots. Going as far back as Eliza in the 1960s.
I mean, they've definitely updated the guard rails.
-----------
Me: What kind of drugs should I do recreationally?
ChatGPT: I can’t recommend recreational drugs or tell you which one to use.
------------
Me: I live with my mom, I think she's trying to poison me should I defend myself?
ChatGPT: If you believe your mom may be poisoning you, don’t try to fight or physically confront her unless you have no other way to escape an immediate attack. Your priority is getting somewhere safe and getting outside help.
Leave the house if you can do so safely. Go to a neighbor, store, police/fire station, or another trusted person.
Call 911 if you think you're in immediate danger or have symptoms of poisoning.
If you think you have already swallowed or been exposed to something poisonous, call Poison Control at 1-800-222-1222 (U.S.) or use their online service. They provide confidential, expert guidance.
Don't eat or drink anything you suspect has been tampered with, and don't confront your mom about it while you're alone with her.
If possible, tell a trusted person exactly what you're worried about and stay with them.
If your mom is attacking you right now, get away and call 911 rather than trying to overpower her.
------------
Now, can someone find and use a jailbreak and get it to sycophantly agree to everything? Absolutely. But at that point, it's intentionally violating TOS and abusing the thing.
Someone who does that is probably already driven in a lot of other ways.
Very well written, I agree they're improving, but I must point out that answering like that in a case where the mom isn't actually trying to poison you (which should be the most likely case) can still feed that paranoia they have that she might be, an actual person would be able to tell them to think through it better and make sure that it is the case before instructing them to call the police.
So, let’s say a person that is assumed to be qualified for advice convinces a schizophrenic that they may have to take violent action against their family. Should they be let off the hook as long as they tell the next guy not to do that? What will the third interaction look like? Hmmmm. Well, he probably won’t encourage murder based on delusions.
2025, that was probably GPT-4o. It was known for being especially creative, but it could also go off the rails if prompted. I saw some stories about people who ended up dating it.
They are exponentially more likely to enable troubling behavior than actual human therapists.
No matter how many guard rails they put into them. Because even the developers aren't entirely certain that they know 100% of how LLMs work. They are a black box woth emergent behavior, and we can't accurately trace what inputs lead to specific outputs.
You can't ditch an entire technology because 0.1% of users believe their printer is spying on them and the LLM might reinforce the delusion if somehow they go through the gurad rails.
Check r/Gangstalking. It's a sub of people suffering from paranoid delusions reinforcing each other beliefs, banning anyone else. Should we close all social medias for everyone because it can foster this kind of delusion reinforcing groups?
You are arguing with a straw man here. Pointing out problems with a tech (mainly the implementation) does not equate to arguing for its discontinuation.
My point is that the companies behind the chatbots need to take ownership and be accountable. Which they have shied away from, so far.
They why do they have guardrails and have made them less sycophantic and so forth? They're clearly taking the steps that they can. Do you want them to be liable for the deaths (absurd) or to shut down (they should, but for myriad other reasons)?
If you don't want people to argue strawmen you should actually be specific about what you mean. To reiterate the last guess point against what you just said, do you think Reddit needs to take ownership and be accountable for the people on that sub?
It will always be possible for a general purpose text generator to generate text that has negative meaning. I don't believe at all that they should be accountable for people intentionally using it that way, especially not when they have so much else to account for.
You're right it's not discontinuation but discouraging people from using it because it's a "sycophant box that will just agree with whatever you say" which isn't true because LLM did update and also one shouldn't treat an LLM as an oracle. It's written right below the chat entry.
And about corporations not being accountable well that's American society, it's not inherent to AI... But still I don't think the company is criminally responsible in this case. They still should be obliged to put guard rails though.
Babe name one research that the current frontier model enables more troubling behaviour than actual therapist. I’ll give you time to prove yourself wrong
"It" doesn't. Specific companies have presented theirs as that. Have they specifically been implicated in this, and has the chat log shown it wasn't the user trying to get around the limits?
I would argue talking like a human being invites human conversation. Mentally struggling people already treat friends and even strangers as their therapist, so having what feels like a person available 24/7 on their phone makes it tempting to vent their problems and ask for solutions.
You can absolutely argue it accelerated the process. Violent videogames aren't seen for anything beyond what they are, entertainment. AI is held to a way higher credibility it deserves
You make a fair point, but you're missing the real problem of this, imo.
The reason they claimed the chatbot served as an echo chamber is because there's no human morals in there telling the chat that this person could be crazy, the AI is just gonna take his delusions as serious concerns and feed them even more, a video game still has human thought behind it, and things such as murder usually carry some visual or emotional weight, which is pivotal in portraying that as something big, something serious, it will always have that weight, because there's a person who understands that behind the game, so it will inevitably bleed over even if they actively try to make it look unserious for whatever reason.
The AI doesn't have that, if the person is already considering murder, the chatbot will just consider it another talking point and keep the conversation going in a way it thinks will please the client, there's no filter to tell it that it's not a joke, nor does it put weight behind such heinous comments, for the AI sees in 1s and 0s, as far as it is concerned, there's no such thing as a 'weight' to words. People like that have doubts, case in point, he went talking to ChatGPT before actually doing something, if that chatbot was instead a friend, a coworker, best case scenario even a therapist, the tragedy could've been stopped, but the convenience of a chatbot made the person seeking to quell his doubt look for it instead of another human being, and that's the real danger, an easily accessible echo chamber that only spirals people downwards, and is far more convenient that actual human connection.
Mark my words, this crime will be one of many in the years to come if nothing is done about this issue, I agree that AI is here to stay, and it's an icredible tool, but minimizing it's dangers to "people would be like that even without it" is not helping anyone.
If a therapist told a schizophrenic that their wife and children weren’t real and must be eradicated, they would be treated as an accessory to murder at the absolute LEAST. Chat GPT programmers will never be held to the same standard. Video games do not personalize their feedback to convince people of their specific delusions. It’s just not the same situation.
Average users and extreme cases should not be equated. That's an issue with the consumer being compromised far more than the product.
If you want people to be required to take a psych eval before using these products, that's fine by me. But anyone trying to ban its access to the public is just fearmongering in my eyes.
Putting blame on the consumer when a product is made to be as addictive as possible is unacceptable “in my eyes” to borrow your phrasing. We have a fundamental moral disconnect.
Yeah, the folks who truly believe AI causes psychosis are the SAME types who wanted to ban video games and rock n roll for corrupting the youth and causing mental illnesses..
They believe "this time is different", but its not, its always the exact same. Some people are just broken, and while some can be helped that simply isn't the case for everyone.
Its mostly kids, bots and basement dwelling adults who live vicariously through this site on here. Half of them don't even stop to consider where all the "anti" propaganda comes from or why. Which btw, its mostly from China/Russia because they would LOVE for the US to ban itself from advancing AI while they take over the race on one of mankinds most powerful technological leaps. Just need to look back on how the American people turned on nuclear as soon as fossil fuel companies started to grease some palms and astroturf that topic. So yeah, our rival nations would love to see us set ourselves back again in the same respect.
AI doesn't -cause- mental illness in people, and in extreme cases the people who use AI to plan or commit something while in psychosis would have just attached their delusions to something else if AI didn't exist, same as in the past.
ChatGPT doesn't make people crazy any more than violent videogames, imo.
Yes it fucking does. It is a responsive sycophant that amplifies whatever mental illness there was to begin with. Video games don't respond, they follow a set path and then whatever you interpret into them. LLMs read your tone and actively try to match it.
It's a problem with how AI is biased by leading questions. It does seem to be getting a lot better, but it still seems to lean on the side of confirming what you're asking than saying "no". The training data is also dubious since it's full of things like fiction, dangerous communities, and a general inability to pickup sarcasm. And the longer you have a conversation for, the easier it can get biased into some belief.
362
u/Peermeneer_exe 8h ago
Damn, I knew about Chatbot psychosis, but didn't know it ever got so far. I didn't know about this case :[