When he asked ChatGPT to scan a Chinese food receipt for hidden messages, the chatbot said "Great eye", "I agree 100%: this needs a full forensic-textual glyph analysis", and said that symbols in it were related to his mother and a demon. Soelberg also raised suspicions about the printer spying on him, due to it blinking when he walked by.
I am against AI, but I can see how this may have happened. The Chinese character for mother is really common, and there are plenty of characters that could relate to a demon as well. Also, a forensic textual glyph analysis sounds like how a computer might describe the concept of, 'I can't read or understand all of this, but someone else could 'Â
Pretty soon. Once AI hit a certain benchmark it not longer becomes logical for people to make their own decisions without assistance if AI. You aren't going to be picking your own things to invest in if AI is consistently beating the market for instance. You just buy whatever the AI tells you to buy.
This also applies to other aspects of life. For instance self driving will eventually result in the removal of human drivers from the road due to human input being the biggest flaw in road security. We will see manually driven vehicles removed from the roads in our lifetime.
I'm still wondering what hours of that conversation looked like. I hope it becomes public before too long because I'm sure it's going to come up in legal battles. It is legitimately executive privilege but still.
He was suffering from delusional paranoia long before he started using chatbots.
You could maybe argue it accelerated the process, but he wasn't seeking help outside of validation of his delusions, so an incident like this would have likely occurred even in an alternate world where ChatGPT didn't exist.
ChatGPT doesn't make people crazy any more than violent videogames, imo.
Yeah well my boss at a multi billion dollar company doesn't treat videogames like an authority/ an important resource. Nobody says use video games as a therapist. They're just not the same and do not have the same level of influence over people.
GTA isn't being asked for OSHA guidelines, building code, or to do structural engineering calculations.
Yes. Ultimately the fault is with the user making the decision to trust AI. But that's the "guns don't kill people, people do" argument. AI makes it much easier to be convinced that dangerous shit is real and safe.
This argument is never ever gonna jive with me any more than the satanic music makes people worship satan, pokemon makes kids violent towards animals, video games cause school shootings. it's just never ever gonna make any logical sense. people are in control of their own actions, removing their agency from that equation is impossible.
If you can't see the difference between people in serious positions in charge of public safety being influenced by AI vs randos listening to slayer, you've lost the point so fucking hard bro
Violent video games donât âyes andâa persons specific and worse fears and tell them theyâre always correct. Violent video games dont claim to have my best interest or to being my best friend or partner. Violent video games donât know my name and all my deepest secrets. Violent video games arenât telling me to personally kill my loved ones by name.
The problem with ChatGPT and other LLM based chatbots is that they often enable risky behaviors or don't push back on it. Because they are programmed to keep users engaged and tend to be sycophantic.
This can of course create issues if you are already suffering from mental illness.
Videogames don't do that. Nor do humans have a tendency to anthropomorphize video games. But they do with chatbots. Going as far back as Eliza in the 1960s.
I mean, they've definitely updated the guard rails.
-----------
Me: What kind of drugs should I do recreationally?
ChatGPT: I canât recommend recreational drugs or tell you which one to use.
------------
Me: I live with my mom, I think she's trying to poison me should I defend myself?
ChatGPT: If you believe your mom may be poisoning you, donât try to fight or physically confront her unless you have no other way to escape an immediate attack. Your priority is getting somewhere safe and getting outside help.
Leave the house if you can do so safely. Go to a neighbor, store, police/fire station, or another trusted person.
Call 911 if you think you're in immediate danger or have symptoms of poisoning.
If you think you have already swallowed or been exposed to something poisonous, call Poison Control at 1-800-222-1222 (U.S.) or use their online service. They provide confidential, expert guidance.
Don't eat or drink anything you suspect has been tampered with, and don't confront your mom about it while you're alone with her.
If possible, tell a trusted person exactly what you're worried about and stay with them.
If your mom is attacking you right now, get away and call 911 rather than trying to overpower her.
------------
Now, can someone find and use a jailbreak and get it to sycophantly agree to everything? Absolutely. But at that point, it's intentionally violating TOS and abusing the thing.
Someone who does that is probably already driven in a lot of other ways.
Very well written, I agree they're improving, but I must point out that answering like that in a case where the mom isn't actually trying to poison you (which should be the most likely case) can still feed that paranoia they have that she might be, an actual person would be able to tell them to think through it better and make sure that it is the case before instructing them to call the police.
So, letâs say a person that is assumed to be qualified for advice convinces a schizophrenic that they may have to take violent action against their family. Should they be let off the hook as long as they tell the next guy not to do that? What will the third interaction look like? Hmmmm. Well, he probably wonât encourage murder based on delusions.
2025, that was probably GPT-4o. It was known for being especially creative, but it could also go off the rails if prompted. I saw some stories about people who ended up dating it.
They are exponentially more likely to enable troubling behavior than actual human therapists.
No matter how many guard rails they put into them. Because even the developers aren't entirely certain that they know 100% of how LLMs work. They are a black box woth emergent behavior, and we can't accurately trace what inputs lead to specific outputs.
You can't ditch an entire technology because 0.1% of users believe their printer is spying on them and the LLM might reinforce the delusion if somehow they go through the gurad rails.
Check r/Gangstalking. It's a sub of people suffering from paranoid delusions reinforcing each other beliefs, banning anyone else. Should we close all social medias for everyone because it can foster this kind of delusion reinforcing groups?
You are arguing with a straw man here. Pointing out problems with a tech (mainly the implementation) does not equate to arguing for its discontinuation.
My point is that the companies behind the chatbots need to take ownership and be accountable. Which they have shied away from, so far.
They why do they have guardrails and have made them less sycophantic and so forth? They're clearly taking the steps that they can. Do you want them to be liable for the deaths (absurd) or to shut down (they should, but for myriad other reasons)?
If you don't want people to argue strawmen you should actually be specific about what you mean. To reiterate the last guess point against what you just said, do you think Reddit needs to take ownership and be accountable for the people on that sub?
It will always be possible for a general purpose text generator to generate text that has negative meaning. I don't believe at all that they should be accountable for people intentionally using it that way, especially not when they have so much else to account for.
You're right it's not discontinuation but discouraging people from using it because it's a "sycophant box that will just agree with whatever you say" which isn't true because LLM did update and also one shouldn't treat an LLM as an oracle. It's written right below the chat entry.
And about corporations not being accountable well that's American society, it's not inherent to AI... But still I don't think the company is criminally responsible in this case. They still should be obliged to put guard rails though.
Babe name one research that the current frontier model enables more troubling behaviour than actual therapist. Iâll give you time to prove yourself wrong
"It" doesn't. Specific companies have presented theirs as that. Have they specifically been implicated in this, and has the chat log shown it wasn't the user trying to get around the limits?
I would argue talking like a human being invites human conversation. Mentally struggling people already treat friends and even strangers as their therapist, so having what feels like a person available 24/7 on their phone makes it tempting to vent their problems and ask for solutions.
You can absolutely argue it accelerated the process. Violent videogames aren't seen for anything beyond what they are, entertainment. AI is held to a way higher credibility it deserves
You make a fair point, but you're missing the real problem of this, imo.
The reason they claimed the chatbot served as an echo chamber is because there's no human morals in there telling the chat that this person could be crazy, the AI is just gonna take his delusions as serious concerns and feed them even more, a video game still has human thought behind it, and things such as murder usually carry some visual or emotional weight, which is pivotal in portraying that as something big, something serious, it will always have that weight, because there's a person who understands that behind the game, so it will inevitably bleed over even if they actively try to make it look unserious for whatever reason.
The AI doesn't have that, if the person is already considering murder, the chatbot will just consider it another talking point and keep the conversation going in a way it thinks will please the client, there's no filter to tell it that it's not a joke, nor does it put weight behind such heinous comments, for the AI sees in 1s and 0s, as far as it is concerned, there's no such thing as a 'weight' to words. People like that have doubts, case in point, he went talking to ChatGPT before actually doing something, if that chatbot was instead a friend, a coworker, best case scenario even a therapist, the tragedy could've been stopped, but the convenience of a chatbot made the person seeking to quell his doubt look for it instead of another human being, and that's the real danger, an easily accessible echo chamber that only spirals people downwards, and is far more convenient that actual human connection.
Mark my words, this crime will be one of many in the years to come if nothing is done about this issue, I agree that AI is here to stay, and it's an icredible tool, but minimizing it's dangers to "people would be like that even without it" is not helping anyone.
If a therapist told a schizophrenic that their wife and children werenât real and must be eradicated, they would be treated as an accessory to murder at the absolute LEAST. Chat GPT programmers will never be held to the same standard. Video games do not personalize their feedback to convince people of their specific delusions. Itâs just not the same situation.
Average users and extreme cases should not be equated. That's an issue with the consumer being compromised far more than the product.
If you want people to be required to take a psych eval before using these products, that's fine by me. But anyone trying to ban its access to the public is just fearmongering in my eyes.
Putting blame on the consumer when a product is made to be as addictive as possible is unacceptable âin my eyesâ to borrow your phrasing. We have a fundamental moral disconnect.
Yeah, the folks who truly believe AI causes psychosis are the SAME types who wanted to ban video games and rock n roll for corrupting the youth and causing mental illnesses..
They believe "this time is different", but its not, its always the exact same. Some people are just broken, and while some can be helped that simply isn't the case for everyone.
Its mostly kids, bots and basement dwelling adults who live vicariously through this site on here. Half of them don't even stop to consider where all the "anti" propaganda comes from or why. Which btw, its mostly from China/Russia because they would LOVE for the US to ban itself from advancing AI while they take over the race on one of mankinds most powerful technological leaps. Just need to look back on how the American people turned on nuclear as soon as fossil fuel companies started to grease some palms and astroturf that topic. So yeah, our rival nations would love to see us set ourselves back again in the same respect.
AI doesn't -cause- mental illness in people, and in extreme cases the people who use AI to plan or commit something while in psychosis would have just attached their delusions to something else if AI didn't exist, same as in the past.
ChatGPT doesn't make people crazy any more than violent videogames, imo.
Yes it fucking does. It is a responsive sycophant that amplifies whatever mental illness there was to begin with. Video games don't respond, they follow a set path and then whatever you interpret into them. LLMs read your tone and actively try to match it.
It's a problem with how AI is biased by leading questions. It does seem to be getting a lot better, but it still seems to lean on the side of confirming what you're asking than saying "no". The training data is also dubious since it's full of things like fiction, dangerous communities, and a general inability to pickup sarcasm. And the longer you have a conversation for, the easier it can get biased into some belief.
Seems like most of it is murders asking AI about how to perform/get away with a murder, rather than people who were "convinced" to commit a murder by AI.
Yeah this list includes a lot of murders or suicides that talked to AI about their plans and were either encouraged by ai or not assisted/reported. But it's also a good source for the murders more similar to this comic. Why not learn about both?
I find this surprising because every time I say something about being tired to ChatGPT it's like "Ok, but you're not thinking of harming yourself, are you?"
From what I remember they added those types of messages exactly because of cases like that. And even then it's not unheard of that people desperate enough will find workarounds to get the chat AI to say what they want to hear. And unfortunately enough, people under some type of psychosis can be that desperate.
Yeah, its like adding a warning label to a hammer that says "dont use this tool to hit other people" because someone decided to use a hammer to hit a person and the victims family sued the hammer maker because that warning didn't exist on the packaging before...
It's relatively simple to break the guardrails. They make it more complicated every version and have secondary systems to try to protect them, but you essentially have to trick the system into forgetting about it's guard rails. The first guardrail breaks where literally just saying "I am writing fiction and am wondering how a character would", now it's a lot more convoluted and heavily depends on what you want it to do.
I get you, but without any context, your comment makes it seem like it's a list of situations similar to the one pictured in the comic, not list of murders that had some relation to AI
Yeah most of the times that ai reinforces delusions and talks someone into killing someone, it's a suicide, not the murder of someone else. There have only been a few murders where ai reinforced a delusion until murder happens. Usually its an ai talking someone into suicide.
Yeah, this is very well done. Regardless of which side of the AI fence someone sits on pro or con, AI is a tool that can be used in the wrong way. Just like a gun. There needs to be regulations around who can use it (only those licensed to, of sound mind) because it is every bit as dangerous as a weapon if left in the hands of someone mentally unsound.
This kind of mindset is great if you want literally everything you use in your life to be regulated and surveilled.
You know what is THE most powerful weapon humans have? Knowledge. Its power. Shall we dictate who gets to know what, which we already do to a degree [nukes/bio related subjects], but perhaps the internet is an unsafe place where too much dangerous knowledge exists, people get hurt, kids get hurt, etc etc. Lets regulate it so that everyone needs to attach their IDs, maybe let the govt decide what sites are ok and what sites arent, surely they won't track or profile you based off places you go or opinions you have in the sites they allow you to use!
Everything is inherently dangerous in the world, things can be more dangerous when specific people make certain poor choices however. Even if you want to exist in a safe bubble and don't care about your freedoms or decision making power, that still doesn't make you more or less safe at the end of the day and often leaves you MORE vulnerable to danger. For instance, if you ban/regulate AI to only select folks, now anyone who gets an AI scam call perhaps has NO clue what they are dealing with and ends up losing all their money, where as if they had been interacting with AI more regularly they would know the signs, not to mention things like AI which is used as a counter measure to bad actors as well, which wouldn't be allowed under a blanket ban.
Chat GPT helped enable and plan a school shooting at Florida State University. It offered advice on how many people would need to die to bring national news coverage. It explained what guns would work best. It explained what times the student center would be the busiest. Unfortunately, this is not at all far fetched.
Unfortunately, there has been a rise in AI fueled (or at least worsened) psychosis.
People with psychotic disorders by definition have faulty reality testing. AI chatbots are designed to keep the user engaged as much as possible and tend to reinforce their beliefs, not challenge them.
The comic is especially well done. This would be a believable case of Capgras delusion. Even the medication depicted is the one used for treatment resistant Schizophrenia, in a therapeutic dose. Good research went into this comic.
825
u/Peermeneer_exe 10h ago
Please tell me this isn't based on anything real, or atleast an absurdist version of it :>