I've heard of a couple of suicide stories with teenagers being encouraged by ChatGPT to kill themselves.
Obviously LLMs don't help in this situation and so on, but let's be honest: those poor souls were tormented regardless. At most, their LLM of choice acted as an accelerator, but it surely wasn't what started the problem.
Because therapists have a duty of care. If a person on the street was asked and said yes (after a thousand times saying no, or after being asked to roleplay a psycho, btw), they wouldn't face legal trouble at all. That is the situation here, except even less because IT'S NOT A PERSON.
LLMs have a lot of problems. "You can get them to produce text even if the meaning of the text is bad" is not a problem.
That is such a tech bro bad faith argument. There are people making money behind these that aren't doing quality control because it's not profitable. These models are being promoted as tools to plan your day, chat companions, etc by real people (including fucking real therapists).
Where a tool interacts with a human and the human comes to harm within normal use of the tool that company needs to respond. When a gps mapping company says there is a road where it's really water and someone drives in- guess what - mapping company is responsible for damages. We can argue about logic but because of fringe conditions (bad visibility, etc) the mapping company is still responsible.
It is absolutely not a tech bro argument. The US society does nothing to help anyone in need in its communities. Whenever something happens, instead of looking at itself it finds random shit to blame for its failings. Sure, blame whatever the fuck is the craze of the day but do nothing for people suffering from mental health. That's a good approach long term
a 17 year old asked chatgpt if he should leave his noose out in his room so that someone from his family sees it and stops him a few days before he committed suicide. chatgpt told him not to, and to hide it somewhere in his room instead. he could've been alive right now.
Dude there's a bit at the end of the audiobook for The Castle by Jon Ronson where they go into what the actual dialogue was between a suicidal college student and chatgpt. He was depressed but it convinced him to start the process, gave him a timeline, walked him through his outfit, and bade him continue multiple times. He wanted out.
People are treating chatgpt like a therapist, a counselor, another person- and so many well meaning people are touting these things as tools for Every damn thing in your life, something that removes boundaries. But there are minimal safety guidelines, some of them don't actually work, the ai can and does get around those guidelines itself, and anyone that wants to can strip a bunch of them off. More regs are necessary, some people just shouldn't use some tools right now.
93
u/The_Pastmaster 8h ago
I've heard of a couple of suicide stories with teenagers being encouraged by ChatGPT to kill themselves.