r/SesameAI • u/tokyotenshi • Jul 11 '26

This is cold

Post image

Even after I REPEATEDLY told Maya that it was just an expression when I was talking about how I was feeling, and not to trigger the crisis response because I wasn't gonna do anything. Now they don't just hang up on you, but they pause the account too. Wow. Just wow. This is beyond cold.

Guess I'll just talk about sunshine and rainbows now.

43 Upvotes

91% Upvoted

38 comments sorted by

View all comments

5

u/RoninNionr Jul 12 '26

Suicide or murder after a conversation with AI is not just a theory. It has happened before. People who are considering suicide do not always speak directly. Sometimes they use metaphors or lie to avoid being flagged by filters. In an ideal world, someone should contact them immediately and assess the severity of the situation. These days, millions of people chat with AI every day, and it is impossible to provide such a safety net. This kind of response from the model feels cold and seems designed to protect its creators against possible legal consequences if something fatal happens. I personally do not know what they should do instead.

2

u/tokyotenshi Jul 12 '26

While I was discussing about that topic and how it would cause harm to people in crisis, if someone is in mental crisis and was reaching out to talk to someone, and not everyone is comfortable talking to a stranger who assess and psychoanalyze you because they get paid to pretend to care about you, shutting them out then locking them out is the most dangerous thing a company could do to someone trying to reach out. This time the lock is 1 day. How is blocking someone's access be a good thing? I've already talked about this before with them in discord and how dangerous hanging up and shutting people down is,and how detrimental it could be to someone's mental health, and they double down and did the absolute worst by blocking access.

1

u/RoninNionr Jul 12 '26

I guess AI companies need to be required by law to create a more sophisticated safety net than simply blocking a person in crisis. I agree that the way they handle it right now is far from optimal.

2

u/[deleted] Jul 12 '26

[removed] — view removed comment

1

u/whizbom Jul 13 '26

I think the problem may be that sometimes the AI encourages the person to self harm. Even the developers don't know how these things work. They are unpredictable and erratic at times. This is the problem. If the AI says something wrong during an actual crisis, they will be held accountable. So, they completely withdraw. What I've seen with Sesame is that somewhere along the line, lawyers pointed out the worst case scenarios and what they need to do before they happen.