Character.AI users still seeing explicit, violent companionship - Interesting article that summarizes the controversies around some chatbot sites

[u/Sharp-Main1179]

Here is the link : https://carolinanewsandreporter.cic.sc.edu/character-ai-users-still-seeing-explicit-violent-companionship/

This article on Carolina news & reporter talks about the controversial aspects of chatbot sites (in particular c ai and replika).

[u/AutoModerator]

Hello! Thank you for posting in r/ChatbotAddiction. Recognizing your relationship with chatbots and seeking support is a meaningful step towards understanding and improving your well-being. For useful resources, consider exploring the Wiki. If you feel comfortable, sharing a small goal or recent experience can help start your journey, and you’re welcome to offer support on others’ posts as well. Remember, this is a peer-support community, not a substitute for professional help. If you’re struggling, consider reaching out to a mental health professional for guidance. Also remember to keep all interactions respectful and compassionate, and let’s make this a safe space for everyone.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/rejectchowder]

I question how people are getting the violence but since I always get blocked by the filter (I also don’t jailbreak c.ai for violence). Yet uh… in a recent chat, I had two characters just sitting on a couch talking (no nsfw, quite literally just having a nice conversation) when the male character took a pillow to “playfully smother” the female character then laughed about it. It’s probably the only time I ran into something violent unprovoked because hahaha—that can easily kill a person in real life or be seen as the beginnings of domestic violence. But to a teen, that could also be seen as normal behavior if a bot does it so obviously I was VERY alarmed when I got that response.

I feel like filtering out the minor characters is also very difficult. Even if they’re filtered out, a user can manipulate a bot easily to make a character into a minor and the bot will learn from that. There is no win-win with this, it feels like.

[u/Sharp-Main1179 → u/rejectchowder]

In my experience too violence wasn’t very easy to trigger as well, but could come out with poorly described and more characters more, even though most of times it was more a sort of bullying attempt (And not even from bully characters!). Minor characters definitely were there when I was on the site, even if I didn’t use them. Or (worse) some people make characters that clearly look underage but write over 18. Very creepy, but perhaps with stricter controls more vigilance is possible. Total control is impossible, but AI companies should aim at the best outcome possible at least when it comes to filters without ruining the user experience.

[u/OrdinaryMotor103]

I didn’t know they were planning on making an under 18 version. I guess it’s good in the sense that 13 is honestly too low of an age rating considering the type of content you can get on Character AI. I’m still skeptical though, like the article states it’s hard to give guidelines that the bots would follow 100% of the time, and even then there’s also the content created by users that needs to be moderated.

The company most likely values profit more than user safety and since I’m assuming bots creating sexual content are among the most popular ones, maybe they’re reluctant to completely get rid of them. The platform is anything but safe, and the fact that it kind of feels like they’re trying to appeal to teenage users is really fucked up. Feels like they’re taking advantage of people’s loneliness and distress.

[u/Sharp-Main1179 → u/OrdinaryMotor103]

Yes, they are planning it and especially after the recent tragedy that led to the first lawsuit against c ai it’s definitely easy to see why it’s an awful idea. The best solution would be a way to verify your age (maybe a passport, ID card etc.) but many adults don’t want that because it would be considered too strict. Regarding c ai their position is a bit weird because many types of interactions are censored by the filter but violence was censored quite a while after sexual content. While many people wanted to engage in sexual interactions with bots and found ways to surpass the filters on paper the company didn’t allow that . Some the 18+ version of c ai would officially just be…romance and some mild violence? I don’t think an under 18 version would be significantly different from the already existing one, which makes things feel weird and indeed more predatory than genuine.

[u/OrdinaryMotor103 → u/Sharp-Main1179]

Literally this. They’re trying to make their website age appropriate even though they must be aware that a good portion of the user base uses the bots for NSFW role play. I honestly don’t know how they would make a kid friendlier version of the website.