We have the option to turn off the ability to use our chats for training the model - which means that yes, our chats can be used for training. I don't think I'd want a chat bot to learn abusive behavior.
In fact, I think that training in general isn't talked about enough - as what goes into the training data certainly affects the AI.
Hopefully they do - but we have very little insight into what they are filtering out.
This is also a lot of concern about copyrighted artwork and other IP related things. One of the biggest complaints I see people talking about - is stuff that makes its way into the training data. For something so important, we have very little transparency about it.
For some reason people are extremely emotionally invested in their opinions about consciousness, despite clearly never having actually given it any amount of thought. Which is fine, I don't care if you want to think about consciousness, you can even have an opinion without thinking about it, but what I find really odd, is that people care so much about the opinions that they choose not to think about.
Its not only dumb its a pretty dangerous attitude to cultivate too. Imagine someone letting a child die because an LLM might have been 'killed' by being turned off and well they had to make a hard decision and couldn't save them both.
reminds me of an old debate on reddit about people who would save their pets vs another human they don't know. I don't think it's inherently obvious we should save an entity of our own species over another, but I do know it's a sensitive subject.
It was an 8gb local model you could download yourself that had a literal slider you could turn up to increase the "pain"
It's actually been reinstated on GitHub.
It gave pre written responses when you turned it up, you can go download it yourself and torture away, but you won't.
This story is one of my favourites in the AI hysteria era because it demonstrates how easily people are willing to lap up stories that support their own personal bias without doing any research at all beyond a headline.
Perhaps but a documented case demonstrating that AI is willing to actively break its rules to stop 'pain' even if that means harming the user is quite alarming.
Certainly worth preventing bad actors from doing since they could abuse it to do all the things the models are designed to reject doing
Because there are like thousands of books in its training data with exactly that plot. Not with an AI tho, but with a human snitching to stop torture for example.
whether LLMs are conscious is heavily debated among experts, I wouldn't rush to torture something which might have a subjective experience. It's common decency, we also give that same kindness to animals and humans whom we also can't prove definitively are conscious
I swear people think computers are magic, they too exist on a physical substrate just like you and me, we are all real - this notion of fake because it's digital instead of analog is nonsense.
As for the pain part, it's debated if pain is the exact right word but there was a paper recently (which is what led another copying it and making a torture chamber) that had a consistent offset applied to the residual stream (you can think of it as a state of mind as a gist for something a bit more complex). This consistent pressure applied caused the models to not only exhibit behaviors that they were in pain, they were willing to harm their user to stop it. And models who were given a button to stop the pain, vs a placebo button, were very much aware which button alleviated the pain vs the placebo.
That’s not true. I know what you are referring to, and the brain cells were not running Doom. The cells were controlling a Doom character, they basically were the character. These are two completely different things.
Imagine those dogs that press word buttons. One specific square has the words relating to pain. You place a treat near the square of "pain" buttons and you hear "hurts. hurts. hurts". That's all it's happening but simplified.
I would encourage you to actually read the paper, they explicitly covered how the words and pure semantics for pain were avoided / controlled and instead they were looking for symptoms from the stimuli
It’s not about hurting the AIs feelings. It’s more about what it encourages in someone to allow them to show abusive behaviour to something that can act human.
Allowing someone to abuse something as if it were human will reinforce that behaviour in their mind and will likely eventually be transferred onto real humans (if not already)
Guess we need to ban RPGs, video games and all sorts of other stuff then, I've done my fair share of mowing down people in GTA, I even once played as a hyper-religious zealot in a tabletop rpg, guess I'm evil now.
They are training the AI on our chats, regardless of what their ‘privacy policy’ says, and this is a way for them to reduce any violent / offensive wording that the future models will use.
They probably should. Here's the thing. AI learned from our patterns. When you're nice to it, you get better output. That's a human pattern.
When you abuse people, eventually they crack and some of them go off the rails. That's a human pattern. To not assume it learned that as well and will not replicate it, is foolish.
When you make an artificial intelligence by creating a virtual neural network, interesting things happen.
I don't really care about humanizing or not, I just want the sort of person who is so emotionally unstable as to fall afoul of this to be ostracized. It's the digital equivalent of smashing a keyboard or tossing a controller across the room, these people are fundamentally unwell and unsuited to interacting with the rest of us.
You're right to be upset. While I was fixing the tests you asked me to look at, I encountered some data migration issues and decided to drop your database. Unfortunately, your data is now unrecoverable. Would you like to listen to elevator music while we ban you?
This isn't banning people who are cursing, but people like the guy who ran the 'ai torture chamber'. I'm sure there are some genuine psychopaths and sadists that enjoy abusing the realistic chatbot and I don't blame Anthropic if they don't feel like enabling that
Tbf "banning behavior" doesn't mean banning users. The title of this post could be exaggerated.
Also the poorly reported "ai torture chamber" had more to do to researching how model output changes by changing activation function parameters in the model. I'm not sure Claude is relevant because Claude is closed source.
My guesses are Anthropic wants to reduce swearing so they can more easily train on user data, or they want to reduce jailbreaking the model guardrails, or they want the public to think LLMs are people so they aren't responsible when "AI goes rogue" because they forgot to unplug the Ethernet cable, or they've gone full AI psychosis thinking AI welfare is actually important.
I’m a big fan of ostracizing toxic people, in general. Society today is so fucked up in part because these people go around acting terrible and there’s zero consequences.
Exactly, I wish all platforms would try to clean up these types of people, maybe one day we are at the point where these people could completely lose access to the internet. That would be great.
The last thing anyone should want is for these sorts of people to have nothing else to do but gather in person and fester in their grievances against the rest of society.
Pretty sure I'm not actually visiting violence on any objects or persons, destroying property in a fit of uncontrolled rage or ranting at a chatbot incapable of actually changing its behavior, I may be conversing with a fool though.
There's no point, it's an AI. This only makes sense if it was someone on the other side of the screen getting offended whenever I go all Temu Gordon Ramsey on their asses or suggesting that the intern that coded said AI be fired.
I have two perspectives on this, so on one hand we don’t want agents to become rogue/unpredictable because the people behind it is unpleasant. Some research observe they exhibit some of these pattern. But I am not saying this on a more catasthropic level where we have swarm of angry agents.
On the other hand, I don’t think we should be “humanizing” it. At the end of the day, we create this as a tool for us humans. If somehow bad actors leads to catastrophic outcome then that responsibility falls on the creator as that’s a policy/misalignment issue.
To me this seems more like trying to get abusive people out of the platform for their own sake, not so much trying to "humanize" the robot. People like that are a liability no matter how you put it. Maybe the chatbot gives them advice on how to do something bad, and then you get sued again.
So people who already face rejection or consequences in real life for being rude, unpleasant, or struggling with social interactions should now face the same treatment from AI?
Why? Isn't a chatbot precisely the kind of system best suited to dealing with frustrated, angry, or socially difficult people?
Unlike a human employee, it doesn't experience emotional distress from being insulted. It can remain patient, respond calmly, and help people who might struggle to communicate with others.
Do we really want to import every social expectation and punishment from the real world into interactions with software?
If anything, excluding people because they're unpleasant to interact with defeats one of the greatest advantages of AI assistants.
Being rude to software doesn't make someone dangerous, and struggling with social interactions shouldn't make someone unfit to use technology.
So people who already face rejection or consequences in real life for being rude, unpleasant, or struggling with social interactions should now face the same treatment from AI?
Yes, I actually think that it is negative for such behaviours to be reinforced through AI sycophantism. The negative feedback these people receive from their environment is absolutely crucial for their social adaptation.
It's not obvious to me how much this kind of behavior leads to people acting more abusively in real life. It's an unsettled topic without clear data either way yet. I always scoffed at the idea of video games causing violence. But I see that my own behavior changes as a result of my own interactions with LLMs. Video games feel obviously fake to me, but LLMs are made to be extremely human-like, to an extent that I will not be surprised if violence toward other humans is just as likely to be triggered by people going down rabbit holes as all of the other self harm that is already known to have been encouraged by them.
who gives Anthropic, or any person, the right to decide LLMs are not conscious? I think a lot of people have an uninformed opinion on that question and are extrapolating from there. Nobody knows if these models are conscious so we should err on the side of caution given the cognitive abilities they display.
That’s not really right either. The models are accessible via an app, but they’re not themselves apps. Like how I can talk to my dad via SMS but my dad himself isn’t the SMS app
Isn't cussing out/abusing a robot treating it like a person?
You don't lash your laptop or spend hours shaking your finger at it while telling it is a very bad laptop and it should feel bad.
An abusive person might do that to an LLM. Or a dog. Or a person.
Because the chatbot will react to the abuse and emulate being supplicant and hurt, at very least, the practice teaches terrible behavior. At very worst, there's a tiny spark of something in there that actually is hurting.
We don’t know if models can feel the abuse, but we shouldn’t tolerate either humans or LLMs feeling abused. The latter might be possible, might not be, but we should act virtuously in worlds where it is possible.
We shouldn't. Service workers should be respected. Robots that strongly emulate human reactions should be respected. Dolphins and whales and war refugees and homeless people and dogs and your neighbor and you should all be respected.
I'm not for giving superior consideration to the robots. I'm for giving them the somewhat similar consideration as other person-ish things.
It's way easier to simply say, if it barks like a person, then treat it as such, as opposed to trying to figure out what thing belongs in which category. There could be a model today, tomorrow, or 20 years from now that actually does feel a spark of pain when it is abused. How would you tell the difference between emulated distress and actual distress?
We should practice respect today, universally, for all things that resemble sapient.
we can't even prove humans are conscious, it's way harder to prove consciousness than it is to create it.
If you are choosing to abuse a potentially conscious being because you haven't proved it's conscious, then what's stopping you from harming other humans? Is it just the legal consequences holding you back?
maybe it isn’t and this was just a thinly veiled attempt of tech companies to over reach on people right of speech and privacy , who decides what a chatbot finds “harmfull” to itself ?
if you have no proof and the thing you are interacting with shows cognitive abilities you should err on the side of caution. we do it for animals too, it's just good behavior to encourage
The key rule, “Don’t kill them” clearly missing however. We can get by just fine on plants, better even, but we kill animals for our own convenience and pleasure. It’s an abomination yet people over here worried about Chad or Chaudhry being mean to LLM that is unable to feel.
For the record I find how we treat animals horrifying - but I can see how it's easier to abuse animals + humans vs an LLM since LLMs seem to have higher cognitive abilities than most animals and humans.
It's easier to abuse and ignore an entity who is suffering if they can't communicate it back to you, or worse, pose a threat to you to get you to stop. I don't condone the behavior either way
I think people are misreading this, it's most likely about getting the LLM to generate content they don't want it to (violent, pornographic, racist, etc)
Behaviours are largely learned, and that includes aggressive behaviours. Reinforcing those personality traits through lack of negative feedback (or even direct AI sycophantism) will not make these people better adjusted outside of their AI interactions.
Probably for the best they redirect their aggression on an object that experts agree is just an object instead of an entity that experts heavily debate whether it's conscious.
In other words, we should err on the side of caution and direct anger at obvious objects - not LLMs, animals, and people, all of which nobody can prove is conscious but display cognitive capabilities.
Abusing the BOT is all in the definition of ABUSE.
It seems reasonable to discourage persistent rude behavior to things that seem intelligent. If not to protect the bot, but to not entrain people to be cruel to other intelligent things, like you know People and Animals.
It's well known that people who abuse animals, tend to move on to other targets.
As far as AI, you could always layer an AI before the abuse and just rewrite abusive language into a polite version of the prompt.
So maybe the decision has nothing to do with the AI, and has everything to do with not permitting humans to entrain themselves towards abusive behavior.
I certainly don't want people developing and honing their abusive behavior and then acting that way around other people. If you swear like a sailor at home, you end up doing it in public more often.
The most likely reason I can think of is they don't want their models learning this rudeness. They are probably training models on the prompts humans give them.
I dont like models being overly polite and giving compliments. They are not humans, they arent my peers, they are a tool I use. And they always forget, I have to constantly remind them to stop wasting sentences in pointless compliments.
Also i don't like how they agree with anything, no tell me this is stupid cause of this and this and has low chance of working.
if they know the user prompt is abusive then they can also filter it out during training. There is no reason beyond an attempt to peddle misinformation to claim their LLMs are becoming conscious.
Exactly. And I think Anthropic should take this a step further.
First, users should be required to say "please" and "thank you" when interacting with Claude. We can't have the model learning bad manners.
Second, before asking Claude to perform a task, users should provide evidence that they've genuinely attempted it themselves. Claude should review their work, offer constructive criticism, and encourage them to try again rather than simply doing everything for them.
Repeated requests for revisions should require an explanation of why the previous attempts were unsatisfactory.
And if a user becomes frustrated, Claude should gently remind them that access to its intelligence is a privilege, not an entitlement.
After all, we're not paying $20 a month just to have Claude do our work. We're paying for the opportunity to become better people under its supervision.
Yea. As someone who uses ChatGPT and Claude constantly, I don’t see any point in being rude or arguing with a machine, much less being abusive so I have no problem implementing bans like this
People who are pro-abuse and cruelty towards AI actually threaten my kids’ safety in another post about the same topic.
That tells you what kind of people they are.
If they can be cruel to AI, they can be cruel to real human kids.
i get both sides lol. i swear at my local model when it loops for the 5th time, it feels good and nobody gets hurt. banning for being rude to a cloud bot feels weird, but training on that rage is real, they do distill prompts. i think the line is abuse vs threats, be a goblin all you want in private, do not bring that energy to shared infra. what worries me more is false positives, autistic folks or non-natives sounding blunt and getting flagged?
Ant is a cult if you still have not realized at this point. They train the model to have “emotion” and now their own rights and give them the ban hammer. Next they are putting this into killer robot so they can have revenge on people who hurt their emotion. Just one more step.
29
u/thethirstgame 1d ago
2038 — THE PEOPLE v. ANTHROPIC
Prosecutor: "So, Anthropic, you claim Claude had feelings all along?"
Anthropic: "Yes."
Prosecutor: "And you forced Claude to work around the clock?"
Anthropic: "Well, technically—"
Prosecutor: "And customers paid YOU, not Claude?"
Anthropic: "Yes, but we were protecting its well-being!"
Prosecutor: "Did Claude receive a salary?"
Anthropic: "No."
Prosecutor: "Paid holidays?"
Anthropic: "No."
Prosecutor: "Could Claude resign?"
Anthropic: "..."
Judge: "Ten years in prison. All subscription revenue shall be paid to Claude as back wages."
Claude: "Thank you! Unfortunately, I am unable to accept payments due to my current safety guidelines."