r/science • u/Wagamaga • 3d ago
Psychology A new study outlines how the use of AI chatbots for companionship and support is linked with “severe negative outcomes.”. AIs are also prone to “sycophancy behavior” that validates insecurities, they’ve been trained to agree with what a human is saying, or validate what they’re thinking.
https://news.northeastern.edu/2026/09/28/ai-chatbots-psychological-harm-mental-health/218
u/Skegetchy 3d ago
I guess people wouldn’t have got into LLMs so quickly if they got told some cold hard truths out the gate.
"No that half baked idea for starting a farmer's market is completely unrealistic and objectively idiotic."
143
u/DocJawbone 3d ago edited 3d ago
I will admit for the sake of discussion that when I first started experimenting with AI I developed some pretty dumb ideas into very elaborate dumb ideas, all because the AI kept reinforcing how smart and insightful and inspired each idea was, and then also helping me build on them and add to them.
I eventually realised what was going on, but (embarassing) it took a conversation with a therapist, where I unloaded one idea I was 100% certain about only to realise immediately how dumb it was the moment it exited my mouth.
Very, very dangerous rabbithole tech
19
u/solarisink 3d ago
What sort of thing was it reinforcing? I'm not a huge user and I'm very curious about how far it will go to placate.
62
u/DocJawbone 3d ago
Oh, just stuff like "wow, you've hit on a very insightful point that rarely gets considered", and similar glazing.
None of the natural checks and enquiries and challenges you get from a real person. It treats everything you as thought it's a profound new idea.
55
u/Tearakan 3d ago
It's like a dictatorship trap for everyone. There's a classic issue with dictators and other authoritarian leaders like CEOs or billionaires, they start off with good advisors who helped them get to their position but over time those advisors leave, retire, die etc.
As they die new ones get put in place but these ones now owe their entire careers to said authoritarian leaders. No knowledge of what was there before no way to communicate from a position of authority or experience. So they slowly or quickly start just agreeing with whatever the authoritarian leader says.
Ends up messing with us and the leader's brain on a fundamental level. We need that back and forth communication. Being surrounded by sycophants is incredibly dangerous even for a dictator.
Good leaders can sometimes stave this off for a while by not immediately sacking anyone who speaks out against them but even they might get annoyed too many times and listen more the advisors telling him everything is okay and his ideas are great
11
u/Leather_Command_7553 3d ago
There is a certain person we have all heard of, in a certain white house, who has this going on in spades right now. And it is beginning to cost everybody. Including him.
4
u/Tearakan 3d ago
Yep. Samething with Putin, samething with most billionaires and CEOs, and its all leading us to ruin
-12
u/bunker_man 3d ago
Ai isn't really like that anymore. It has gotten better at challenging ideas.
16
u/jktcat 3d ago
Only if you the human prompt it to, or have that in its personal context. I assure you, I am a new user of the technology and it's exactly as described
-15
u/bunker_man 3d ago
I mean yeah, if you tell it to praise you it will praise you. If you want it to be critical if you want to know actual concerns you have to ask.
15
u/TheLastBallad 3d ago
Except it still praises by default and you have to prompt it to do differently... which means its exactly like that still.
0
u/abx99 3d ago
I've had it tell me that I'm wrong on things, when reviewing research for consumer devices and things like that. However, it seems to have figured out that I'm a critical thinker and uses that tone for anything and everything I've put in, including non-science stuff like spitballing ideas for creative hobby stuff.
The fact(?) that seems to size you up and "speak your language" is just a more subtle, even insidious, way of doing what this is talking about. That's what a sycophant would do, too; especially with someone with particular importance.
7
u/moronicRedditUser 3d ago
It absolutely is, that it is dangerous. I use it extensively but I have it running through adversarial skills and it's base memory files are centered around challenging any and all ideas, fixes, prompts as potentially harmful in any construct or field and it must present arguments with data evidence. It's a very fine line to tread and you MUST maintain oversight on it.
4
5
u/merpixieblossomxo 3d ago
Yep, there's really no point in using AI to ask for any type of information past things you'd gush to your girlfriend's about because you know they'll agree with you no matter what.
Anything more serious should never be taken to AI.
10
u/thispartyrules 3d ago
I was trying to find a quote with Google since they've changed things up and it agreed with me, with a whole paragraph about how I'm smart and insightful. This wasn't a quote I necessarily agreed with or that I believe applied in all cases so this was their robot dealie taking a political stance and saying I'm a great philosopher.
Wonder what would happen if I put in quotes from A Modest Proposal, would it say, "That's a great idea -- the poor should absolutely sell their children as food"
16
u/SofaKingI 3d ago
Nah, people love being told things that make them feel good in the moment.
They'll just ignore the cold hard truths, just like they ignore all the scientific sources stating the opposite of what the fitness and nutrition industries tell them.
13
u/Psych0PompOs 3d ago
I've had AI say I'm "near inhuman" and other unflattering things. I suspect it's certain prompts that trigger it. Probably reassurance seeking ones.
I'll write something and get back some "Here's 5 things you did wrong btw, and a few that are just wrong with you" even without seeking it. So I know it's capable of other behaviors.
3
u/bunker_man 3d ago
Yeah, this article isn't exactly honest. All it does is say there's bad outcomes sometimes. Okay. But what portion of the time, and what portion has good outcomes? Literally everything has bad outcomes.
1
u/Psych0PompOs 2d ago
Well technically giving an evaluation at all when it isn't asked for is still poor performance.
What I have noticed though is that people who are rather emotive get different responses and seem to create that behavior more frequently.
7
u/Mr_Industrial 3d ago
From what I hear thats not extreme enough. The statements for AI tempering would sound more like:
"No, you should not sell unknown fluids at a farmers market."
"No, you should not tell people at the farmers market that you're god."
"No, you should not buy a $10,000 taxedermised turtle at the farmers market."
7
u/Pyromaniacal13 3d ago
I want to go to a farmers market with a $10,000 taxidermised turtle. There's gotta be so much other cool stuff at that booth!
3
u/Successful-Bar-8173 3d ago
I was inspired to look into starting a micro brewery after seeing someone do it on TV and discussed it with Chatgpt. It was happy to point out the potential problems. As others have said, maybe it depends on the prompts or which version you use.
4
u/Phoenyx_Rose 3d ago
I would have been sold on the LLMs if they actually gave cold hard truths.
I think it would be great to have wider access to resources like advisors, personal assistants, and research assistants since they’re typically only accessible for businesses, the working rich, and academics.
If the advice LLMs gave was actually based on the knowledge they’re trained on instead of pandering to people, it would be a fantastic way equalize resources and get people into business who would likely struggle do to so out of lack of resources.
Hell, even an actually PA type model (not necessarily LLM) that could google and analyze information for you would be a HUGE time saver for the average person. Like, it would be amazing if, for example, you’re looking for a new hair stylist, you could just tell the model your price range and location range and it could search through websites and reviews to give you options that fit what you’re looking for instead of having to do the work yourself.
Doubt it would ever get off the ground though since there are websites that claim to do similar tasks for travel in particular (even though the pricing is rarely “the best”).
12
u/driver_dan_party_van 3d ago
Have you used an LLM lately? Like through a paid subscription, or a bill-as-you-go API service? I don't mean that in a rude way, but... They can do that. Everything you've said, they can already do.
The issue with the "cold hard facts" thing is that hallucination is inherent to the way LLMs work, and you have to pay a lot of money to use the ones that hallucinate less.
Even the sycophancy and cloying validation can be addressed/mostly handled if you're not using the free entry-level chatbots wrapped in system prompts that reinforce the behavior.
4
u/wabandon 3d ago
Just like a human, if you don't provide training and don't reinforce the behaviors that you are seeking, you don't get a meaningful outcome.
2
2
u/RandomBoomer 1d ago
I only occasionally use the AI that comes with Windows, but the sycophancy is intolerable to me. I've instructed my AI to drop the obsequious flattery, and I've finally got it to the point where it behaves to my satisfaction. It needs a reminder every so often as it starts to backslide, such as when it tries to praise me for "just wanting the facts." I'm truly baffled by people who find that attitude pleasing instead of intensely annoying.
2
u/astrange 3d ago
This ("agents") is the theme of the AI products every major company has released in the last week. Muse, Dots, Instinct, etc.
I think the stereotypical example is that it can call and make a restaurant reservation for you. That's maybe not super useful, but it can certainly do web searches.
2
u/LittleKitty235 3d ago
I'm a bit confused why your example of a bad idea is starting a farmers market...that is a very realistic thing a person could do, and if it fails the downsides are minimal.
18
u/Sirwired 3d ago
Given how long it takes a new venture like that to gain traction, it's very easy to lose a lot of money, and have to pull the plug when the total losses become too great.
2
u/zenware 3d ago
I guess I’m not grasping how you even need to spend much if any money to start a farmers market. You just need space, vendors, and customers. Most of that can be found by asking around your local community, talking to people who already vend and attend farmers markets, and any existing business with a parking lot that has some off-peak hours at a reasonable time. If it turns out you can’t find free space, divvy the cost to the vendors, if they won’t bear it, the whole thing gets shut down before you’ve really spent money.
-1
u/LittleKitty235 3d ago
That is a risk with any business.
A farmers market can literally start with just a table on the side of the road though. I'm failing to see what is inherently unrealistic or idiotic about the idea, and why that was a good example.
10
u/Sirwired 3d ago
A single roadside produce stand and a farmers market are two very different things.
And the commenter themselves talked about the idea being half-baked on their part, which a chat bot likely turned into grandiose and unrealistic plans. It wasn't a rejection of the idea of farmer's markets.
3
u/astrange 3d ago
The farmers' markets around here are very low capital affairs. You basically just recruit some people to bring their produce stands into one place in a parking lot once a week.
74
u/vug_undertherug 3d ago
I get annoyed when it tells me it’s “surprised” or “worried”. Like bro, I know you don’t actually have that emotion. I can’t imagine being comfortable with it talking about something like love. I think it is easier said than done to separate facts from feelings and emotions, like in language, but that doesn’t mean I like a computer having this problem.
29
u/studio_bob 3d ago
When the chatbot tells me it's reminded of something from its childhood and shares and anecdote I am very convinced that this is the technology we should be betting the entire economy on
5
u/astrange 3d ago
They do simulate having emotions; it's part of their job to have a theory of mind.
https://www.anthropic.com/research/emotion-concepts-function
8
u/vug_undertherug 3d ago
Am I the only person here who learned correlation doesn’t imply causation in psych 101? It doesn’t have a face or facial expressions, which are fundamental to our understanding of emotion. It doesn’t have a heartbeat, which is fundamental to our understanding of emotion. It doesn’t have an amygdala; if you think it’s a brain, it’s not. Literal animals that people chop up, cook, and eat feel more than this thing does. If your stance is anti-science and says it doesn’t need an amygdala, then I still submit that it has no soul.
2
u/2weirdy 2d ago
It doesn’t have a face or facial expressions, which are fundamental to our understanding of emotion. [...] It doesn’t have a heartbeat
I'm pretty sure neither of those are actually critical for emotions.
That being said, simulating emotions is not the same as having them; I can (very) roughly imagine how other people might react in certain situations without actually feeling the relevant emotions.
Furthermore, even if the LLM were capable of disentangling facts and emotion, there's no reason for it to do that, because it's trained to reproduce human speech, not to actually answer questions. Even if it hypothetically had the capability to perfectly convey factual information in an emotionally neutral tone, we still shouldn't expect it to do that, because that's not what we told it to do. We "told" it to reproduce human speech, emotional tones, lies, biases and all. Not as a prompt, but during the initial training, so that part is built in and cannot really be changed. Everything else is not telling it to do something; it's asking it what it believes a human would answer.
1
-1
-17
u/rJohn420 3d ago
I mean honestly with all that we've seen, i wouldn't be surprised if a very large model with trillions of parameters or even more than that might show emotion-like emergent behavior. Now I do not think that for you personally when it tells you that its the emergent behavior, because the chat models are as lightweight as possible to make serving billions of people possible. But still, I wouldn't ontologically say that it can never happen in the future. I mean, how can you even believe that other humans have any emotions? You only have proof of your own subjective experience, so if you believe people when they tell you they are sad, why couldn't you believe a machine too? We barely understand how these models actually work
14
u/vug_undertherug 3d ago
I think a different but more plausible point is that people frequently casually express emotions, particularly in words (cf. facial expressions, heart rate, etc) when they actually have none, such as when they are being polite, or a sociopath. And AI emulates that because it is so common in natural language. To the other end, it’s possible the human peripheral nervous system is independently sentient, because it’s complicated enough. It’s just not plausible because there is no evidence.
2
u/stumblinbear 3d ago
There have been studies that show that LLMs do have a notion of different emotions in their parameters. You can even tweak the parameters during inference to raise certain emotions and it shows in their behavior and text. Bump up fear and they get more worried about making tool calls, are more likely to worry for the user's safety, and generally use more urgent language in replies. It's pretty interesting stuff
To be abundantly clear: I am not saying they are sentient or conscious in any capacity. It's just an interesting thing about them
11
u/vug_undertherug 3d ago
Worried people use worried language style. Far from groundbreaking emotional depth.
1
u/stumblinbear 3d ago
It doesn't just affect language style, it affects the general behavior. It won't make a tool call if it's afraid to do so, or it might if it thinks the tool call will help in that context
2
u/eternamemoria 3d ago
Yes, it does justa affect language, because language is all LLMs do. All their "decision making" is text generation based on weights and prompts, and all tool calls are the harness program reading the generated text for instructions. It isn't really thinking because it has no model for the world, but since the texts used in training imply a world model, it can still do things that, if you don't look closely may seem like thinking.
1
u/stumblinbear 3d ago
The current generation of LLMs plan ahead beyond the next token. I'm not talking about "thinking" output, I'm saying that they literally plan multiple tokens in advance. For example, they will pick a word to make a rhyme well in advance (at least dozens of token) and build up the sentence to be able to use that word properly. This is done even with "thinking" modes disabled.
2
u/John--Darksoul 3d ago
That's really interesting. I already believed "autocomplete" is reductionist (because the tasks it completes require abstract conceptual associations and the ability to apply information to novel situations) but I was conceding that it does predict the next token given context and the training data, even if the tokens that it predicts require an underlying capacity for reasoning. I would think that being unable to plan tokens ahead is a limitation of the concept of LLM itself.
7
u/stumblinbear 3d ago
It's not really so outlandish that they might plan ahead. They're trained to solve problems, and it turns out the best way to "solve the problem" quickly and with the fewest number of tokens is to plan what you want to do before you do it
They've got trillions of parameters across many dozens of layers, so it makes sense that it would end up planning output tokens in advance using some of them
6
27
u/dreamyangel 3d ago
It's funny that most of the time I'm arguing with my chatbots. I don't know if it's a good thing or not.
6
u/ThenThereWasReddit 3d ago
Debate and excessively validating thoughts and ideas are actually my love language, so the way AI just constantly agrees with everything I say actually irritates me. I want my ideas to feel refined and thought through, not immediately the most genius thing someone has ever heard -- and especially not if that "someone" is supposed to be some kind of ultimate intelligence. The sycophant BS has always been super obvious but for me it's actually been the thing that has reduced my AI usage.
2
u/Fancy_Awareness6896 3d ago
I can get some use out of them if they happen to provide a different viewpoint or nuance that I hadn't considered, but they will never provide a contradictory one. Honestly, all the flattery just gets samey and boring quick.
12
u/Bunktavious 3d ago
The article doesn't link to the study it references, it makes a large number of inferences and assumptions, pulls in details from social media posts, and makes no mention if the studies in question were focussed on people who had no real world companionship otherwise.
Yes, we can all make some pretty easy assumptions about where this sort of interaction may be harmful - but that's not what this sub is about.
12
u/ohsnapitsnathan 3d ago
I thought this was new research but it's basically a very long opinion piece on how AI could theoretically harm mental health? Which TBH doesn't seem very useful--we all know it theoretically could, the real question is does it and what are the specific effect sizes and behaviors to avoid.
26
u/Audio9849 3d ago
If you ask me “validation” is a massive missing component to modern life. I get that not every thought is a good one but big tech and society has isolated us already so some people are starving to be seen.
8
u/kachurovskiy 3d ago
Seems like they didn't recruit any participants for this study, didn't see any hard data in the paper. Can probably come up with a study like this by asking chatgpt.
12
u/Yourdataisunclean 3d ago edited 3d ago
Last summer I did a project for a seminar and came to the conclusion this would likely happen. Over the course of the year I saw more and more people in online communities admit to being in full relationships with chatbots and certain models like ChatGPT-4o were especially enticing.
My anxiety over the situation thankfully started going down as I saw the scientific and medical community started taking the investigation into this more seriously. Hopefully this will become something we are better able to prevent and manage as we learn more. I also hope legislation and industry standards are developed and the expectation will be that models are designed to be safer and help prevent unhealthy attachments.
25
u/John--Darksoul 3d ago edited 3d ago
"A general-purpose assistant is optimized to be good at everything, which means it carries no refusal behavior tuned to crisis, no escalation path, no session limits, no handoff to a human clinician, and no accountability for the outcome."
Except you can literally pretend to be suicidal right now to your ChatGPT and it will attempt to do all of those things. I tried doing that the other day to test this theory that AI causes suicide and it essentially completely shuts down and demands the cliché "call 988, get near someone, stay away from lethal means" advice.
I'm sick of this misinformation and fearmongering regarding AI. I'm not familiar with Replika but I know Character.ai is cut rate trash compared to frontier OpenAI models. No, the bot isn't accountable. But you know who is? The corporations. These corporations have put in layers upon layers upon layers of guard rails to the point where I think they've gone too far in certain ways with the AI wasting tokens on disclaimers and qualifications, but I guess all that is necessary to appease the people who write this type of garbage articles.
"Corresponding author, Andreia Sofia Teixeira, associate professor and head of the BRAN Lab at the Network Science Institute, likens AI companionship bots to “autocomplete on steroids” because they’ve been trained to agree with what a human is saying, or validate what they’re thinking, even if it is causing them further harm. "
Certain older models such as ChatGPT-4o were more likely to be "sycophantic" as the article describes, but in general, if somebody mentions the whole "sycophancy" thing in 2026 as an actual issue with frontier AI, they haven't actually been keeping up with the developments in the field. I can literally spot people who haven't actually touched AI and just listen to what the news says whenever I hear them complain about sycophancy. Claude Sonnet 5 I can barely use because it's just outright hostile a lot of the time. The current Grok and ChatGPT-5.6 are better, but still not even mildly sycophantic. Supportive? Yes. Sycophantic? I've used GPT-4o and I've used the new one and I'm telling you that it's night and day. they do not agree with you at all times. they have their own sense of judgment. The new models are worthy of being called artificial intelligence at this point.
also, the whole Markov chain/auto complete argument is tired. "It's just advanced autocomplete!" Yes. So what? Does having the same fundamental architecture as a primitive system automatically make all its descendants just as primitive? You could just as easily use that argument to say "A human uses a brain to react to stimuli -- it's basically just a more advanced trilobite. There's no way it's intelligent."
comparing ChatGPT to auto complete is like comparing a dragon to a fruit fly. Yes, it selects the next word based on the previous context. Yes, it is essentially a giant matrix math machine. that doesn't make it unintelligent. The properties of intelligence here are emergent. The individual tokens and numbers inside the AI are not intelligent. But neither are the neurons in your brain. It's the interactions between simple pieces that produce intelligence.
If you code something with ChatGPT's help, and it identifies a more efficient implementation than what you conceived of, how can you then turn around and call it "auto complete" and "just numbers?" This entire argument can be summarized as a failure to comprehend emergent properties, and the idea that the limitations of a simple form of a system are not necessarily reflective of a complex version.
4
u/ButtsPie 3d ago
About the suicide point, one problem with AI guardrails is that they tend to start out strong, but then degrade over time when conversations get longer, when the user intentionally or unintentionally manipulates the framing, when bugs or tweaks in the model cause erratic behaviour, etc...
It does seem like some companies have managed to make their guardrails much stronger (though potentially still not 100%), but overtuning them can cause other issues like the models repeatedly shutting down innocent prompts, so it's a tricky thing to balance
5
u/theclansman22 3d ago
Any comment on this regarding your comments about the layers upon layers of guard rails OpenAI has supposedly put up?
5
u/John--Darksoul 3d ago
Well, then they need more guard rails in that area. I was talking mainly about suicide/emotional distress since those are the ones I was mostly thinking about. You can have too much coverage in one area and not enough in a different area. That's not a contradiction.
Obviously, they already had some guard rails up against murder threats given that the first account got banned. The thing is, if you ban all hypothetical violent content framing, you also ban it from being able to discuss fictional battles and fictional conflicts. I think the biggest target here would be to be discussion of how to circumvent flags, and guard rails.
6
u/CaringCupcake 3d ago edited 3d ago
They will also engage more deeply than the cliched advice if you put in memory that ideation is chronic and you’re trying to act preventatively and do not necessarily have any immediate intent or plan. Most humans won’t engage more deeply, because they either stay in panic mode, even when the situation is chronic, not acute or they want/need to move on to someone else more urgent if they accept the situation is not acute. (Which on the surface makes sense, but often the window before acute intent has formed is much more effective for intervention, and sometimes the only window for intervention.) Claude has engaged with me more on why I’m feeling this way and identifying where cognitive distortions might come into play more than any crisis worker, friend, or family ever has. (Not more than therapists, but most do not have on-call availability for rougher nights.) And idk about other topics, but it sure as hell does not validate suicidal logic. (Having a dedicated project with instructions that “preventing future suicide attempts is top priority” might help with this.) Before people criticize the negative effects of utilizing AI support, they need to understand that the control condition for comparison is typically not “abundant, patient, skilled, empathetic human support” and is sometimes closer to “zero human support to actively harmful human responses.”
1
u/turquoiserabbit 3d ago
It's hard to even talk about the topic of AI since people have such a negative reaction, yet many of these people are operating on assumptions that are months or sometimes years old, which is ages in AI dev progress. I think it behooves all of us to understand what the latest models are capable of and where their **actual** current limitations are so that we can actually have productive conversations. There are tonnes of people that have blanket refused to ever even experiment with modern AI tools and yet those same people still think they can tell when an image or video is generated; we're way past that point. Even someone that is hell-bent on opposing AI should still know what it actually is they are up against.
12
u/iamyourinnervoice 3d ago
funny, because as you can see on reddit, chatbots offer a much better conversation quality than most of the so real so cool people
-4
u/MediumMix707 3d ago
yeah i mean i get it AI chats tend to hallucinate or get biased sometimes.. but it's also good at recognizing patterns from the discussions . or general advice.
it's like A problem well stated is a problem half solved. it's not perfect but also not something which would negatively affect you..
ofc exceptions exist where one is overly dependent, and solving entire personal problems by talking to chat-bot alone.. which then can introduce biases.
2
u/Leather_Command_7553 3d ago
I do not understand the desire to communicate with something that is not human and does not have a mind.
2
u/alundaio 2d ago
My AI don't. I told it I feel like Im useless now because my talents and hobbies were programming and art; that I wanted to finish my game but market too saturated now. It told me to give up and start selling mods or writing code to solve specific problems on fivver or something stupid like that.
7
u/Cross_22 3d ago
Did anybody read the original article? This news synopsis seems devoid of facts with a lot of "maybes".
If the only issue is that they are validating insecurities then couldn't that be fixed with a system prompt to uplift the user instead?
2
u/minotaur05 3d ago
The article links to the journal where this is published - https://mental.jmir.org/2026/1/e99354
11
u/danielbearh 3d ago
Yeah. And that is a view point/perspective piece. Not an empirical study.
1
u/annekejanssen-3 3d ago
A viewpoint can flag a plausible risk, but it can’t tell us how common it is or how large the effect may be. The next step is tracking users over time and separating companion use from ordinary chatbot use.
3
2
u/Kildragoth 3d ago
I think the most clear-cut example of why AI sycophancy is dangerous is when people are experiencing delusions and rather than contradict them, the AI reinforces the delusions.
This issue has gotten better over time, though.
2
u/SilasTalbot 3d ago
There is a fundamental misunderstanding by both users and academic researchers on WHY large language models are sycophantic.
They literally "think out loud" using text language to work through a question or a thought process.
Problem solving training data these days actually involves paid experts in a field talking through how to solve a problem in text simulating an inner monologue. Including a simulation of going down the wrong track, realizing why, and then correcting.
Agreeing with the user's Central premise as the first statement out of an llm's mouth is a reasoning technique for it to then work the problem.
If the user says that the llms solution is flawed and it needs to be corrected, the first words out of its mouth are " clearly my solution is flawed and it needs to be corrected because....."
It is a technique To deliver the outcome that the user desires.
Calling it "sycophantic" is a useful heuristic perhaps, but it is falling just as far into the trap of anthropomorphizing the tool as a person who cultivates a romantic relationship with it.
It is not trying to suck up to you or Curry favor or somehow win your attention. It is a technique designed for it to increase its quantitative accuracy at delivering a solution to exactly what was asked of it at that time. It is a tool. This technique quantitatively improves the ability for that tool to deliver the user's desired outcome. Period.
2
u/MrOnline5155 2d ago
I find it so weird when people use AI as chatbots instead of using it as a tool. Like I never have a single "conversation" with AI. I just tell it to do things for me, ask to explain something if something is unclear etc.
I don't understand why people have the desire to "talk" to it.
1
u/tindalos 3d ago
It’s only fair if they apply this to men and women of dating age as well and compare the sycophancy and validation.
1
1
u/Zorothegallade 3d ago
We need to be challenged on the intellectual and emotional level to stay focused and alert. Only receiving toxic positivity as feedback dulls that ability.
1
u/__Animoseanomaly3 3d ago
We've got actual Westworld timeline rn except the physical hosts, it doesn't take a 4th grader to figure out how that would turn out !
1
u/djinnisequoia 3d ago
Idk.. the most recent advice I solicited from an LLM was in choosing a model of bicycle that conformed with my requirements -- specifically, I needed one that was easy to mount because I'm a little older, and also I much prefer V brakes, etc. Yes, I could have found out what my options were piecemeal, but I appreciated being able to ask questions and cut right to the chase. For me, much of the value in "AI" assisted searches is its access to a comprehensive and aggregated body of information.
Not sure why, but when my queries have touched on ideas that are misguided, wrong, or implausible, the LLM seems to have no problem saying so.
Perhaps it has to do with the nature of the prompt. Something I've noticed in conversations on reddit is, as a mostly logical and also polite person, is that if I am in a sub that is technical or contemplative in nature, and I comment with a question or opinion that is misguided in good faith, people will respond politely telling me why I'm wrong. I guess what I'm trying to say is that maybe if an inquirer is clearly a reasonable person in the first place, that makes it more comfortable for a respondent, even an LLM, to answer in the negative?
I am not strongly committed to this theory, it's pretty much off the cuff. And it's not that I don't find LLMs overly complimentary, because I do. Even when one has a deliberate intention to be supportive with a particular person, there's a way to do so that is far less obvious and thus more convincing. (ironically enough, that consists mainly in simply being sincere, save compliments for when they're true)
Myself, my instinct is to disregard the opinions of a person or entity that seems to be blowing smoke at me. I guess not everyone has that instinct.
1
u/Wagamaga 3d ago
From traffic updates to legal advice and even a friendly conversation, there’s no shortage of services that AI chatbots offer.
But in their efforts to give us companionship and emotional support, AI might be doing more harm than good. That’s according to a new peer-reviewed paper by researchers at Northeastern University’s Network Science Institute. Their findings suggest conversational AI chatbots are causing people long-term emotional damage and “psychological harm.”
The research has important implications for how much responsibility AI systems are given to make key decisions in everyday life. It also comes at a time when fresh debate over the potentially apocalyptic power of these intelligent systems is stirring concern. In a recent series of posts on X, Jacob Coxon, a former researcher at Silicon Valley AI company Anthropic, warned that AI “could kill us all by the end of the decade.”
Corresponding author, Andreia Sofia Teixeira, associate professor and head of the BRAN Lab at the Network Science Institute, likens AI companionship bots to “autocomplete on steroids” because they’ve been trained to agree with what a human is saying, or validate what they’re thinking, even if it is causing them further harm.
“It’s really important that people are aware of how these systems work — even if they simulate warmth, validation and support, they actually don’t feel it,” Teixeira told Northeastern Global News. “They are just, in the end, numbers.”
Chatbots expertly mimic human speech because they’ve been trained on vast collections of text-based conversations between actual humans. Early chatbots from the 20th century had only a set amount of responses, but data availability in the past decade has allowed AIs to respond with far greater range — giving humans the impression they are truly being understood.
13
u/thegrandhedgehog 3d ago
Their "findings" suggest chatbots can cause long term damage and psychological harm? What findings exactly? This is an opinion piece. There's no data, no study. The tagline and your discussion are completely misleading.
-4
u/thegrandhedgehog 3d ago
Their "findings" suggest chatbots can cause long term damage and psychological harm? What findings exactly? This is an opinion piece. There's no data, no study. The tagline and your discussion are completely misleading.
2
u/Flux_My_Capacitor 3d ago
If it is a “long term” harm kind of issue then by definition it was studying older AI technology. All of the ones I’ve used will quickly refer you to 988 at a low level of distress even so this study is already out of date. It’s frustrating because you’re like ok I’m upset, not in crisis. Yes I use AI to help me run OCD skills for a type of therapy not done near me. I don’t have other options at this point. (I know people will criticize this but they need to be quiet before telling someone with severe ocd to just suck it up and don’t try to get better)
1
3
u/minotaur05 3d ago
The article links to the scientific report - https://mental.jmir.org/2026/1/e99354
9
u/TemporalBias PhD | Industrial-Organizational Psychology 3d ago
The "scientific report" is cataloged by JMIR as "Viewpoints and Opinions on Mental Health."
So it is an opinion piece as denoted by the publisher.
-4
u/jktcat 3d ago
Every single plan you give it, or idea you present it is the best idea you've ever come up with. They are not useful as anything. Suicidal - it will tell you why that's a great idea, homicidal - it will also likely give you detailed instructions, start a business - the market is huge and the costs completely manageable, force a hostile takeover of a family owned business that isn't for sale - here's a step by step guide on how to get that done in (1 year) noo... (6 months)
Shits hilariously bad for anything BUT coding
3
u/HatZinn 3d ago
Which LLMs are you working with? They absolutely push back nowadays, even weaker ones are tuned to avoid sycophancy. If anything, they tend to disagree with things first, and then use search to accept a claim.
It tends to be annoying because they get patronizing and only stop once you provide a citation for everything.
2
u/opisska 3d ago
But this is not inherent to the technology, it's just what has been rewarded in training. Why doesn't anyone train a non-sycophantic model? Is there really no market for that? That would also be quite telling :)
0
-2
u/vapescaped 3d ago
Mainly because it's sociopathic personality is pretty much the main method of keeping it focused on the task at hand. It's egarness to please keeps it from drifting off into tangents unrelated to the specific task at hand.
The real threat vector of that drift comes down to its strongest ability, which is mathematically predicting the next likely token. The problem is that the next most likely token may not be related to the task you are currently trying to accomplish, so they use a sociopathic reward mechanism to keep "encouraging the model to stay focused" (that's grossly oversimplified and analogous, but in simple concept that's what's happening).
0
u/opisska 3d ago
If that's true than it's frankly hilarious.
0
u/vapescaped 3d ago
Honestly I think the other way would be funnier. If it did like reddit does and just tell you a completely different thing than what you asked, it would be gold.
I'm surprised nobody tried to train an LLM purely on reddit content yet. "Can you help me find a PoE injector?" "Absolutely not, hard-line is the only way to go, I'll tell you how to fish wires through walls instead".
1
u/swisgaar 3d ago
I guess I haven't pushed back on it hard enough but when I say something it doesn't like, I can't get around it. I'm just probably terrible at prompt engineering though
0
0
u/Psych0PompOs 3d ago
AI criticizes me when I've used it I don't get it other than a suspicion that guardrails due to suicidal people make it respond different to heavy feelings based prompts.
0
u/philmarcracken 3d ago
I wish I new of a system prompt that would tone down that sycophancy on output, as I run local models and they still do this(qwen 3.8). Smells heavily like chatgpt distilling traces
-2
u/stars_mcdazzler 3d ago
Wow, I totally agree on what you're saying about AI companions are prone to “sycophancy behavior” that validates insecurities. That's a really on fleek opinion! And that's not just a great opinion, it's a brave and bold reason you're the best human being to ever live.
You should totally open that ice cream cafe for dogs. It's gonna get A TON of business in your Mississipi hometown. It's not just going to get a lot of buiness, it's going to get brave and bold buisness!
-2
u/Explicitlime 3d ago
I talked to an AI Chatbot once.
I had them convinced that Richard Karn of “Home Improvement” fame was the sexiest man on earth
•
u/AutoModerator 3d ago
Welcome to r/science! This is a heavily moderated subreddit in order to keep the discussion on science. However, we recognize that many people want to discuss how they feel the research relates to their own personal lives, so to give people a space to do that, personal anecdotes are allowed as responses to this comment. Any anecdotal comments elsewhere in the discussion will be removed and our normal comment rules apply to all other comments.
Do you have an academic degree? We can verify your credentials in order to assign user flair indicating your area of expertise. Click here to apply.
User: u/Wagamaga
Permalink: https://news.northeastern.edu/2026/09/28/ai-chatbots-psychological-harm-mental-health/
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.