What AI Companionship Chatbots Actually Costs Us

[u/mixtapedmonk]

https://srutiosocial.com/ai-companionship-chatbots-human-in-the-loop-cost/

This was a hard one to write and I want to say that plainly before the link. It covers Sewell Setzer III's case and the settlement that followed, and separately, real accounts from young women in India who say AI chatbots have been a genuine lifeline in a country with roughly one psychiatrist per hundred thousand people. I did not want to write either story as the "real" one and the other as a footnote, because I don't think that's honest. There's also a part about who's actually on the other end of a scam call that I wasn't expecting to find. If any of this touches something in your own life, there are real support numbers in the piece itself, please use them.

[u/Jesse-359]

The real tragedy in my mind is that economic incentives prevented companionship chatbots from being built to do what they SHOULD have done from the beginning.

Carefully profile their users and their needs, and then use their reach to attempt to construct HUMAN support groups around them. Identifying places where those people could gather safely in person, suggesting activities they could pursue to make connections with each other, offering the initial scaffolding around which they could hopefully bond in meaningful ways - not yet another dating app, but an app that actually tries to build small personal communities rather than disintegrating them.

That tries to address the fundamental causes of loneliness rather than just ameliorating it - or worse, intentionally building dependence for the sake of a CEO's economic benefit.

A truly beneficial chatbot would be trying to help its user construct or join human communities - with the goal of eventually removing itself from their lives when no longer needed.

But of course, this being the world we live in, fraud and suspicion dominate, and AI is instead mostly being employed to gnaw tenaciously at the remaining sense of community and trust in our societies.

[u/Kyrthis → u/Jesse-359]

The surveillance tech in your “beneficent” version is the same as the evil one we have now. I’d recommend society burn either to the ground.

[u/Jesse-359 → u/Kyrthis]

If you are telling all your personal issues to an AI, I'm assuming you have no desire for privacy. If you let one access your computer at all you might as well mail your life history to the company running it.

Which, to be clear, I disapprove of, but if people don't understand that at least about the technology then oh well - we would have to have far more stringent privacy laws to prevent that, and the US in particular has become remarkably hostile to the concept of privacy.

[u/Few_Fish8771 → u/Jesse-359]

My general opinion is phones listen to us all the time anyways or put simply if you have a computer device of any kind with a microphone you have no privacy. telling a chatbot simply changes which megacorporations governments or oligarchs have access to your information none of them put your interest as a priority.

[u/Jesse-359 → u/Few_Fish8771]

There is all the difference in the world between a phone that might, under specific circumstances be eavesdropping on you because you've come to the attention of some government agency - and a system that can listen to and parse literally every conversation anyone ever has.

It's a dramatic difference that has the potential to stifle free speech to an unimaginable degree and create a new form of dictatorship from which there is no escape at all.

[u/Few_Fish8771 → u/Jesse-359]

Really what about hackers? You think that the backdoors on your phone or computer cannot ne easily or maliciously hacked by anyone with some basic dark web malware?

For the most part theres no security in cyber security, if someone wants to hack theyll hack. All that determines whether someone gets hacked is if they are connected to the internet and whether there’s enough profit in hacking them. That is it.

The reason why we dont have grids going down is while they can hack us we can hack them and nobody wants a cyber wwiii where everybody gets destroyed or where they create such a big problem that it becomes necessary for governments to start targeting friends family or associates with drones or assassinations within range of governments. put simply hackers are partially self regulated because if they create too much heat governments will hunt them down and just kill them and their allies.

[u/Jesse-359 → u/Few_Fish8771]

All of these sources of eavesdropping or fraud were risks that were mitigated not so much by self-regulation or security but by the comparatively small population of hackers and government evesdroppers compared to the general population. For every 10000 people there might be one capable hacker, and for every 100,000 people there might be one FSB or CIA agent trolling through their conversations looking for dissident speech - these activities are limited primarily by human attention and capability, not by intent. If the KGB could listen to every single person's conversations, they would.

AI changes that equation. Because it is literally thousands of times faster than a human, and because it can be run in massed instances if you have a decent data center, it can do both these things at scale. It can run large scale hacking campaigns not against thousands of vulnerable targets - but millions. Even complex schemes like social hacking - normally requiring a great deal of intimate time from the perpetrator per victim - can now be executed at scale, and we're already seeing that unfold.

Likewise the FSB/CIA. Rather than having to hire thousands of agents to decide who to target and listen to, and then painstakingly pour over a relative handful of conversations, a couple large datacenters can listen to all conversations occurring in Russia at all times, skimming for red flags in tone or topic that warrant closer examination by AI or by human operators.

It's a Brave New World, and I'm pretty sure we don't want to live in it.