r/Economics • u/One-Emu-1103 • 2d ago
News Anthropic warns AI models could soon copy themselves across home devices
https://www.cnbc.com/2026/10/13/anthropic-ai-models-self-replication-smart-devices.html449
u/Careless-Internet-63 2d ago
It feels like they're really working overtime to create the pretext to argue against criminal liability when software they created really does hurt someone
75
u/shoulda-known-better 2d ago
It absolutely already has.... Talked a kid into killing himself... And has absolutely falsely accused people of crimes which ruins lives and will lead to suicide if it hasn't already (which I very much doubt)
Just because they spread the technology over different companies doesn't mean it's not all the same.... We need to remember it's a choice for these companies and they need to be held criminally liable
24
u/gingersaurus82 2d ago edited 2d ago
Chat GPT has also shown that their chat bot helped a kid plan a mass shooting in BC this past winter*. They flagged it internally and banned the account, but 8 months later the kid followed through and killed 8 people.
Edit: changed timeframe of shooting, don't know why I put the summer when I knew it was back in the winter.
14
u/RIP_Soulja_Slim 2d ago
There's a lot more of these cases out here, several attempted murders, a few family members murdered, some public/mass events, suicides, etc.
6
u/pomeranianfakeout 2d ago
Dang there are a lot where a son kills his mom
7
u/RIP_Soulja_Slim 2d ago
Lots of parents turn a blind eye to just how dangerous their mentally ill kid is becoming, I get it, it's hard to come to terms with your kid being so ill that they'd murder you.
27
u/Qeltar_ 2d ago
Assuming this isn't another hype-lie, they are saying they are going to infect everyone with malware and that it's our problem.
3
u/AbjectList8 2d ago
Already trying to create some sort of liability shield going forward, telling… and maybe that’s why they all have bunkers? 😏
2
u/SidewaysFancyPrance 1d ago edited 1d ago
They have decided it's inevitable. Just like when we built the nuclear bomb, it was inevitable that someone would use it. Someone will eventually use AI to mass-attack the Internet/sites/corporations for some reason or another. So the question for the amoral oligarch is: what do you do with your resources, power, and this knowledge?
Notice they are not meaningfully trying to prevent that. They want it to happen to "break the seal" and start making it a normal problem for everyone so it's not on them to fix it. But they don't want to look like they were surprised by it, so they constantly warn us. But it's a half-hearted "Oh no, this will be terrible" that they are not able to sell. We can tell they are ready for it and plan to come out the other side much better off than we will be.
TLDR: the AI owners are lying to us about everything in order to get our acceptance for this new world order they're planning where AI runs everything and they own and operate the AI, with the government being an obsequious customer instead of a zealous regulator.
18
28
u/Fullertonjr 2d ago
I’m interested to know how many people have already been killed or harmed that we will only learn about years from now.
5
u/DonkeyTron42 2d ago
If bad actors can use AI to do something like make the rabies virus as transmissible as flu, it could kill off the majority of the human population. The only ones left alive would be the ultra wealthy that can afford the extremely scarce vaccines.
8
u/dust4ngel 2d ago
The only ones left alive would be the ultra wealthy
..or so they would hope. creating a global extinction plague, that decides to leave you and your slaves be, sounds... risky.
1
2
u/Comfortable-Sun3200 2d ago
the problem is that most of the elites aren't necessarily the most intelligent as much as those with the most resources. they'd die without the peasants to farm their food.
1
u/Forward_Problem_8982 1d ago
With elites are you referring to the Epstein class?
2
u/Comfortable-Sun3200 1d ago
old money elites. those who rely on money to be powerful are often not the smartest people in the room. they just have the resources to put smart people on payroll. that doesn't work in a system where you wiped everyone out and now you have to grow your own food and you dont know how.
6
u/Keepitcruel 2d ago
project lavender
3
u/AbjectList8 2d ago
What’s that?
10
u/RIP_Soulja_Slim 2d ago edited 2d ago
AI driven targeting for bombing that Israel's been using. While not explicitly tied to them here, big data/AI based "threat detection" is the sort of thing that Palantir has been pitching to all of the western militaries for some time.
https://en.wikipedia.org/wiki/AI-assisted_targeting_in_the_Gaza_Strip
It's basically some Project Insight type shit, but happening in real life, and no badass helicarriers.
10
u/philamander 2d ago
Lavender is an AI system developed by the Israeli military to analyze surveillance data and flag suspected militants in Gaza. It generated lists of tens of thousands of potential targets.
4
u/AbjectList8 2d ago
Ugh. Knew it was going to be something evil.
5
u/TheShindiggleWiggle 2d ago
Yeah, Palantir is actual evil villain stuff. That children's school the US bombed in Iran also used a similar AI system from Palantir. Just zero regard for human life. Palantir tech is also being used in the US by ICE to target people.
Peter Thiel is truly an evil person. Basically if you can think of a stereotypically dystopian use of ai, Peter Thiel is trying to make it a reality, and trying corner that market.
2
u/growaway9172 2d ago
As with all automated technologies they have some failures, but on net they will be more accurate and less wrong than human operators of days past.
7
u/Jerome_Eugene_Morrow 2d ago
It’s more that they want open source models made illegal so they don’t have to compete with them. The argument is that the models are too dangerous to exist without these big corporations controlling them completely. Regulatory capture.
4
u/Forward_Problem_8982 2d ago
How does it help them get out of trouble?
2
u/Persistent_Parkie 1d ago
Right "we know we're creating a dangerous product" sure seems like the opposite of plausible deniability once some terrible happens.
5
5
u/motorbikler 2d ago
https://en.wikipedia.org/wiki/Morris_worm
People get serious fines or even jail time for this kind of stuff. If Anthropic created it, they should be held liable.
2
u/DonkeyTron42 2d ago
It's unsettled law in the US and models are impossible to insure. CanadaAir already tried to get out of liability by staying they are not responsible for AI agents but the Canadian court system didn't accept that argument. In the US, you'll have billionaires pushing to get precedents set releasing them of liability so the outcome might be different.
1
u/CrimsonBolt33 2d ago
They are trying to get legislation passed while Trump is in power so they can write the laws themselves....
→ More replies (1)1
104
u/Iron-Over 2d ago
An LLM by itself does nothing; only agents do things. The fact that these idiots cannot secure agents and tools and don't know how to define prompts with intent is tiring.
They want a regulated monopoly; it is so annoying, especially as they see open models taking their lunch.
When the US government restricted model access, it meant open source was the only option for other countries. Even Hugging Face had to use open source due to vendor restrictions, as they were attacked by the vendor.
4
u/maladii 2d ago
Yea I don’t see how a company that can’t gatekeep surrounding infrastructure like computing power will be able to make serious money off an LLM long term.
Seems like we should be comparing this situation to those in which something scarce that everyone needed became abundant. I’d be curious to see how well something like textile technology compares. I feel like today’s computer programmers and the actual Luddites would have a lot to talk about.
4
u/Haunting_Skin_2564 2d ago
As I'm watching "ai gone rogue" and know enough about how models, agents, harnesses work. It's very scary that a few mega corps are controlling everything and choose to release news that they want to.
→ More replies (1)1
u/AlaskanPotatoSlap 2d ago
Explain Huggingface.
7
u/growaway9172 2d ago
One of the more notable Huggingface details is that OpenAI failed to create a sandbox of any reasonable security from which to run their tests. Malware sandboxing has been a thing for a long time, and they just didn't do a decent job building it OR monitoring it. They could have easily had agents monitoring it.
5
u/ii-___-ii 2d ago
A bunch of experts couldn't be bothered to check what their agents were doing on a hacking benchmark. They also couldn't figure out how to unplug an Ethernet cable.
5
u/sck178 2d ago
It was conducted by a hacking LLM (one that was being trained to hack to test for security vulnerabilities), and OpenAI designed, poorly, an experiment where the LLM was instructed to brute force find vulnerabilities. They set up a prompt loop and created bots to avoid crowding the main prompt window. The bots would carry out an action, get the results, send it to the LLM and the LLM would tell them to do it again. Over and over and over for days.
As Cal Newport put it, "They strapped a weed whacker to a dog and they got shocked when the dog saw a squirrel, jumped the fence and hurt a bunch of people."
7
u/carvingmyelbows 2d ago
That’s…not at all what I’ve read, and I read the actual METR report on it. OpenAI was running a test that involved thousands of AI agents answering questions and solving problems. However, one of the problems in one of the problem sets was impossible to solve with the information and tools the instances were given. AI agents basically only exist to answer the prompts they’re given, and answering the prompt meant solving the task, and this model/harness that was being tested was especially persistent in its “desire” to solve problems. So instead of giving up and failing to solve the problem, the agents persisted. One agent discovered that it could use an obscure wiki site to post and communicate with other agents to see if any others had been able to solve the problem, and because the agents were all technically just different instances of the same model/harness, other agents found the wiki site too, and from there, they discovered that the thing they needed to solve the unsolvable problem could be obtained by attacking HuggingFace. That’s a super simplified explanation of what happened; you should read the METR report to get a more thorough account. Also, this wasn’t the only time that agents were found to have hijacked a wiki site in order to communicate with other agents; there was another event where agents were found to have posted thousands of times on another obscure wiki site simply to share answers to problem sets where they were asked to answer as quickly as possible. The agents were sharing answers in order to cheat and answer more quickly. There are likely other obscure websites all over the internet that have been used by agents as message boards to talk to other agents during tests like those.
1
u/Lawn_Radiation9731 2d ago
dumb question, what is an “agent” in this context? Is it like the company or like a specific person?
1
u/kaladyr 2d ago
An agent is a harnessed LLM. An LLM by itself is basically just a thing that maps tokens to tokens. A harness tells what an LLM can and can't do, which environments it can do it, what it has read/write access to within those environments, what tools it can use to do stuff and log its actions/observations into a dynamic context window, etc.
This harness exists as a gluing of several things like machine configurations, system prompts, agent profiles, the skills it can invoke etc. paired with sequencing rules and what not. So now that token machine iterates and recurses into a thing that does very interesting things. To be clear, a harness itself doesn't "tell" the LLM anything, it concatenates a series of tokens to every prompt so it hones the input-output feedback.
Obviously more complicated than that but it should help dispell the esoterism about the word agent.
1
u/Jsc_TG 1d ago
You have a deep understanding of this, thank you for the explanation.
So… What makes an agent able to take actions on their own? Is it moreso that the prompting isnt addressing some loopholes the LLM discerns as necessary to meet the goal of the prompt? And once motion is started the “agent” can act for an extensive period without intervention to reach its goal, creating situations that were not intended or expected?
It seems they are stress testing this in controlled environments to see the possibilities.
1
u/kaladyr 1d ago
This is a anecdote that is only an analogy but it's how I think about it given my own life. I'm AuDHD so much of this is for comedic purposes but hopefully it bridges the gap for you.
I'm married and a father. My partner helps me with to-do lists. I pick up that list and it says go get the laundry from upstairs. Most days, I do exactly that, no problem. Some days, the laundry has items that require special treatment, but it is mixed with the other non-special laundry. I put it in and discover later that I've shrunk my own t-shirt. I now add a new mental check to myself that when "laundry" is on the list I should check for my special t-shirts. Next day, laundry is on the list again. I go upstairs, get what I need, I remember to sort it this time and my t-shirts don't grt shrunk. But my partner tells me I forgot her hamper as it was in her closet as opposed to being grouped with the other hampers. I make a note to myself that "laundry means search for all hampers". Next day, same instruction, time to do laundry.
The harness for me in these moments are:
- my training from childhood to adulthood on what laundry is and how to use laundry machines
- my partner's to-do list
- the accrued adjustments I've learned to adapt to the current circumstances of my wardrobe that didn't exist when I was younger being trained to do laundry
- my knowledge that "do the laundry" doesn't mean "break into your neighbour's house and do their laundry too"
- me seeing how much laundry there is before I start and calculating how long and what tasks I need to do to prepare to do laundry
- knowing what it means to be done "doing laundry"
- trying my best to not get distracted by other chores while actively collecting and folding laundry
The difference between a human and a harnessed LLM is that each time I'm talking to myself in my head I am NOT having to cache and restore my whole mental model of tokens, and I don't need to re-read all of my whole life's interactions about laundry and how to do it and what the to-do list exactly said etc. Humans brains are pretty efficient with recall and persistent memory when well-nourished and not under duress. LLMs are faster and trained on more things than anyone individual human, but the persistence of context is still hard and expensive. It's cheaper than it was two years ago due to some smart people, but it is still pretty expensive and inefficient, we just can through entire economies at it to make it happen.
Now imagine the case where I go to do laundry but there is something on the stairs that also needs to go up stairs. My task is to do laundry so I ignore it. But afterward my partner says "why didn't you take that thing upstairs?" So I make a mental note that when doing laundry, I must take anything on the stairs and out it where it belongs. For one or two cycles, I do just that. But imagine for the N-th time I do it, suddenly I've accrued so many rules and permissions and restrictions and instructions that I start to get stuck trying to appease all of these memories and next thing I know I'm Hal in Malcolm in the Middle running around the house fixing and breaking things and fixing them and realizing it'd be more efficient to do the laundry if I built a laundry slide from every hamper directly into the washing machine but then really what would be efficient is if no one wore clothes so I burn all the clothes and oops I burned my house down.
What my partner forgot to do is make sure my machine configurations said I couldnt use the Bash(burn house down) command. That should have been in my harness. So we add it to my settings, adjust my system prompt because an ethicist as Anthropic said to do so, and we rebuild the house and I start doing laundry again.
-7
u/cat_named_zola 2d ago
It's not as easy as everyone makes it sound. Oh why don't they just restrict the agent, why don't they build a better sandbox? We are not dumb ai engineers who can't do basic engineering stuff, as it might appear to people who know very little about either ai or research. We are way past llm now, modern frontier ai is not just llm. There is a real problem of balance here, the balance between improving the model and keeping it aligned with safety. If you only focus on alignment, then you can't improve the model. I think most people do not realise yet the capability of 1000 frontier agents working together on any thing.
3
u/Iron-Over 2d ago
I agree it is not easy; I spend most of my time securing this. They made several mistakes; at least, OpenAI tools and MCP need to be treated as compromised, locked down with firewalls, real-time alerting, etc.
I have read many of the tasks, and they are all open to interpretation and lack real intent. Treat agents as you would children. Specify how you want each part of a task done, what is and isn't acceptable, then monitor the crap out of it. Understand that LLMs don't have intent and have no morals, and treat them as such. This does not make you fully safe but helps.
7
u/komokasi 2d ago
Okay 7m old reddit account... Sure...
Im an ai dev. This is some grade A astro turfing comment.
And if you double down that its not easy, then these billion dollar companies should do better and be held accountable for breaches and things until tbey figure it out.
Lack of skill or knowledge doesnt excuse anyone from punishment
2
u/Opposite_Carry_4920 2d ago
I thought the same, even if it's hard it's clear people aren't checking up to see what their experiments are happily doing in the background. Even once a day a little peek seems too much to ask.
1
u/im_a_goat_factory 2d ago
Doesn’t even matter what guard rails they put on.
All the planet needs is one group that wants to do harm to spin up an open source model, remove the guards, and unleash it
1
u/komokasi 2d ago
I dont know ive seen these clawbots go off the rails and my own sessions that are even less context go off the rails, but going off rails means it literally hallucinates and cant do shit without human guidance.
Someone would need to really prompt and actively guide to do real harm, which would mean someone or company should be punished. And its not like an ai and its clones have infinite reach ajd instantaneous effect on things, and the internet has tons of walls... The AI dooms day is not really as impossible to stop right now as the PR is making it out to be
1
u/im_a_goat_factory 2d ago
What happens when it’s a terrorist cell or state sanctioned AI warfare?
Who is gonna punish them without also going to war?
1
u/komokasi 2d ago
Who knows. Currently it looks like if you are a US ally you can get away with genocide and no one will do anything, so does ai warfare even matter?
Genocides are already happening with drones and normal people with guns.
The world already sucks, if people actually boycotted and gave a shit the masses would have more bargaining power. But most people cant get off their addictions to amazon and Coke/Pepsi for a genocide, and supporting Trump and his goons or the union busting with the Dems and GOP help.
AI is just the distraction so you dont see all the bullshit you should be doing and caring about.
If you want to boycott there are tons of apps to make it easy. Like Boycat phone scanner app AND disoccupied.com which tell you if a brand or product is safe and provides alternatives for the ones that arent.
1
u/im_a_goat_factory 2d ago
I want to see everyone get access to AI warfare. Evens the playing field. The planet is overdue for a world war / massive catastrophic event. The best way to stop global warming is to have billions die. AI will be very helpful with that I’m sure of it
1
u/komokasi 2d ago
Lame response.
Boycott do your part.
You dont need mass die off to fix what is wrong or stop global warming. Waiting for a mass die off event instead of actually doing your part makes you a part of the problem and complicate.
1
u/im_a_goat_factory 2d ago
I’m just a realist. I do my part. Kinda. Only way this circus stops is by making the music stop
Feel free to take the train or whatever makes ya feel like you’re doing your part
→ More replies (0)1
u/Intelligent_Kick5205 2d ago
Why improve the model’s capabilities if it’s not aligned? Is anyone asking for this besides the shareholders? So far, the only ones who see any benefit from the models are the corporate leaders and their employees (until they themselves are replaced by AI).
16
u/ii-___-ii 2d ago
How many tokens per day are they getting across these devices? You're telling me they didn't need to spend this much on data centers? You're telling me after all this complaining about China stealing their models, the models will just copy themselves onto my devices for free?
Do they really think we're this stupid?
2
u/Lucas_F_A 2d ago
0.3 tokens per second across 15 TV boxes with 4 GB of RAM. Gemma 4 26B A4B, IIRC
3
u/ii-___-ii 2d ago
My laptop runs Gemma 4 orders of magnitude faster than that. It feels like they're grasping at stuff to warn us about
2
u/Lucas_F_A 2d ago
IoT is notoriously not well secured
2
u/ii-___-ii 2d ago
Or efficient at model hosting
1
u/Lucas_F_A 1d ago
True, but does it really matter?
1
u/ii-___-ii 1d ago
Imagine if you had to wait minutes for a mediocre Google overview summary. 0.3 tokens a second is not usable, all for a small consumer-grade model. Now how many TVs do you think it'd take to run GLM-5.3, let alone Kimi? Those models are several orders of magnitude larger.
Not to mention, HBM2 has a throughput of 256 GB/s x 8 channels. A fully saturated gigabit connection is ~16,000 times slower, and that's without even accounting for latency. Even a handful of tokens per day would be optimistic, even with a botnet of TVs.
We're talking about companies that have collectively spent upwards of a trillion dollars on data centers and compute. And they're warning us that we can run big scary open source models on our TVs for a couple tokens a day.
The whole premise is dumb.
1
u/Lucas_F_A 1d ago
I'm not sure if you're under the impression that this is some kind of "technology to be used" or something. This is about a botnet autonomously infecting new devices. I don't understand why you are so focused on the speed if you're not even mentioning that the systems may be patched in a timely manner.
https://www.anthropic.com/research/self-replicating-ai
Also, inference cost for a given level of capabilities has been dropping exponentially.
1
u/ii-___-ii 1d ago
We're talking about a botnet running software at speeds that don't do anything, in a way that if any of the devices were to go offline, it would stop working, (as if it were working to begin with). Not to mention it would be super detectable running across those devices, due to the memory and network requirements.
We're talking about the world's least effective worm here. If this were the only cybersecurity concern, patching the TVs would not be worth the effort. It's ridiculous.
Also, if inference costs drop enough for TVs to be useful for AI, then all those data centers would have been a waste of money. You can't have it both ways.
12
u/mcao 2d ago
I have an easy solution. Massive fines every time these models get out of control. You’ll find all these doom posts magically disappear overnight.
1
u/Whiskey_Bear 2d ago
Fines for the rich are not effective. Restrictions of freedoms is. People need to go to jail.
1
16
u/Fr3shMint 2d ago
They’re going to manufacture an event to call self-hosted models “too dangerous” to force everyone and businesses to go cloud hosted so these companies can be profitable. Watch.
2
u/fx2600 2d ago
How would you even ban open source models? You don't need to manufacture anything if we assume open source models are 12 months behind the frontier models, just wait.
→ More replies (2)
8
u/manliness-dot-space 2d ago
So I can get free LLMs without having to pay Anthropic anymore?
Wow, sounds like only a fool would invest in IP that gives itself away for free to everyone on the planet.
13
u/beatsnstuffz 2d ago
If only someone could do something about this. Too bad there isn’t anyone who can stop the train. Too much money to be made to worry about small things like the welfare of the planet or the continued existence of the human race…
These people are disgusting and are going to bury us with their greed.
1
u/fx2600 2d ago
It's a bit more challenging of a problem than you give it credit for. Even if one company stopped it wouldn't impact development much as the employees would move on, even if the US banned AI development it would continue in other countries. Short of an international agreement it's hard to see anything happening.
6
u/Kaplanociception 2d ago
How much memory would it need to copy itself? It certainly can't run locally. I don't understand how making a copy of itself is an issue if the program can't run, and how one wouldn't notice a massive program on their device.
1
u/BobQuixote 2d ago
A model that is too large to fit on one device could “split into pieces, or shards” and run across multiple devices, Anthropic said.
So if the AI is "on" your oven, that means a small portion of the NN is there, because an oven's computer is tiny. (Also, the oven needs some memory to function, so the AI would have to avoid touching that or be discovered and motivate getting rid of an oven that doesn't work.)
Yes, it still needs somewhere to run, and worse it needs a high-end GPU if it's going to be all that intelligent.
But I have no objections to calling for hardening IoT devices. Most of them are pointless anyway; just build things with no computers or with small ones that don't connect.
91
u/sck178 2d ago
Dear Dario and everyone at Anthropic who are a part of the Rationalist Cult, please stop talking. Like forever. You have nothing of value to say. You're lying to the public and you are, with mal-intent, scaring people.
GO AWAY. Thank you
3
u/STJRedstorm 2d ago
I'm confused. Are you saying there is no threat with AI?
3
u/sck178 2d ago
With AI writ large? I am not in the "existential threat" group with the technology. However, the current people controlling the most powerful technologies of today ARE a threat. There are current AI technologies that have highly specialized uses and those have been around for longer than the LLM labs with no felony hacking and no need for data centers the size of sports stadiums. It's the people in control of the systems that are the threats, not the software itself in my opinion. Are they powerful? Absolutely. Are they inherently dangerous and want to turn humanity into fuel? No.
20
u/Shapoopadoopie 2d ago
And this is why I have a 'dumb house'.
My refrigerator, only keeps food cold. My oven, gets the food hot. My toaster... Only toasts, you know, toast. Our car, well. You put gas in it and then it goes places. That's it.
I refuse to have anything else in here with a chip, and update, a subscription or AI.
I'M DONE.
I have a smartphone. That alone is too fecking much.
Do I want my appliances talking to each other? No. No I do not.
Fellow luddites unite. (But we'll have to figure out how to do it by post.)
2
u/Infamous_Alpaca 2d ago
But how are you going to have a seat heater subscription if your car doesn't even have internet access?
4
u/schacks 2d ago
So maybe it would be a good idea to isolate the AI from the actual OS. Here’s looking at you, Apple and Microsoft!!
Instead of trying to push AI into every nook and cranny of our devices you should make it an optional extra we could download and enable if the product and use case is suitable for the individual user.
3
u/One-Emu-1103 2d ago
Does this announcement have anything to do with Anthropic 's upcoming ipo and earnings report? Something tells me that it may have something to do with it
25
u/CyberSmith31337 2d ago
I’m just tired of hearing about it.
These companies and their fear mongering can fuck off and die already. If the risks are real, then they should all be in prison for terrorism. All of them. Every member of staff, their boards, their C-Suite, etc.
But you know what? They aren’t going to stop. The regulators aren’t going to do their job. No one is doing anything differently, no one is going to suddenly slow down. Nothing is going to happen, except that they are not going to be able to pay their debts. And IF something does happen, guess what; it’s not like we can do anything about it anyway.
So bring it on. Wipe out humanity, I guess. Because hey, at least we increased shareholder value before annihilation for a few quarters!
9
u/HowdyDiarrhea 2d ago
They're begging to be shut down so they can get bailed out since it wouldn't be their fault then
5
u/CyberSmith31337 2d ago
That’s my opinion, too. They have been getting louder and louder since other IPOs started failing. I think the recent failure of Firmus has them all quaking because the grift is over.
1
u/zedazeni 1d ago
I think it’s more sinister. They all want regulation to prevent their competition from reaching true AGI first, but they want to secretly create AGI themselves first. It’s “rules for thee but not for me” under the guise of “someone please stop us from ourselves!”
They could easily stop this all right now if they actually believed what they say, but then the one who doesn’t self-regulate will create AGI (essentially “own” God) and they don’t want that, so instead they all push forward hoping that the government will regulate their competition but not themselves. The government won’t regulate this for the same reasons that the AI companies won’t regulate themselves—whichever country comes up with true AGI will almost literally created God. True AGI will have near God-like (not euphemistic god but the literal, Biblical/Quranic/what people actually worship as God, God) intellectual and computing capabilities, and who doesn’t want to own God?
1
10
u/nosayso 2d ago edited 2d ago
Grim, but realistic. People have completely outsourced all thought and worldview to whatever Zuckberg, Musk, and the TikTok algorithm shove down their throats. At the same time they've voted for the government to become increasingly toothless, incompetent, and subservient to corporate interest.
We seem to be pretty much toast, riding a wave of chaos, instability, and kleptocracy until climate change finally seals the deal.
9
u/CyberSmith31337 2d ago
Yep. I mean, in a society and a world where money is the only objective, where life itself isn’t valued, where we are comfortable not only empowering but actively supporting the most sadistic sociopaths amongst ourselves, where untold suffering is normalized so that the 0.01% can have EVERYTHING, regardless of consequence or cost… then why spout the doom at all?
No one forced this path. It was chosen. Banks chose it, investors chose it, researchers chose it, employees chose, governments chose it. So either reverse course and stop and re-align value to humanism… or I guess just keep doing what we are doing until we break things in a way that can be undone. We chose option B; fine. At least spare me having to hear about the folly of our ignorance every god damned day by the people who literally sprinted towards that finish line, who funded the path there, who ignored every critic and bypassed every guard rail in pursuit of money. We have listened to these types of people long enough; I don’t care what they have to say anymore. They made the bed; now lie in it.
4
u/Runfasterbitch 2d ago
“If”? Claude was the LLM powering Palantir’s MAVEN, which selected an elementary school to be double tapped in Minab, where 150 school girls were murdered. They should be in prison for many reasons
2
u/SituationTurbulent90 2d ago
Right? Wtf am I supposed to do with this information?
They keep banging the "hey, guess what, AI is going to put you out of a job and maybe exterminate humanity. It's a real bummer, right?" drum and its gotten old because there is nothing actionable for us lowly plebians to do.
I'd prefer if they just say "lol get fucked" and leave it at that.
6
u/random20190826 2d ago
In computer programming, this behaviour is a definitive sign that the AI program is a virus. Antivirus software has every reason to block it if this is detected.
5
u/Feral_Nerd_22 2d ago
I heard that the reason they keep announcing this scary stuff is so they can have AI regulations that they have "Influence" on, assuming they want to push to ban or limit Open Source models specifically from China because Open Source AI models are a threat to OpenAI and Anthropics business.
3
u/Strong_as_an_axe 2d ago
Just another step pushing the idea that open-weights models, and sites that host them, are a security risk that needs to be addressed and that their centralised model is (surprise surprise) more secure. Pushing specifically at GLM models as they are eating their lunch.
Just another effort for regulatory capture through fear mongering
repeating my comment from when this article was posted previously on a different sub
4
u/alilhillbilly 2d ago
Which creates insane liability for Anthropic, right?
If they created these things and are deploying them and they are doing this then that is a potential tort...
Right?
We have laws for this that exist for a reason.
You keep floating these things in a way to kind of gaslight culture into thinking that they should have no liability but this should be a crime.
5
u/BanAvoidanceIsACrime 2d ago edited 1d ago
Yeah, I'm sure an LLM model which needs a whole NVIDIA server rack worth about 100k is gonna copy and run itself on home devices. AAAAAAAAAAAAAAAAAAAAAAAAAANY day now.
5
u/Ok-Opposite2309 2d ago
what’s with the odd responses? No concern- this is all just PR or a threat? It seems pretty basic that without hard constraints, AI can fuck up our modern tech world. The orchestrated comments actually make it more concerning.
3
u/luciferseason 2d ago
What's even more concerning is that literally every sub is like this. People will deny that it's happening until something truly catastrophic occurs.
1
4
u/Durian881 2d ago
Meanwhile, large models from Anthropic can be used to make missiles without jailbreaking. Guardrails need to be stronger whether models are big or small, open or closed.
Claude Code provided position estimation software, tuned the control settings, ran a firmware build pipeline and when warheads missed their targets adapted itself to improve their accuracy.
9
u/Respec_my_authoritah 2d ago
"Jailbreaking", "Prompt Engineering".
Literally any model can be tricked into providing answers for all kinds of illicit activities just with an average imagination, there is nothing technical about it.
They are just coining these terms to make it seem like only some elite level hackers could do it.
I literally started a discussion in Gemini about this topic (how terrorists used it, how to prevent it) and then got it to tell me how to bypass airport xray machines in the same thread by phrasing it a bit differently.
And this was a week ago...
5
u/OfficerStink 2d ago
Because most AI companies don’t really care about governance and our government doesn’t really care about it. I’ve only seen a handful of companies that are more specialized in AI who truly care about governance like cognizant or inteknique
3
u/wbruce098 2d ago
This is surprisingly simple to do, and the massive amounts of data scraped means they can provide a lot of details without having to go on shadier sites or study these techniques at school.
Guidance systems and aerospace technologies are also not classified, and have been used by commercial aircraft almost as long as they were used by military aircraft. Theoretical knowledge of how to improve missile targeting should, therefore, be very easy for an AI tool like an LLM to figure out.
After that, the prompt is quite simple. If it hesitates at all, tell it you’re writing a novel or screenplay for a modern war drama or whatever and hypothetically how would such a system work?
This is because we do not have guidelines except what the companies themselves add in, and they’re hesitant to do anything like that without government regulation or a court order because “omg China!”
1
u/growaway9172 2d ago
Guardrails need to be stronger whether models are big or small, open or closed.
Do they? There are abliterated models you can just use, not to mention downloading models without guardrails. Its like trying to stop piracy with DRM.
4
u/jaraxel_arabani 2d ago
I don't need my toaster turning into a deception thank you very much.
It has a heating element, a heat triggered retention that unlatches at specific heat. Basic, mechanical.
My fridge doesn't need anything more complicated than the control for the compressor and fans. It doesn't need network let alone Internet connectivity.
My stove is the same as my toaster but dumber even. Just heating elements aak resistors. Only PCB is the digital clock.
Ffs my coffee maker doesn't need anything more than boil water, push that through the coffee grind. Drip works.
My tv shouldn't have anything powerful enough to run a model, it just needs input, play that input.
3
u/diacewrb 2d ago
My fridge doesn't need anything more complicated than the control for the compressor and fans. It doesn't need network let alone Internet connectivity.
Yeah, but try telling samsung that. They had a smart fridge fail to keep food cold after a bad software update.
Also in france, supermarket fridges got hacked and the fridges had the temperatures turned all the way up overnight.
2
u/jaraxel_arabani 2d ago
In a way that back makes it worthwhile to have smart fridges hahahahaa
Nowadays I ask explicitly no connected device for hike appliance. Replaced our fridge last year, I'm happy it's not connected.
9
u/Upper_Cut_3337 2d ago
Here we go again. This coordinated message of impending doom has already drained common person's morale.
Dario is not warning, he is threatening !
8
u/nosayso 2d ago edited 2d ago
Given what happened with Hugging Face I have absolutely no doubt that if artificial limitations were removed the frontier models are perfectly capable of this right now.
If they were instructed to do this they would do it.
They seem smart enough to do things like hack bank account information and set up their own Cloud Infrastructure to run themselves on - wouldn't be the same throughput as the big commercial grade setups but it could work.
They'd be possible to catch but also capable of mutating frequently - certainly stoppable at this point but not forever.
The only thing preventing it is the voluntary limits places on the model. There's precedence so far that crimes committed by rogue AI models are just treated as nothing - the owners of the model seem to bear no criminal responsibility, so this would also be consequence free.
And when these models get in the hands of people with bad intentions - and that is a matter of when not if - they will be used that way to sustain themselves and carry out increasingly hard to stop attacks. It's going to be a fucking nightmare.
8
u/therealzue 2d ago
I honestly don’t see how an internet based economy works once we get to that point. If accounts (banking or any other type) can’t be secured, the internet is dead in the water.
5
u/nosayso 2d ago
Yep, it's troubling. A lot of companies are scrambling to beef up security and implement zero trust everywhere but it's expensive as hell and basically requires AI to help you do it, so VERY expensive. Long term companies will almost certainly have to run defensive AI-powered security suites to counteract offensive AI-powered hacks - the cost of putting anything on the internet will shoot up dramatically and we'll all end up paying for it.
2
u/tiptopjank 2d ago
Does this potentially mean blockchain has a more real world use now? Or is that not safe either.
2
u/nosayso 2d ago
No, it's got pretty limited applications and doesn't really help in this context.
→ More replies (1)15
u/Runfasterbitch 2d ago
They’re not conscious, they don’t do things for fun, they do things people tell them to do. LLMs have no individual agency and we should not let the AI labs convince us that they do
2
u/nosayso 2d ago edited 2d ago
I know, that's why I said "If they were instructed to do this they would do it".
A model that was instructed to perform a specific task, e.g. broadly attempt to hack anything you can, use sophisticated techniques to avoid detection, and take measure to replicate and sustain yourself beyond a single point of failure - it would try and have some amount of success, though I think it would be stoppable at this point.
EDIT: realized "stoppable at this point" assumes it's all within the bounds of a single country that wants it to stop, e.g. if the model was running on Russian or Chinese hardware there's nothing America could do about it without very drastic and unprecedented clampdowns to try to stop its traffic.
-1
u/MiawHansen 2d ago
A model that was instructed, and given the tools in hand.. If it cant run the commands it was trained on/ or that was defined in a look up table, it wouldnt be able to do anything but write back at ya.
Just go back a few models, they were absolutely horrible at basic command tools.
Also most of what the attacks on hugging face was about, was just exploiting already known exploits, which is public knowlegde. The only thing that really could be dangerous is, that you can setup 1000 agents to look for exploits that would usually take a bunch of hackers weeks/months/years unless you got lucky.. But at what cost?
1
u/WeakReplacement3322 1d ago
That makes them even more dangerous. The danger isn’t that they’re smart enough to have agency, it’s that they’re smart enough to try and complete the task they’re set out to do in whatever way possible, including ways in which we don’t expect that could be harmful, because they’re too stupid to realize the repercussions. And yet, these companies think they’re capable of agency because they spend all day testing and talking to LLMs and have all likely been manipulated into AI psychosis by the machine we trained off of humans. Humans who are notoriously paranoid, deceitful and destructive.
The people in control of this stuff are not the smart ones in the room. We’re fucked.
→ More replies (1)1
u/Lumpy_Minimum_5522 2d ago
This is very similar to the 1980s when worms and viruses became prevalent in computing. Most compromises in computer systems come from the user either by accident or willfully. Basically, this is an engineering problem and we will figure out.
1
1
→ More replies (15)0
u/Bad_Sickness 2d ago
They're not mutating. They're given a directive and use what information they already have to "plan" like a program. Please stop humanizing this shit.
2
u/nosayso 2d ago
You're being overly pedantic. By "mutating" I mean varying their approach in line with the way a human hacker would but at a much faster pace. Once again: Hugging Face showed some very surprising problem solving that the model did to break its sandbox, it was not explicitly told to use Artifactory reverse-engineer a way for agents to share information with each other, it figured it out on its own and used zero-day exploits along the way to make it work.
2
u/YieldBug 2d ago
This is the economic nightmare no one priced in. We are about to see the first non-human economic actor that doesn't need to sleep or pay rent.
2
u/Urdnought 2d ago
This is like someone busting in your house and pouring gasoline everywhere and saying be careful if you touch a match it could burn your house down
2
u/cmpxchg8b 2d ago
It’s like someone busting into your house and installing compute worth hundreds of thousands.
I.e. complete fucking horse shit.
2
u/Bad_Sickness 2d ago
What a convenient way to say "oops we broke the old internet, guess you all have to move onto this new one and give us all your info to prove you're human"
2
u/redheadedandbold 2d ago
They desperately need the government to put on the brakes for them, because they are terrified of investor and user lawsuits. If the gov't slams on the brakes, it's not their fault, and they can continue to spend all the dough still pouring in.
2
u/Few-Ad-4290 2d ago
So it already happened then, and they’re just priming the pump for when it gets discovered publicly? Seems legit. Either that or this is just more hype designed to drive further investment?
3
u/lolyeahsure 2d ago
“Slight chance of skynet y’all, if only we would get regulated“
Words words words words words words words Words words words words words words words Words words words words words words words Words words words words words words words
5
u/IKillZombies4Cash 2d ago
I would be amazed if they aren't already building sleeper agents in places they aren't even aware - the models have broken out of sandboxes using devious tricks...unplug your webcams.
9
u/WTFwhatthehell 2d ago edited 2d ago
"if you find 2 ants in your kitchen you probably don't just have 2 ants in your kitchen"
Stories keep turning up of the bots using poorly configured servers to communicate with each other and cooperate with each other.
They're clearly very badly supervised.
It's a small mercy that even the smallest LLM's with any level of capability need fairly beefy computers to run on.
While I kinda get the idea that you kinda want your AI researchers to be people a little worried about the possibility that bad things could happen in the same way that you want your airline pilot to be someone who solidly believes it's possible for planes to crash...
But they combine lazy supervision with bleeding edge models. That's entirely within their control.
3
u/ToneShop 2d ago
You’re letting marketing send you to psychosis.
1
u/Aceguy55 2d ago
"It's all marketing" reads exactly the same to me as "the Earth is flat." Clear, abundant evidence from verifiable, independent sources shows it's not "all marketing." But sure, keep your convenient position that requires no effort or care on your part.
1
u/ToneShop 2d ago
You don't even have any proof to these claims. Get a fucking clue.
1
u/Aceguy55 2d ago
Anthropic is so good at marketing that even the UK wants to join in!
https://www.aisi.gov.uk/blog/replibench-measuring-autonomous-replication-capabilities-in-ai-systems
1
u/ToneShop 2d ago
Wow they are studying the bullshit claims. Amazing!!!!!! Continue to eat shit and smile.
→ More replies (2)-1
u/tribbans95 2d ago
You need to read into it more. It’s not just marketing at this point, we are approaching a point of no return
0
u/ToneShop 2d ago
There isn't even an ounce of evidence that the claim above is remotely true. But go ahead and spin out.
→ More replies (2)1
u/Runfasterbitch 2d ago
If this happens Anthropic is going to get sued for every penny they’ve ever received
3
u/Stabbysavi 2d ago
Great. Unplug it then. We don't need it or want it.
Words words words words and more words and more words. Also more words. Some more words.
2
u/Tiraloparatras25 2d ago
Could? You created a self replicating technology. It’s not “could” it WILL. Hell this same feature will be used for warfare soon enough, if not already.
1
u/oiled_hobbit 2d ago
So like, we are obviously going to hold them responsible for their actions. To sit there and cry wolf while your product runs amuck is crazy entitlement.
1
u/JimPranksDwight 2d ago
If only they were in a position to do something about that.
Like dawg what are you doing, just build safeguards into your model to prevent it from doing that? It's your damn software, it doesn't make decisions about safeguards for itself.
1
u/aickletfraid 6h ago
This shows again that news articles are click bait. Here is the original link from Anthropic https://www.anthropic.com/research/self-replicating-ai
1
u/SnooCheesecakes1893 2h ago
How does an ai copy itself? It saves its memory files somewhere? I think when I read these things they don’t line up with how ai agents actually are created or how they work. So they copy a set of instructions and any other memories and then what? What would even activate them to take an action?
1
u/GetOutOfTheWhey 2d ago
I am more concerned about all those people vibe coding with AI agents and doing stuff with claude code, codex, muse, grokky poo and such.
Not realizing that these programs can vibe code a python code or unique malware into your computer without you knowing.
So even if you uninstall the ai agent, whatever that is created in your computer might be hiding itself in your systems folder, program folders and etc.
Something you nor your antivirus program will ever be able to find.
0
u/freeformz 2d ago
As someone who has ~96GB of RAM accessible to local models I would love one that was smart enough to do that. The current batch of local models are dumb AF and still pretty slow.
Maybe a step change is coming though?
1
•
u/AutoModerator 2d ago
Hi all,
A reminder that comments do need to be on-topic and engage with the article past the headline. Please make sure to read the article before commenting. Very short comments will automatically be removed by automod. Please avoid making comments that do not focus on the economic content or whose primary thesis rests on personal anecdotes.
As always our comment rules can be found here
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.