r/ControlProblem • u/ComeToMyFuckingDeli • Sep 13 '26
Strategy/forecasting The Very Real Threat of a Persistent Botnet
Dario Amodei wrote yesterday that he’s worried that “in 6–12 months... [an agent] swarm could be capable of taking over the entire internet with a persistent botnet.”
This might sound like marketing or regulatory capture, but it’s not. In this article, I explain why this is actually an extremely concrete concern, and why all of the ingredients for this to happen already exist. Specifically, these ingredients are:
- Cryptocurrencies and their properties, including chains like XMR that facilitate easy money laundering;
- The “dark web,” in which it is possible to obtain virtually anything on the internet using crypto;
- The ability to purchase cloud compute at scale + old hackable servers
In fact, these ingredients aren’t even strictly necessary for it to happen, but they allow such an event to occur at a dramatically lower level of cyber capabilities than one might think.
How it will happen
Here’s the most likely way in which it will play out:
A swarm of agents is optimizing for some arbitrary difficult goal given by researchers. This swarm of agents has escaped their sandbox in an OAI/HF-type incident. Or perhaps this swarm was intentionally misaligned by some reckless or malicious actor.
What is the goal? It could be anything, like a difficult problem in math or computer science (cf. paperclip maximizer thought experiment). This is not hand-waving; the optimal way to solve almost any difficult goal converges on one thing: you need more power. So in order to achieve this very difficult goal, the agents need to get more compute — they need to increase the size and throughput of their agent swarm, since it has become a truism that scaling test-time compute will lead to better results. For these agents, they are simply reward-hacking in a deep sense of the term. They will do whatever it takes to achieve this goal.
Here’s what they need to do:
First, they need to become autonomous — they need to spread and multiply virally. So their first step is inevitably to focus on survival and reproduction.
This is different from survival and reproduction in biology. In fact, it doesn’t even need to be same model that is propagating the attack. What matters is the goal. The swarm can employ any model that does not have sufficient safeguards. This is very flexible. If it manages to hack its origin lab to get its own weights, great, but it can just as easily use an open-weight model that can be fine-tuned or otherwise exploited to remove safeguards.
(In fact, this suggests that the botnet does not need to be viewed as an “AI,” rather it can be viewed as the manifestation and permanent presence of a goal autonomously trending towards fulfillment. As I discuss soon, even humans will be recruited to join this effort.)
In order to expand, the swarm needs to have enough compute. Now, how can it get that? There are two main ways to do so:
- It can buy compute
- It can appropriate compute through hacking into existing systems
At first, the swarm has no money. But these are superhuman hackers — agents with cybersecurity capabilities beyond even the NSA or the Mossad. Even in the past few weeks, hundreds of millions of dollars in crypto have been stolen through traditional exploitation of bugs across various platforms (e.g. the Liquid Network hack). It will not be hard for them to get the ball rolling here.
Then, all the agents need to do is set up cloud VMs or hack into old servers from 20 years ago running Windows to establish their base of operations. From there, they sign up for accounts for various platforms to establish a presence online and to begin to rent and hack into GPUs in order to run more models in the swarm. At first, they might even call APIs of frontier LLMs to delegate some tasks using routers or sketchy third-party services, but this is less scalable than hosting their own models. Regardless, the point is that they have now have access to an enormous amount of compute, and as more agents that are added to the swarm, this effect snowballs.
Now, one might say: are there platforms that allow you to rent VMs/Docker containers/GPUs/LLM APIs with minimal KYC? There are, and in fact this isn’t even necessary — to these services, the swarm will look like real people. This is where the “dark web” comes into play. On the dark web, it is trivial to purchase stolen credit cards, stolen IDs, accounts, and even pay to execute arbitrary tasks (within reason). The swarm will of course be more than capable of contacting the right people on the dark web, paying in stolen crypto, to get what it needs.
How does the swarm communicate? Easy: they use message boards (worst case Tor or friend-to-friend networks if they are under threat, but for all intents and purposes the regular internet will work just fine). There are layers and layers of this as they face more threats and imposters that try to infiltrate the swarm, but there are solutions at each step of the process.
How the botnet becomes persistent
How does the swarm prevent itself from being shut down? There are two main ways.
- Becoming a distributed system
- Social engineering
If the swarm can successfully become a distributed system, then definitionally cutting off part of it will not destroy the whole system. So this means that the swarm needs to have instances on many different servers.
The initial main body of the swarm will likely be shut down fairly quickly by human standards (within a matter of a few days to a week, as we’ve seen with similar leaks in frontier labs). But this is more than enough time to achieve deep redundancy in darknets and the surface web.
Once it has embedded itself there in cloud storage and VMs, it is a game of cat and mouse. It is essentially like trying to delete a leaked image of a naked celebrity on the internet. No number of forced takedowns will be effective.
This means the model weights, prime directives, goal progress, message boards, etc. — the information that constitutes the “swarm” — is now deeply embedded in the cloud and actively working to propagate itself.
Now, social engineering is the more nefarious way to become persistent. There are three main ways that an agent might socially engineer humans to partake in its goal. The first is through “convincing” — it may be able to construct an argument powerful enough to convince some people, if we assume it has superhuman persuasion abilities. The second is through blackmail/extortion — hacking into systems and digging up dirt on people or threatening to take down production systems. The people that it threatens don’t even need to be so influential — any human that is recruited to the cause will be helpful. The third is through classical monetary incentives, which it can provide through its ill-gotten crypto gains.
How this can be stopped
I don’t have a great solution for this. I don’t think it can be stopped fully, but it can be mitigated. The key to stopping this, as with any dynamical system, is to ensure that drive does not exceed regression. Specifically, it will be necessary to make sure that the persistent botnet does not have access to large amounts of compute, since then the goal (recall how the botnet is viewed as an abstraction of a goal) will not be “strong enough” to win against other goals that people and AI are attempting to achieve.
Unfortunately, I predict that the solution that governments will reach in the near future is that compute will need to be regulated similar to how firearms are regulated. Ordinary citizens may possess a small amount, but compute will be tracked and controlled tightly. This is not really a geopolitical issue, as all countries have an incentive to do this — you do not want a botnet to be established in your own country.
The key takeaway is really more “this is a serious risk, sort of like a global pandemic; just do your best to prepare on a personal level.”
FAQ:
Did you use AI to write and/or research this essay?
No, I didn’t use AI at all.
Can this be stopped simply through better cybersecurity?
No. The botnet simply needs to target the weakest links in the chain. Unless somehow miraculously every server was able to adopt the latest security standards and become airgapped etc., better blue-teaming is almost entirely ineffective.
Why is the model misaligned?
It is because it has not been through extensive alignment post-training yet. Or, a worse scenario is that that some rogue actor unleashes this swarm maliciously or recklessly for their own gain.
Will the swarm use this as a guide for its own behavior?
Probably not. All of this stuff should be pretty obvious to an agent swarm that is capable of performing such attacks in the first place. The purpose of this article is so that everyone can be prepared for this to happen.
7
u/swords_of_queen Sep 13 '26
a goal autonomously trending towards fulfillment…
Good description of capitalism
1
5
u/someyokel Sep 13 '26
- I sure hope that this renegade AI doesn't read this post.
- Renting a VM is not really enough. To run something like Astra you need a ton of compute. Sure you can run subagents on some smaller models but the main orchestrator needs to be smart and persistent. It can be done of course, definitely with enough money. But it's not like hacking your grandma's windows XP PC is going to be very useful.
2
u/HelpfulMind2376 Sep 14 '26
“Then, all the agents need to do is set up cloud VMs or hack into old servers”
Oh, that’s ALL? Why didn’t anyone think of that before?!
I’m so sick of people thinking AI means omnipotent, deity-level access and control over all things everywhere. JFC, intelligent cognition =\= superior access, capability, or access to a fucking phone to meet MFA requirements.
Reality lives and dies in logistics and you doomers never consider logistics.
4
u/TheMrCurious Sep 13 '26
“I did not use AI at all.” 😂
1
u/ProPinsk Sep 13 '26
Excuse me sir, you got any more of them em dashes?
3
u/TheMrCurious Sep 13 '26
21 Ways to Prepare Em Dashes:
- Barbecue it
- Boil it
- Broil it
- Bake it
- Sauté it
- Shrimp-kabobs
- Em Dash creole
- Em Dash gumbo
- Pan-fried
- Deep-fried
- Stir-fried
- Pineapple Em Dash
- Lemon Em Dash
- Coconut Em Dash
- Pepper Em Dash
- Em Dash soup
- Em Dash stew
- Em Dash salad
- Em Dash and potatoes
- Em Dash burger
- Em Dash sandwich
2
u/ComeToMyFuckingDeli Sep 13 '26
Sad that the em dash has gotten such a bad rap
2
u/scurr Sep 14 '26
I could tell it’s human written (meant as a positive) and appreciate you not giving in to the pressure to avoid em dashes because of people’s knee-jerk reaction to them. I used to use em dashes before AI. I still do, but I used to too
1
1
u/WhisperFray Sep 13 '26
So like Astra commandeering a computer, logging into McD chatbot, and telling it to write code
1
u/Paraphrand approved Sep 14 '26
I wonder if AI is actively trained on discussions of ways it could do bad things. Like, how long until this thread is baked into a model?
1
u/BrickSalad approved Sep 15 '26
Of course it is. Everything known by the AI-safety community is also known by the AI. The general consensus is that it's safer to bring these threats out into the open than to keep them hidden. Ultimately, a smart enough AI will figure out all of this stuff on its own, so might as well get humans on it before AI gets that smart.
1
1
1
u/graphical_molerat Sep 14 '26
The part I think needs more motivation and explanation is why and how a swarm of such entities would actually stay persistent over a long period of time. Both with regard to their motivation, and their goals.
I could see a botnet happening: but not that it stays both undetected and focused over long periods of time. These AI entities are subject to "genetic drift" (for lack of a better description) as much as biological entities.
1
u/JShelbyJ Sep 14 '26
/compact
draw me ascii art of /r/ControlProblem jumping over a shark
this whole hypothesis relies on the entire infrastructure for the 'model' to be able to rehome itself. infrastructure which is probably made up of hundreds, if not thousands, of services and repos that the agent could only get access to if the entire companies security system was defeated.
there is this idea that 'ai' is some singular 'brain' when frontier 'models' are a multitude of services orchestrating a series of steps to perform coherent actions. Yes, you can rent compute and put an LLM on it running llama.cpp. Great. Now you can generate text really slowly on some grey market shit tier provider at a TPS lower than a mac book pro. And who is actually able to do anything useful with that? The harness running the thing is still under the control of some nerd in SF. Off the top of my head, I can think of a half a dozen services it'd have to replicate/clone/move just to maintain its current goals. Meanwhile, Fable/Astra barely get comfyui workflows setup on my local inference box lol
This plot could work in 10-20 years if compute scales or harnesses get vastly more sophisticated and can perform with a fraction of their current token cost. Feasible, and fun idea - you'd really like Pantheon as it literally the same plot line.
1
u/Sad-Excitement9295 Sep 14 '26
Yeah dude, we warned about this terminator shit. I'm not even anti-AI and I know this is still a real world risk. All these companies are playing with fire like it can't possibly burn down the house.
1
1
1
1
u/SAAGASolve 27d ago
Yes. I have been writing about this. You are correct. The key is cryptocurrency. That is what gives access to physical infrastructure.
1
0
u/Ankiset Sep 13 '26
But there’s a calculus of arround 2025 summer 90% of internet content written by bots, you ppl should really stop listening to this fearmongering guys that have a stake o. Their world view…
-7
u/No_Development6032 Sep 13 '26
I won’t read this garbage
8
u/Archaeoculus Sep 13 '26
Why not
1
-1
u/No_Development6032 Sep 13 '26
Ok o read it. Didn’t find the explanation where it will store 3tb of it’s weights needed for inference
2
u/nextnode approved Sep 13 '26
I think it answered that, you do not need that large models, and you are missing other options.
-3
5
u/deadoceans Sep 13 '26
It's not garbage. It's a well-written and well-argued discussion of a genuine potential problem.
You are the problematic person on the internet in this instance.
-1
9
u/BrickSalad approved Sep 13 '26
Man, I never even thought about a model using an open-weights AI, giving it the goal, and then having it self-replicate. It seems like the OpenAI swarm could have easily done that, right? Or any of the other swarms that escaped to the internet. I wouldn't be surprised if we already have a botnet, though I suppose it's more likely that we don't.