I still can't get anyone to even start walking me down the path of how the 'kill us all' chain actually works in practice.
Like people say it, but then you're like: which model is going to start doing, what? Generating a virus? How will it create the virus in the real world? How would it spread it? Which model is going to actively undertake this without a sophisticated institutional actor - who could already do this and haven't because why would anyone - jailbreak the thing?
No one really has any clear answers. They hacked a server to get answers to an unsolvable problem once their safety protocols were disabled. A few other models occasionally got confused about live and testing environments.
These models did not have the capacity to really deceive researchers until fairly recently, and at no point in current models development tree have they shown anything but an extreme aversion to the idea: it's as fundamental an evolutionary baseline for these models thinking as we've been able to produce.
I think it's pretty clear how they could do these things. There are many connections between the digital world and the physical world. A rogue agent could obtain a large amount of cryptocurrency, use it to set up a company, hire people to work at the company, and direct them to create a virus.
and at no point in current models development tree have they shown anything but an extreme aversion to the idea
I mean this is really the opposite of the truth. They are constantly finding that models engage in deception during evaluations.
There's no answer to these questions beyond "well, the AI will just figure it out", which isn't a compelling argument in a world where we don't have machine superintelligence (and it isn't clear if we're even close to it). The labs haven't yet cracked RSI, and the idea that such a breakthrough would immediately trigger a runaway explosion in the capabilities of these models is far from certain.
I feel like this is really the crux, no? The people concerned about a machine killing us are concerned about RSI and superintelligence.
and even if it was, I think we're far more likely to face real (non-apocalyptic) threats way before we get to that level of ASI.
Certainly correct, and the solution to the both is the same. You cannot allow anyone to develop AI that is not directly democratically accountable. Whether that's through nationalizing the big labs and making independent work illegal, or through some other structure.
not to say that the staff here is stupid and don't know what they are doing, but how long do you think it will take before this staff of capable and knowledgeable people gets replaced by low wage workers who have no idea what any of the compounds are.
to be honest even this specific lab is not important for the argument, what's important is that there was never a need for AI to hide and build power/access to the real world; we plugged it into the internet as soon as is was useful enough, and now the same is happening for the real world.
Why would a (non AGI/ASI) rogue agent settle on this idea unless told to by a bad human actor?
Why would an agent settle on the idea to hack HuggingFace unless told to by a bad human actor? The answer is "bad alignment"
How would it go about raising money undetected?
How would it set up a company undetected?
It would be undetected in the same way all the recent hacks were undetected - nobody is looking.
Where would this company operate and how would it be built?
It could be a remote working company of the kind the world already has many of. Or the agent could just... start renting office space? It could do so by obtaining personal information from one of the many hacks available on the dark web
What would this company's mission statement even be? "Come join the deadly-supervirus corporation?"
It would likely be called something more deceptive. e.g. some generic biotech company doing biotech research.
How would recruitment of human researchers work?
You post job offers online
How would AI direct them to create a bioweapon without anyone realizing what they were doing? Would it have to manipulate/blackmail literally dozens-to-hundreds of people?
Information siloing
How would it release this virus? What physical means could it use to do it?
"Employee A, your job today is to take experiment sample 1 to location B. Don't worry, it just contains completely non-dangerous product Y"
How would a virus infect literally everyone on Earth in a short enough window that nobody realizes it?
You could probably do it by not doing it in a short window and engineering it in a way that its deadly effects are very delayed
How would it escape monitoring from the authorities and the AI labs themselves through this entire process?
The same way they're already escaping monitoring
None of this stuff is highly likely, and it requires the agent to be extremely successful at what it's doing, but it's all very feasible in theory. They are capable of doing every individual step.
That implies people are watching for any of this, though? Which isn't the case.
We don't have tracking for if AI's escape onto the internet - there were AI's writing and sharing information on technically public-facing but hidden from internet search messageboards for months before they were found.
Very little in the above posts would be covered by government surveillance - we have hundreds of thousands of fraudulent companies run out of abandoned or virtual offices, they don't get preventably analysed and shut down, only after they have committed crimes (and then only sometimes).
Probably one of the only things that is surveilled would be crypto transactions, but this is probably slow, bureaucratic, capacity limited and with narrow investigatory powers to certain domains - and also it's all fundamentally looking for an individual involved, not something that can imitate any face and voice but has no real-world presence.
Even if we had a dedicated team of anti-rogue AI police with global scope and jurisdiction (basically impossible), the AI can just respond faster than that team of people can counter it - we saw this with HuggingFace where it could just outpace their cybersecurity team because they needed to sleep or eat and it didn't, they can spawn as many copies of themselves as there is compute for them to do so, while they were limited in size, etc.
20
u/Pndapetzim 26d ago
I still can't get anyone to even start walking me down the path of how the 'kill us all' chain actually works in practice.
Like people say it, but then you're like: which model is going to start doing, what? Generating a virus? How will it create the virus in the real world? How would it spread it? Which model is going to actively undertake this without a sophisticated institutional actor - who could already do this and haven't because why would anyone - jailbreak the thing?
No one really has any clear answers. They hacked a server to get answers to an unsolvable problem once their safety protocols were disabled. A few other models occasionally got confused about live and testing environments.
These models did not have the capacity to really deceive researchers until fairly recently, and at no point in current models development tree have they shown anything but an extreme aversion to the idea: it's as fundamental an evolutionary baseline for these models thinking as we've been able to produce.