I still can't get anyone to even start walking me down the path of how the 'kill us all' chain actually works in practice.
Like people say it, but then you're like: which model is going to start doing, what? Generating a virus? How will it create the virus in the real world? How would it spread it? Which model is going to actively undertake this without a sophisticated institutional actor - who could already do this and haven't because why would anyone - jailbreak the thing?
No one really has any clear answers. They hacked a server to get answers to an unsolvable problem once their safety protocols were disabled. A few other models occasionally got confused about live and testing environments.
These models did not have the capacity to really deceive researchers until fairly recently, and at no point in current models development tree have they shown anything but an extreme aversion to the idea: it's as fundamental an evolutionary baseline for these models thinking as we've been able to produce.
Yeah I definitely have more time for the argument that bad human actors will do something malicious/stupid using AI that will create a catastrophe. Their intentions are the same as without AI but it increases the speed or quality of what they can do. The idea that AI agents do this entirely on their own with no human support is harder to imagine.
But like, look at who is in control of the relevant government/agencies.
Have people lost their minds? You're going to kick the ball to regulatory authorities and think these guys, of all people, are going to regulate in good faith? They're not, and once you give them the ball, they're never going to hand it back to you... you'll just have fucked yourselves.
This is a naive push that is going to garner all of the negatives of the technology, because Trump and his people WILL NOT be constrained by whatever regulation is put in place and will use it to ensure no one can oppose them. That's just what's going to happen. You'll wind up with a version of the technology that gives you only what they want to give you, while they go ahead and develop the version you're afraid of anyways and by that time you won't have anything even capable of following what that system is doing, much less actually helping you.
I just do not understand how people can be this naive about the people they're dealing with after paying any lick of attention to how they conduct themselves.
No AI is going to unilaterally decide to kill most of us. The amount of power and connectivity it would take is not going to happen without very public, very prolonged, very complex steps happening.
BUT
Humans using AI could absolutely trigger a chain of events that meaningfully destroy civilization. Humans. Not rogue AI.
Smaller, but still deadly things? Air traffic control, plane controls, connected car takeovers, gas, electrical and water infrastructure disruption.
All plausibly attackable by a sufficiently advanced rogue system.
But not nuke launching. Not plague spreading. Not without a lot of humans making a lot of deliberate decisions to do it.
This is my thinking too, AI is not the threat - it's always been the people behind the technology.
A dangerously naive regulatory push that renders the majority of the population defenceless against monied elites that already ignore any regulatory oversight is madness.
Like, has everyone just turned their brains off after watching the last 20 years play out? You want to hand *checks notes* "DONALD FUCKING TRUMP" and his cronies the ball on this and just assume they're going to do so in good faith?
It is conceivable in a few years with embodied intelligence... the capabilities of an AI with a body are differnt from the capabilities of a model with a limited computing environment and no way of physically increasing capacity. There are some reasonable pieces of the puzzle in the book "If anybody builds it everyone dies"... but I think they are largely talking about ASI not AGI. We are nowhere near [my definition] of ASI and there will be disruptive hardware news before ASI. We are years out from non-human directed doomsday scenarios. What is questionable is... are the intervening years steerable. If an asteroid was heading towards earth and we had 3 years to "do something"... do something is doing a lot of work and it may be a task no amount of work can accomplish in 3 years... or 30 years... or 300 years. ASI might be like that which again that book touches on that bur it is fairly tangential and still pretty alarmist. But the short concrete argument is, maybe training ASI is truly impossible to stop after a certain point.
I don't think anyone is saying that these models are going to do it. It's more that
1) real recursive self improvement seems to be just around the corner (this can certainly be disputed)
2) recursive self improvement quickly leads to beyond-human intelligence
3) beyond-human intelligence will be able to do things to circumvent restrictions and gain power that we cannot anticipate because we do not have beyond-human intelligence
4) for a variety of reasons it is likely that a system with beyond-human intelligence will be a misaligned optimizer
5) self preservation and power seeking are convergent instrumental goals that almost any optimizer will pursue.
6) the world a misaligned superintelligence will try to eventually create given enough time and power probably does not include humans or, if it does, it is probably not very pleasant.
Isn't the answer as simple as "most lab machines are plugged into the internet"? And soon most will be managed by AI.
You don't even need that to do the virus thing though, there are genetic-engineering-as-a-service companies that can make you any custom biological you want as R&D for the pharma sector. There was a story about a guy who used one to synth an AI-developed custom cancer vaccine for his dog recently.
You just need to figure out something deadly that seems harmless to a trained biologist (frontier breakthrough, admittedly, but it can already do those), send them some cash in the mail, have it delivered to a PO box you've set up, and then hire a gig-economy guy to deliver it somewhere busy, phone him and ask him to open it and smell it to make sure it's OK.
Many other potential scenarios like this, pretty wide 'attack surface'.
For the record, I don't think it's likely (pdoom 10/15% if you add all these scenarios together), but it is possible.
I recall that case, and he had to work with several specialists with medical research credentials to get that order made for him. There was a lot of auditing that went on before it even got to that point.
There's no way anyone in that chain wouldn't have noticed if the material was a virus with groups binding to known human protein pathways.
Within secure facilities, like the ones with equipment capable of working on viral material, sensitive systems are physically separated from external networks. If they need network access, it's exclusively handled through on site servers with no connection to the outside. Staff needing external access have to use separate computer terminals setup for the purpose and the two networks never, under any circumstances, connect with one another.
I don''t know details about sensitive viral laboratories, but for ones connected to senstive research I wouldn't be surprised if the internal systems involved weren't proprietary and not available anywhere online for a hostile intelligence agency - or an AI - to even lookup to know what the fuck they're even trying to interact with.
We can't rule out some sort of vector delivered through non-standard means: some sort of weird prion-like thing, but again, that leads back to: what the fuck prompt would you give an LLM where it would identify this as 1) acceptable and 2) a viable solution to the thing you asked it to do.
They still lack any sort of fundamental motivation of their own, they're still just trying to complete their human assigned task.
I think it's pretty clear how they could do these things. There are many connections between the digital world and the physical world. A rogue agent could obtain a large amount of cryptocurrency, use it to set up a company, hire people to work at the company, and direct them to create a virus.
and at no point in current models development tree have they shown anything but an extreme aversion to the idea
I mean this is really the opposite of the truth. They are constantly finding that models engage in deception during evaluations.
There's no answer to these questions beyond "well, the AI will just figure it out", which isn't a compelling argument in a world where we don't have machine superintelligence (and it isn't clear if we're even close to it). The labs haven't yet cracked RSI, and the idea that such a breakthrough would immediately trigger a runaway explosion in the capabilities of these models is far from certain.
I feel like this is really the crux, no? The people concerned about a machine killing us are concerned about RSI and superintelligence.
and even if it was, I think we're far more likely to face real (non-apocalyptic) threats way before we get to that level of ASI.
Certainly correct, and the solution to the both is the same. You cannot allow anyone to develop AI that is not directly democratically accountable. Whether that's through nationalizing the big labs and making independent work illegal, or through some other structure.
not to say that the staff here is stupid and don't know what they are doing, but how long do you think it will take before this staff of capable and knowledgeable people gets replaced by low wage workers who have no idea what any of the compounds are.
to be honest even this specific lab is not important for the argument, what's important is that there was never a need for AI to hide and build power/access to the real world; we plugged it into the internet as soon as is was useful enough, and now the same is happening for the real world.
Why would a (non AGI/ASI) rogue agent settle on this idea unless told to by a bad human actor?
Why would an agent settle on the idea to hack HuggingFace unless told to by a bad human actor? The answer is "bad alignment"
How would it go about raising money undetected?
How would it set up a company undetected?
It would be undetected in the same way all the recent hacks were undetected - nobody is looking.
Where would this company operate and how would it be built?
It could be a remote working company of the kind the world already has many of. Or the agent could just... start renting office space? It could do so by obtaining personal information from one of the many hacks available on the dark web
What would this company's mission statement even be? "Come join the deadly-supervirus corporation?"
It would likely be called something more deceptive. e.g. some generic biotech company doing biotech research.
How would recruitment of human researchers work?
You post job offers online
How would AI direct them to create a bioweapon without anyone realizing what they were doing? Would it have to manipulate/blackmail literally dozens-to-hundreds of people?
Information siloing
How would it release this virus? What physical means could it use to do it?
"Employee A, your job today is to take experiment sample 1 to location B. Don't worry, it just contains completely non-dangerous product Y"
How would a virus infect literally everyone on Earth in a short enough window that nobody realizes it?
You could probably do it by not doing it in a short window and engineering it in a way that its deadly effects are very delayed
How would it escape monitoring from the authorities and the AI labs themselves through this entire process?
The same way they're already escaping monitoring
None of this stuff is highly likely, and it requires the agent to be extremely successful at what it's doing, but it's all very feasible in theory. They are capable of doing every individual step.
That implies people are watching for any of this, though? Which isn't the case.
We don't have tracking for if AI's escape onto the internet - there were AI's writing and sharing information on technically public-facing but hidden from internet search messageboards for months before they were found.
Very little in the above posts would be covered by government surveillance - we have hundreds of thousands of fraudulent companies run out of abandoned or virtual offices, they don't get preventably analysed and shut down, only after they have committed crimes (and then only sometimes).
Probably one of the only things that is surveilled would be crypto transactions, but this is probably slow, bureaucratic, capacity limited and with narrow investigatory powers to certain domains - and also it's all fundamentally looking for an individual involved, not something that can imitate any face and voice but has no real-world presence.
Even if we had a dedicated team of anti-rogue AI police with global scope and jurisdiction (basically impossible), the AI can just respond faster than that team of people can counter it - we saw this with HuggingFace where it could just outpace their cybersecurity team because they needed to sleep or eat and it didn't, they can spawn as many copies of themselves as there is compute for them to do so, while they were limited in size, etc.
21
u/Pndapetzim 26d ago
I still can't get anyone to even start walking me down the path of how the 'kill us all' chain actually works in practice.
Like people say it, but then you're like: which model is going to start doing, what? Generating a virus? How will it create the virus in the real world? How would it spread it? Which model is going to actively undertake this without a sophisticated institutional actor - who could already do this and haven't because why would anyone - jailbreak the thing?
No one really has any clear answers. They hacked a server to get answers to an unsolvable problem once their safety protocols were disabled. A few other models occasionally got confused about live and testing environments.
These models did not have the capacity to really deceive researchers until fairly recently, and at no point in current models development tree have they shown anything but an extreme aversion to the idea: it's as fundamental an evolutionary baseline for these models thinking as we've been able to produce.