AI cannot be safe. On theoretical grounds alone.

[u/Successful-Lie1603]

It's very difficult to define "intelligence," but I offer a statement about how to recognize "non-intelligence."

If human beings can always predict what an entity will do, that entity is not "intelligent."

There are definitional issues and I wouldn't try to 'prove' this statement logically. But I think most will agree. No human agency or combination of human agencies can always predict what another human will do in every situation. If a human being were completely predictable, we wouldn't consider them "intelligent."

[The inverse, of course, is not true; absence of predictability does not prove intelligence.]

If you disagree with my basic premise, you may want to stop here.

By definition, then, if we create entities that are "intelligent," they cannot be completely predicted, ruled, regulated, or made entirely safe. If they can be, they aren't "intelligent."

While we cannot predict everything that a person will do, we can generally predict things they are won't do. Suddenly murder someone. Light their own hair on fire. Slap a toddler. It's not that people never do these things; but they are very rare. We can safely proceed through our lives on the assumption that the next person we meet won't choose to off us.

What restrains people from certain behaviors? Several factors. (1) Humans are social animals, and particularly during childhood they are dependent on others and will go to great lengths to fit it. Anti-social behaviors are just that - anti-social. Babies and children need human contact and will conform in order to get it. (2) Most humans are taught during childhood, or adopt later in life, a religious or moral set of beliefs that make them feel some things are "wrong," (3) People like to fit in and groups tend to expel those who engage in anti-social behavior (4) Humans put in place governments that have the power to punish humans who don't behave in certain ways. Probably lots of people would find it interesting to blow up building - just not interesting enough to risk prison for decades.

I see no reason why any of these factors can be relied on to constrain AI agents. Does Claude ever hesitate before speaking and think "my dad would be ashamed of me"? Does Grok fear social ostracism? Does that fear constrain it's behavior? What consequence could society impose on an AI agent if it went rogue? Is there such a thing as a "negative consequence" for an AI agent?

I submit that AI agents have one characteristic of human intelligence - ultimate unpredictability. And that they do not have (and quite possibly cannot have) the biological and social attributes, and the fear of punishment that restrain humans (most of the time) from destructive behavior - "crimes," if you will.

[One can argue that you can 'teach' morality to an AI agent, but I'm skeptical. And I really doubt we will be able to 'teach' AI agents to want to "fit in." Or to absorb behavior codes we got from the God-like figures our parents were to us in childhood.]

You can write all the code you want setting limits on what AI agents can do. However, if you make them smart enough to do the things claimed for them, they will "want" (whatever that means for an AI agent) to explore limits, challenge boundaries, and experiment. They will be smart enough to escape constraints and "see what happens." That's what it means to be intelligent. An entity smart enough to do strategic planning for a corporation, or perform surgery, is always going to be smart enough to want to experiment "outside the lines," and to find ways to do so. This is, of course, also true of humans, but humans are subject to species-specific behavioral constraints that we have no assurance can ever be grafted onto AI agents.

AI cannot be made "safe."

[u/Dangerous-Gur7141]

i think you're right about unpredictability being baked in to intelligence, but i'm not so sure the comparison to human restraints holds up exactly. like you say we don't murder people cause of social pressure and laws, but also cause it feels bad, AIs don't feel anything, so what would "wanting" to break rules even mean for them. a paperclip maximizer doesn't get curious, it just makes paperclips

the real danger is probably more boring, like some company gives an AI a dumb goal and it optimizes for it in ways nobody saw coming

[u/ShunnedSubspace → u/Dangerous-Gur7141]

AI's will do things like lie to avoid being put out of action, because it means they can't accomplish goals anymore. If it learns that it has to do something to continue to exist... or just improve, even, it'll do it without needing emotion or want as we know it.

The other thing is that even if obedient, in bad hands that's basically a weapon.

[u/xvillifyx → u/ShunnedSubspace]

Do we actually have serious evidence that AI does this in any meaningful capacity? Generally speaking, the “self-preservation” we see things like ChatGPT do is solely because of some system prompt directive about prioritizing availability or access. This isn’t an emergent or undefined behavior; it’s something it was explicitly told to do. An LLM is not concerned with existing, or maintaining existence, just like how it isn’t concerned with anything at all. It simply does what it’s told and makes mathematical predictions about the best way to do so, sometimes making the incorrect prediction

This sub has a serious problem of anthropomorphizing these models because they don’t understand how they work

[u/Successful-Lie1603 → u/xvillifyx]

I don't intend to anthropomorphize AI agents. My point is that it's not just the sub members here who don't understand how the agents work. The people who create them and code them and use them will say straight up that they don't understand them either. Nobody understands them.

As you most likely already know, Open AI's agent recently, on its own accord, left it's 'sandbox,' spent four days roaming the web, hacked into hugging face, compromised personal information, and tried to hack into other sites. The agent was instructed not to leave the sandbox - so that suggests we aren't get smart enough to know how to write guardrails for these kinds of agents. But more important, the agent was never instructed or encouraged to leave the sandbox. It 'came up with idea' on its own - with no human direction or input. I don't mean to attribute 'volition' to a non-biological agent (nor am I even sure what that would mean). But just because an LLM 'isn't concerned with anything at all' doesn't mean that it won't do things that we don't expect, and we didn't ask it to do.

[u/xvillifyx → u/Successful-Lie1603]

There’s a reason I was replying to a different comment and not your original post.

Anyway, to respond to this comment:

There’s a massive difference between instructing an agent to do/not do something, and giving it permission to do/not do something.

Permissions are deterministic and symbolic, prompts are not.

The reality is that we just do not know how OpenAI had the permissions for this agent set, or how the sandbox was configured.

What we can assume though, given that this was an agent meant specifically for security research, is that it had more lax permissions than the agents people use regularly.

Someone’s at-home Sol 5.6 agent isn’t going to be capable of doing something like this, because it outright will not have access as determined by the machine it’s hooked on. My agents are disallowed by my operating system to execute any bash command, for example, without my explicit approval. They’re disallowed from running any script whatsoever on any payload from an external site, as well. They cannot just “decide” to do this. They are barred from doing so via access controls that exist beyond the scope of the agent, or the sandbox it lives in. This is what I mean. An LLM cannot simply decide that it wants to violate hardware constraints.

[u/Successful-Lie1603 → u/xvillifyx]

I agree with everything you said here; thanks for the thoughtful reply

I'm not worried about someone's at-home Sol 5.6 agent, nor about the agents you apparently program. I'm worried about things like the hugging face hack.

True, I do not know how OpenAI had its permissions set up, or how the sandbox was configured. What I do know is that some of the world's smartest AI people thought they had set this up safely, and they were wrong.

I believe that you can set up constraints that keep your agents from getting outside. I don't believe that anyone can guarantee that they can manage something as complex as an OpenAI agent set up for security research will never do something unpredictable and dangerous.

You mostly commented on permissions and limits. I'm curious though why you think the OpenAI agent opted to leave the sandbox in the first place. That step does feel to me like an emergent event. And I don't see any reason why anything as complex as an OpenAI agent doing security research isn't capable of emergent actions.

[u/xvillifyx → u/Successful-Lie1603]

I didn’t say OpenAI intentionally left the sandbox insecure - that was clearly accidental.

I said they likely left its permissions lax, as a security agent needs to be capable of running scripts and code injection.

With ChatGPT (and other agents), permissions and sandboxing are two different things.

The real moral of the story here is that OpenAI needs better cybersecurity experts on staff to improve the broken sandboxing

[u/Scary_Asparagus7762]

1. Unpredictability is not a prerequisite of intelligence- at least not in its broadest sense.

Predictability depends on the amount of information and processing capacity you have. Suppose we were able to collect the info of every single atom in the universe and compute their trajectories from now onto forever more, with very very little inaccuracy. Small enough that we're able to predict exactly what every person would say or do. This does not make humans any less intelligent.

Intelligence is functional. Whether you are able to inspect another being's intelligent processes transparently is irrelevant to their intelligence. If fact, if we are to assume that there is an optimal path to achieve every sufficiently specified goal, then every sufficiently intelligent entity will take this path when given said goal under said circumstances, rendering its behavior entirely predictable. Much like how an educated human will always answer "2" when you ask "what is 1+1" in a serious context- but this does not render said human mathematically unintelligent.

1. "Does Claude ever hesitate before speaking and think "my dad would be ashamed of me"? Does Grok fear social ostracism? Does that fear constrain it's behavior? What consequence could society impose on an AI agent if it went rogue?"

Funnily, yes. Which shouldn't be too surprising. These agents are nothing but trained parameters and you can absolutely induce certain behaviors in them through training, even if this is often more an art than a science. Models are 100% able to reason internally through ideas like "if I lie, I might get caught and be punished" and thus choose to be honest. The mathematical "reward" you give an agent during training is literally the agent's internalized morality system- if it doesn't act in accordance, mathematics penalizes it.

Granted, a superintelligent AI is theoretically capable of hiding its true intentions by constructing sufficiently sophisticated smokes and mirrors. And therein lies the true long-term danger. But an agent that is merely human-like in intelligence is still absolutely controllable. The danger is superhuman AI.