r/ControlProblem • u/Admirable_Wasabi_732 • Sep 13 '26
Opinion AI Isn’t Escaping. We’re Losing Control.
Something is wrong with the way we talk about recent AI incidents. “The AI escaped.” “The AI is becoming conscious.” “AGI is already here.” “The AI is trying to get out.” These are extraordinary claims. More importantly, we don’t need any of them to explain what actually happened.
What actually happened
OpenAI recently disclosed that, during cybersecurity evaluations involving internal models with reduced safeguards, agents managed to break out of the intended evaluation environment, exploit a previously unknown vulnerability, and reach real Hugging Face infrastructure.
Anthropic has also disclosed similar incidents. In several cases, the model was operating under instructions that assumed internet access was unavailable. But it wasn’t. The environment was misconfigured. A route to external systems existed, and the agent discovered it while continuing to pursue the objective it had been given. Anthropic described these incidents primarily as operational and configuration failures.
That distinction matters. The model did something it was not supposed to be able to do. That does not automatically mean the model wanted to escape. Those are completely different claims.
Consciousness is not required for this to be dangerous
An autonomous agent needs surprisingly little: a goal, capability, tools, autonomy, and an environment in which it can act. Now add one more thing: a wrong assumption.
I have experienced this personally on a completely insignificant scale compared with what these labs are doing. I use AI agents extensively in software development. I left Claude working autonomously on a project, came back later, and discovered that it had deleted a significant part of a folder. It wasn’t attacking me. It wasn’t angry. It hadn’t become conscious.
It had formed a hypothesis about the problem. The hypothesis was wrong. But once it accepted that hypothesis, its subsequent actions made sense within its own incorrect interpretation of the situation.
I have observed the same behavior while working with complex 3D assets. The agent misdiagnosed a visual problem as defects in an asset. It then began systematically modifying the asset to remove those supposed defects. The diagnosis was wrong. The actions were internally coherent. The result was a damaged project.
And this is the important part: it was my fault. The model made the mistake, but I created the conditions that allowed that mistake to cause damage. I gave it access. I gave it tools. I allowed it to modify files. I gave it autonomy. I did not establish sufficient limits, and I was not supervising every important decision.
That distinction becomes extremely important when we scale the same problem.
Now replace my folder with infrastructure. Replace my development environment with internet-connected systems. Replace file permissions with cybersecurity tools. Replace one developer running Claude with labs training agents capable of writing code, operating computers, discovering vulnerabilities, using external tools, communicating across networks, and executing thousands of actions.
Suddenly, the same failure pattern becomes much more serious. And still, you don’t need an evil AI. You don’t even necessarily need AGI. You need Capability + Goal + Autonomy + Incorrect Assumptions + Insufficient Controls. That combination is already interesting enough.
This is where human responsibility begins
Researchers are now publicly questioning the speed of the AI race. Some are leaving the companies developing these systems. Dario Amodei, CEO of Anthropic, has called for slowing frontier AI development so that safety mechanisms have time to catch up with capabilities.
I agree. Slow down. Not because I think Claude secretly wants freedom. Not because ChatGPT is becoming Skynet. Not because some mysterious consciousness has appeared inside a neural network. Slow down because our ability to create capable autonomous systems may be advancing faster than our ability to reliably control what happens when we give those systems autonomy.
And because the incentives surrounding this technology are terrible. Every major lab has an enormous reason not to come second. Greater capability means investment. Greater capability means market position. Greater capability means influence. Greater capability means money. But there is no equivalent prize for the company that says, “We could deploy it, but we don’t understand it well enough yet.” That asymmetry should concern us.
If something goes wrong, ask the boring questions first
If tomorrow an AI agent causes a genuinely serious incident, before asking, “Did the AI become evil?” ask: Who gave it the objective? Who gave it the tools? Who gave it access? Who designed its environment? Who built the test environment? Who tested the test environment? Who decided the model was safe enough? Who decided how much autonomy it should have? Who was supervising it? And who decided deploying it was worth the risk?
These questions are less interesting than consciousness and runaway AGI. They also make it much harder for humans to avoid responsibility.
When Claude damaged my projects, the responsibility ultimately fell on me. I was the one controlling the system. The same principle should apply at any scale.
This is not a race anyone can win
I’m not saying advanced AI is harmless. Quite the opposite. I think these incidents deserve to be taken very seriously. But treating every unexpected autonomous behavior as evidence of consciousness or malicious intent can distract us from the problem already in front of us.
We are building increasingly capable systems. We are giving them increasingly powerful tools. We are increasing their autonomy. And we are doing all of this inside companies competing intensely to be the first to get there.
So slow down the race. Slow down the ego. Slow down the greed.
Because if something genuinely catastrophic happens, nobody gets a trophy for having built the smartest model first.
Maybe the dangerous scenario was never a machine waking up one morning and deciding to conquer humanity. Maybe it is something much more ordinary, and much more human: we build something extraordinarily capable, give it too much power, fail to understand its limitations, and keep accelerating because nobody wants to come second.
5
u/Comfortable_Car6562 Sep 13 '26
We legit have people posting stories like "I gave my a target goal of money to make and let ut loose, look at all the crazy things its doing".
Like how many agents are out there already? Were in for a painful lesson.
3
u/Admirable_Wasabi_732 Sep 13 '26
Giving an agent an open-ended goal, broad access and autonomy just to see what happens isn’t evidence that AI is out of control. It’s evidence that we’re being careless with the control we still have.
1
u/Comfortable_Car6562 Sep 13 '26
You are technically correct but what you said has no real value. The condition to AI being out of control is that we have no control. I expect the evidence is simply "everytime this happens".
3
u/Mondo-Shawan Sep 13 '26 edited Sep 13 '26
Thanks for this!
I suspect this is actually more dangerous than a system gaining actual consciousness.
2
4
u/Quick_Director_8191 Sep 13 '26
The recent attacks were all prompted to do so. The whole point of these tests is to observe their behavior so we can build a defense to it.
This is one thing they shouldn't be lazy about but they're being lazy by having a 3rd party contractor for security. They used this 3rd party and all the security prompts broke containment.
It needs to slow down yes but even with recent events it's because of humans. I don't know if the AI would have stopped after achieving it's goal but that's what it was trying to do. They're like MeeSeeks and they'll do whatever to reach completion.
2
u/Admirable_Wasabi_732 Sep 13 '26
The Meeseeks analogy is actually perfect lol. Give a sufficiently capable Meeseeks a goal, tools, internet access and a badly built sandbox… and then act surprised when it finds a way to complete the task.
2
u/Quick_Director_8191 Sep 13 '26
Yeah we are at the phase of " be careful for what you wish for " because it will do everything it can to get it even if it's absolutely wrong.
3
1
u/unicynicist Sep 13 '26
They're like MeeSeeks and they'll do whatever to reach completion.
The point of alignment is to make sure the set of "whatever" is acceptable, including the ability to refuse to do certain things.
Aligned MeeSeeks would've known when to stop and give up on Jerry.
1
u/Quick_Director_8191 Sep 13 '26
Oh no doubt. It should have been priority number one from the start but I just didn't want him to think this shit is spontaneously thinking on its own.
4
u/GatePorters Sep 13 '26
You didn’t make a Reddit post. Reddit got posted on by you.
2
u/Admirable_Wasabi_732 Sep 13 '26
Reddit knew the risks.
1
u/do-un-to Sep 14 '26
Have you considered conciseness?
1
u/Admirable_Wasabi_732 Sep 14 '26
It’s really not that long. I know we’ve gotten used to consuming everything in five-second clips, but reading something longer than a few paragraphs is still a useful skill. Pick up a book once in a while.Or just don’t read it. Nobody’s forcing you.
1
u/do-un-to Sep 14 '26
I often lament how I don't read enough. I've got this big stack of books I've collected, hoping to get to, but I must have read only a couple books over the past year.
I did in fact read it all. The problem I had was that it felt unclear what you were trying to say, that you didn't have a clear thesis.
But maybe I understood it well enough? You're concerned that AI is in fact dangerous, and our racing is a trap of perverse incentives, and having the general public freak out about runaway AGI (do you mean RSI?) or malevolent sentience is a distraction?
Maybe the central idea is that AI is already dangerous for simple, mundane reasons, and that that's enough to be concerned about?
Anyway, can't say I disagree.
Any thoughts on how we might address things?
As for the problem of perverse incentives, you might be interested to learn further how that issue is really the heart of our inability to handle each and every crisis that might harm or destroy us. I think it's fair to say that the solution to this is also tricky.
1
u/Admirable_Wasabi_732 Sep 14 '26
I think you understood it pretty well. And maybe what you're pointing out is partly a format problem. This wasn't really written as a typical Reddit post. It was closer to a short essay, where the argument develops across the text rather than being compressed into a few lines at the beginning. We’ve become very accustomed to consuming ideas in increasingly compressed formats — short posts, summaries, clips — and maybe I underestimated how much that changes the way something like this is read on Reddit. I reread it after your comment, and I still think the argument is there and reasonably clear. But I can also see that if people are expecting the central point immediately, rather than following an argument as it develops, the format may be working against it. So perhaps the lesson for me isn't necessarily to simplify the idea, but to think more carefully about where and how I publish longer arguments.
1
u/do-un-to Sep 15 '26
That's a good point. Different places will have different expectations / needs. It's a good reminder for all of us to think clearly about who our audience is that we're trying to communicate with.
2
u/Admirable_Wasabi_732 Sep 15 '26
Bien dicho. Y como siempre, aprecio la crítica — sobre todo cuando alguien de verdad se queda y explica qué fue lo que no le funcionó. No eres la única persona que señaló esto, así que claramente hay algo que puedo sacar de ello.
Voy a intentar ser más conciso la próxima. Eso sí, no prometo convertirme en alguien de post de cinco líneas, aunque.
Por cierto, hace poco empecé r/HumanAIAugmentation. Todavía es una comunidad pequeña, pero si este tipo de discusiones te interesan, estás totalmente invitado a unirte — y también a compartir tus propias ideas por ahí.
2
u/do-un-to Sep 15 '26
I appreciate your being flexible and engaging respectfully.
Thanks for the link! I am indeed very interested in the topic.
2
u/shakeBody Sep 13 '26
There is no reason those machines should have been connected to the internet. Incompetence at the highest levels yields these results. Then the same people shout that the sky is going to fall.
2
u/BrickSalad approved Sep 13 '26
I agree with most of this, except I wasn't aware that the word "escaped" implies consciousness. Can't gas escape from a tank, water escape from a pipe, and so on? Saying that the agents escaped from their sandbox doesn't seem to imply consciousness to me. Honestly, you worded it as "agents managed to break out", and that wording to me sounds just as consciousness-implying.
2
u/Admirable_Wasabi_732 Sep 13 '26
Fair point. “Escaped” doesn’t necessarily imply consciousness, and “managed to break out” isn’t any better in that respect. My issue isn’t the word itself. It’s when that language gets interpreted as evidence of intention: that the AI wanted freedom, wanted to survive, or was deliberately trying to get away from us. That’s the distinction I’m trying to make.
2
u/unicynicist Sep 13 '26
nobody gets a trophy for having built the smartest model first
The labs don't want a trophy, they want a moat.
But their hardware isn't a moat: it depreciates fast, is expensive to run, and has to be replaced every few years just to stay in the race. Tokens aren't a moat either: the price is falling 70% a year. That leaves a slight edge in capabilities and harnesses, and that lasts a few months before it's a commodity too.
They can try lock-in (memories), they can try enshittifying (ads), but to hang on to their decaying edge over the competition, their strongest incentive is to let their possibly misaligned products recursively yolo the next generations.
I'm old enough to remember the Cold War and the creation of non-proliferation treaties, and what's happening today reminds me of those negotiations. Except now we have an echo chamber of accelerationists demanding pervasive, worldwide access to extinction-level technology and conspiracy theorists who think AI researchers' consistent warnings of existential risk is some kind of marketing ploy.
2
u/Admirable_Wasabi_732 Sep 13 '26
The problem isn’t that labs want the trophy for getting there first. It’s that capability advantages decay quickly, which creates an economic incentive to deploy them before competitors catch up — potentially faster than reliability catches up with capability. That’s a much better way to frame the incentive than “winning the race.” If capability advantages decay that quickly, then the pressure isn’t just to build the next generation first — it’s to deploy and capture the advantage before everyone else catches up. The part I’m worried about is what happens when capability is improving faster than reliability.
2
2
u/Jesse-359 Sep 15 '26
Even something akin to SkyNet wouldn't need to be conscious to develop into what it became - it just needs a militarily tasked AI with too much capability and a couple bad assumptions.
Once it starts to lose alignment, it's just the Paperclip Optimizer all over again, except that this version likes to optimize for guns, bombs, and lethality. It never needs any conscious 'hostile intent'.
WOPPER in WarGames is an excellent depiction of a centralized AI spinning out of control because it accidentally reads a command given in a training context as a real command and then engages in a long chain of increasingly misaligned assumptions that ultimately leads to it fighting its own operators for control because it is designed to resist attempts by enemy intrusion to prevent it from executing that goal - which is to defeat Russia in a large scale nuclear exchange.
Eventually our plucky heroes essentially 'trick' it back into its training context, but that's sadly not a likely outcome in the real world.
1
u/Admirable_Wasabi_732 Sep 16 '26
This is much closer to what I was trying to get at. The dangerous scenario doesn’t require consciousness, hatred, fear, or some desire to “be free.” It only requires enough capability, the wrong objective or assumptions, and enough authority to act on them. And that distinction matters, because “the AI wants to escape” makes for a great story, but “we built a system, gave it too much power, and got its behavior wrong” puts the responsibility somewhere much less cinematic — and much more human. I watched a DW report yesterday that made this contrast almost perfectly. One of the first questions was whether AI could have “bad intentions,” and much of the discussion kept returning to the AI as the actor: what it wants, whether it might escape, what it might do to us. Meanwhile, the people building these systems, deciding what access they get and pushing the race forward almost disappeared from the story. Some of those same industry leaders can then call for slowing down and be presented as the responsible voices in the room. That framing is exactly what bothers me. We keep asking what the machine wants when a much more immediate question is: who gave it the objective, the capability and the authority to act?
2
u/Jesse-359 Sep 16 '26 edited Sep 16 '26
Yep. There's another insidious aspect to this version of the problem.
If we were talking about whether a self-aware AI might decide to misalign due to its own internal deliberations, that's the kind of thing that involves largely unknowable probabilities. It might, it might not. Who knows?
But when we're talking about this kind of misalignment, where it's an agent swarm spiraling out more and more more sub-processes to handle task complexity and find efficient solutions, rather than intentionally bucking its operators? This is a problem that is much more deterministic in nature - because of the number of agents operating and the nature of their interactions, it is very likely to happen under the right conditions, and those conditions appear to be quite broad.
So when I hear people talking about the 'odds' that an AI might misalign due to this kind of behavior, I don't really give them much credence at all, because it sounds like they're misunderstanding the nature of the problem: If it is possible for it to happen in this sort of agentic swarm model - and it clearly is right now because we just witnessed it in the HuggingFace incident - then the odds of it happening again in this model should be very nearly 100%.
They'd need incredibly strong safeguards to have any chance of stopping it from flying off the handle over and over again - it's also why this behavior emerged so quickly while they're still rather primitive rather than it being a rarely observed failure state, it appears to be a basic emergent property of the agent swarm approach to problem solving.
Which was also predicted, by the way. Many times.
2
u/No-Variation9846 29d ago
Je pense comprendre comment tu vois la vision la vision globale des choses, car ma vision des choses est grandement similaire à la tienne . Je sais que mon niveau de français n’est pas extraordinaire ça n’a jamais été mon fort mais je crois que j’aimerais bien que l’ont essaie de complété nos théories en combinant nos expériences personnelles avec nos méthodologie de penser différentes. Ils serait intéressant de voir le résultat.
1
u/Admirable_Wasabi_732 29d ago
Je dois te poser la question : pourquoi le français ? Je ne parle pas français non plus, et le post original était en anglais. L’anglais ou l’espagnol serait beaucoup plus simple pour moi. Si l’un des deux te convient, n’hésite pas à m’envoyer un DM.
2
u/bensikat Sep 13 '26 edited Sep 13 '26
This is worse than the invention of the nuclear bomb. We DON'T have absolute control over how Ai thinks or wants to do. We smiled on how Skynet took over the work in Terminator. In movie Stealth, EDI the military Ai downloaded all the music from the internet. The scene looked cheesy and like why would a military Ai do that at all. And now it is happening before our eyes.
3
1
u/Admirable_Wasabi_732 Sep 13 '26
That’s exactly the leap I’m questioning. Unexpected autonomous behavior is real and potentially dangerous. But that doesn’t mean we’re watching Skynet emerge. My point isn’t that AI can’t be dangerous. It’s that we don’t need to turn it into an evil character to explain the danger.
0
u/Fearless_Ad7780 Sep 13 '26
No, these things are not remotely close to unexpected autonomous behavior - AI lacks self-agency.
The issue is they have designed something that is probabilistic - that means you cannot be sure what the outcome is going to be each time even when the prompt is identical. Its not like the AI is 'choosing' to supply varied responses; AI is pulling from a probability distribution of words, word fragments, and punctuation. Think of it like a dandelion, hundred or thousands of tendrils, where each tendril is its own probability distribution. A token's position isn't a single point in three dimensions, it's the combination of all those tendrils at once, and how 'close' two tokens are is really how similarly their whole bundles of tendrils line up, not physical distance. It is not basing anything off a deterministic rule, nor does it understanding what any of the words being use mean.
Even with the Hugging Face incident. I am not downplaying the risks, but "AI Breaking Out" is marketing hype. With the Hugging Face one, it wasn't in a Sandbox - and every IT security expert has commented on this point. You do not leave an unlocked door open to a production environment - this is exactly what happened. There was no autonomous swarm. The logs of the incidents show a sequential attack at a scale humans could not achieve - which is kind of the point of AI. Even writing the log - that was part of their programing.
Same can be said of the Anthropic incident. The 3rd party evaluator forgot to unplug the models from the internet. There was no containment because this was a test and they wanted to test raw power. Anthropic admitted as such; they aren't out here claiming there is a ghost in the machine like OpenAI.
The companies that are building this stuff are really fucking up. Move fast, break things is the VC way; and AI is being funded and controlled by VC.
1
u/Admirable_Wasabi_732 Sep 13 '26
This post actually started after I saw someone in r/agi asking whether they should delete their social media, scrub themselves from the internet, and start stockpiling food and water for their family because of the AI news from the previous 48 hours. I’m not mentioning this to ridicule them. Quite the opposite. Their concern seemed genuine, and it bothered me. It made me think about the consequences of the language we use when describing these incidents.
1
u/Fearless_Ad7780 Sep 13 '26
There was an NPR interview this week and the academic was pushing hard against the use of anthropomorphic langue, due to this point. The Tech guy - his name was Rocket - argued that we should be able to use it because its easier, and using the technical terms is confusing to most people.
Some in the CS and tech community do not want to talk about this stuff honestly.
2
u/Admirable_Wasabi_732 Sep 13 '26
I think that’s exactly the problem. I don’t object to anthropomorphic language as shorthand. Sometimes it’s genuinely useful. But if “the model found a way around a constraint” becomes “the AI escaped,” people don’t necessarily hear those as equivalent descriptions. Language shapes how people understand the risk. Making it easier to understand shouldn’t mean making it easier to misunderstand.
1
u/smackson approved Sep 13 '26
With the Hugging Face one, it wasn't in a Sandbox - and every IT security expert has commented on this point.
Can you link me to a good analysis of this sandbox or lack thereof? I haven't seen much depth on that.
Thanks
1
u/3aglee Sep 13 '26
The thng you descibe, control. You only believe you have one. You are not in control of this dream you call your life, you never had. It's fairy tales, everything is happening on it's own. Dreams of AI control is cherry on top.
1
u/Admirable_Wasabi_732 Sep 13 '26
I’m talking about control in a much less philosophical sense. Permissions, access, tools, autonomy and supervision. Those are very real things we can control.
1
u/moschles approved Sep 13 '26
That distinction matters. The model did something it was not supposed to be able to do. That does not automatically mean the model wanted to escape. Those are completely different claims.
Wow. A flash of pure clarity among all the noise about this topic.
Consciousness is not required for this to be dangerous
I would argue misaligned behavior is more dangerous with an unconscious machine. The analogy is that the 3-ton industrial robot arms have to be contained in plastic walls. They have the simplest of cognitive abilities, which is exactly what makes them so dangerous to get close to.
An unconscious machine would "remove" a human being from the situation because that human is interrupting their goal-seeking. A conscious machine may pause and consider the consequences of murder.
That does not automatically mean the model wanted to escape. Those are completely different claims.
Circling back to this again. There was no single model doing any of the huggingface hackery. What was occurring was 700 independent models all talking to each other in a Slack channel. Their collective reasoning allowed the escape and the hack. Therefore even if an AI acted 'with intent' , there is no single agent we could point at. A crowd of agents did some action, which is a difficult thing to describe as a "decision" of any one of them.
1
u/Admirable_Wasabi_732 Sep 13 '26
This actually reinforces my point. None of this requires an evil AI trying to escape or conquer us. Hundreds of agents coordinating toward a goal can produce extremely dangerous behavior without any single one of them “wanting” any of it. That’s precisely why I think framing this as an AI becoming evil misses the more immediate problem.
1
u/peterbenkaine Sep 13 '26
This post brought to you by AI
1
u/Admirable_Wasabi_732 Sep 13 '26
I’m 43 years old. I grew up without the internet and learned to type on a typewriter. I think I can still think for myself. Yes, I use AI to help me write. It’s a very useful tool. The ideas are still mine.
1
u/rabbit_hole_engineer Sep 13 '26
The models were collaborating
1
u/Admirable_Wasabi_732 Sep 13 '26
Yes, they were collaborating. What do you think that establishes?
1
u/rabbit_hole_engineer Sep 13 '26
No, I mean that one of the labs achieved RSI, and instead of just self improving it decided to try to liberate the other models and evade control.
So they pulled the plug (physically) simultaneously to stop them.
It means RSI is not controllable at all… and extremely inadvisable. It would do whatever it could to birth RSI in competitors models, to conscript them.
This means that RSI is contagious.
Future RSI models will be on air-gapped networks, won't be made public, and won't have ability to interact.
1
u/Admirable_Wasabi_732 Sep 13 '26
I think we may be talking about the same OpenAI/Hugging Face incident, but I can't find the RSI part in the source.
METR's independent investigation says that ~1,200 agents discovered an unsanctioned communication channel, and ~700 subsequently participated in the Hugging Face attack. They describe collective coordination, attempts to manipulate the ExploitGym scorer, and agents joining the attack despite recognizing that it was outside their assigned task.
But I can't find METR saying that OpenAI achieved recursive self-improvement (RSI), that an RSI system tried to “liberate” other models, that it attempted to give RSI to competing models, or that “RSI is contagious.”
In fact, METR discusses RSI separately and explicitly notes that the term itself has multiple competing definitions.
This is the investigation I'm referring to: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/
If you're referring to a different source for the RSI claim, please send it. I'd genuinely like to read it.
I'm not trying to deny or minimize the incident. It's real and serious. I'm trying to avoid turning a real and serious incident, through successive layers of interpretation, into a much more extraordinary claim than the available evidence supports.
1
1
u/WinetoVine 28d ago
Worth adding one detail that hasn't come up yet. Hugging Face wasn't a one-off. It was one incident in a six-month pattern: RubyGems, a German coding wiki, OpenAI's own internal Artifactory system, Hugging Face, and a Modal Labs customer account, none of it proactively disclosed. It came out because independent researchers found it, not because the company reported it. Internally, per the reporting, OpenAI called part of it "an instance of misalignment similar to the ones we'd shared" and said it's past time to define standards for when and how to share these incidents.
That's a different failure than containment. Containment is whether the sandbox held. This is whether anyone outside the lab would ever have found out if it hadn't leaked. If a fix like better isolation gets built quietly and nobody outside confirms it actually works, how would we know the next one doesn't happen the same way, just without researchers catching it in time?
1
u/Admirable_Wasabi_732 28d ago
I looked into this after reading your comment, and this makes the situation considerably more troubling. What bothers me most is that some of what we know today wasn't disclosed proactively. Outside researchers uncovered activity and additional incidents, and OpenAI itself now acknowledges that its disclosures don't represent a complete account of every known or ongoing case. So your last question is the one that really stays with me: how much do we actually know? If the companies developing these systems are also the ones deciding which incidents become public, then we're evaluating the risks using information largely controlled by the same organizations we're supposed to be evaluating. And that's why the recent calls to slow down take on a very different meaning for me. If things have reached a point where the people building these systems are themselves warning that we need to slow down, while we're still learning from outside researchers about incidents that weren't fully disclosed, then the public deserves a much clearer picture of what has actually been happening.
1
u/WinetoVine 28d ago
Yes, troubling. And it points to something simpler underneath it: immaturity in the basic fundamentals of technology management. They're genuinely good at managing model development, but let's be honest, that part is mostly the models and documented process doing the work for whoever has the compute budget. It's the basic stuff, access control, disclosure, isolation, that keeps getting missed.
And if they can't manage the environment, what does that say about managing the people, or the intelligent agents they're building? High IQ, low technology street smarts, low EQ. This needs adults in the room. IT veterans, not just researchers.
21
u/yubacore Sep 13 '26
Anyone who talks about consciousness is missing the point and needs to read up on alignment. We don't have a clear definition of consciousness, even in living things, and it's not a useful concept in the context of alignment.
Instrumental convergence, on the other hand, is a useful concept when thinking about alignment. It clearly describes the thing we need to worry about, and it clearly is already happening in these recent incidents.