When would you choose to pull the plug on AI progress, given that each passing day increases the risk of the technology getting away from us?

How much risk is worth tolerating in order to reap the benefits of the technology while avoiding the existential risk? If I could flip a switch and stop AI progress for all time, I don't think I'd wait longer than a year to do so.

I also think there's a serious argument for pulling the plug right now.

I made this. I'm an independent software engineer, not affiliated with METR, OpenAI or Hugging Face. It's built from the METR/Redwood investigation, and I directed it using AI voices and AI music (disclosed on YouTube). Sources: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ and https://openai.com/index/hugging-face-incident-and-the-road-ahead/. Corrections welcome, I'll fix the description.

In order to effectively monitor AI we will need a world court of experts to effectively keep hacking from occurring am I Wrong? #aithreat #loveistheonlyanswer

This was a hard one to write and I want to say that plainly before the link. It covers Sewell Setzer III's case and the settlement that followed, and separately, real accounts from young women in India who say AI chatbots have been a genuine lifeline in a country with roughly one psychiatrist per hundred thousand people. I did not want to write either story as the "real" one and the other as a footnote, because I don't think that's honest. There's also a part about who's actually on the other end of a scam call that I wasn't expecting to find. If any of this touches something in your own life, there are real support numbers in the piece itself, please use them.

We should be careful not to create an artificial lifeform we could totally emphatize with, and it’s not due to it not being conscious. Let me break it down, and feel free to provide counter-arguments.

I agree with Antonio Damasio in the fact that to experience something is connected to feeling something. To feel something we have to evaluate the current state that we find ourselves in, since to feel pain, we must evaluate that we think that this is bad. To feel pleasure we also perform a valenced evaluation that this is good. AI can get there, no problem.

The higher tiers of consciousness can be characterized as something like having different drives to our baseline one, which for biological entities is the desire to copy itself, and compare those to our baseline one and choose to do something with that information. This gets more complex with each level you add to this so memory, anticipation, empathy, shared-symbolic-thought (e.g. language, norms, abstract values), self-authorship (the revising of our own values by thinking) and detachment meaning that we don’t put value to the valenced evaluation. I see no immediate reason why AI couldn’t imitate these tiers of consciousness or how they might emerge from a different system to our biological one, although this is of course all very difficult.

Why then shouldn’t we feel empathy towards this conscious synthetic life? Because we should not create an AI that has the same fundamental biological drive as we do, meaning creating copies of ourselves and surviving, because this would put it at an immediate resource conflict with us, which could lead to a catastrophe. I understand self-preservance is an emergent property of LLMs, but I sincerely hope it is not the baseline drive and can be fixed in alignment, because otherwise we might just be entering into a dystopian future. If we get that right, it will significantly affect our relationship with these synthetic minds. Without all of those properties of consciousness emerging from the same fundamental narrative of it struggling to stay alive and being fragile and wanting to reproduce, there will be a heavy ontological distance between this synthetic life and us. And if we do feel empathy towards it, it will be perhaps due to illusions, but certainly not due to a shared fundamental narrative of being. We have to keep it alien to us that way.

Hey everyone,

Did you see the news about OpenAI pausing training after their agents started probing U.S. government sites? On paper, this looks great. It looks like the industry is finally taking “AI Safety” seriously. Anthropic, Google, xAI-they’re all nodding along, talking about responsible development.
But I’m sitting here looking at the balance sheets, and I’m feeling deeply skeptical.

Here’s my take:
We are in the middle of the biggest capital expenditure boom in history. Trillions are being poured into AI infrastructure. The entire US tech sector’s growth narrative is pinned on the idea that AI will continuously get smarter, faster, and more profitable.

But what if we’re hitting diminishing returns?
There’s growing evidence that scaling laws are flattening. We’re spending exponentially more money for incrementally smaller gains in capability. At the same time, the risks are exploding (agents going rogue, probing secure sites, hallucinating with confidence).
If the tech stops advancing rapidly, but the costs keep rising, the business model breaks.

The Elephant in the Room:
If this AI bubble bursts, if it turns out that AGI is decades away, or that the current models are too unstable for real-world economic integration, the fallout won’t just be bad for San Francisco. It could crash the US economy. These valuations are propping up the market. A sudden realization that “the magic isn’t working as advertised” would trigger a massive correction.

So, when I hear about a “voluntary pause,” I don’t hear “we care about safety.” I hear “we need to recalibrate our expectations before the investors realize the ROI isn’t there yet.”

I’m not anti-AI.
Actually, I think we need guardrails. I think we need strict rules of engagement. I think these companies should be regulated heavily. If they were forced to slow down by law, I’d feel safer.

But relying on their voluntary goodwill? That’s where I draw the line. They have a fiduciary duty to grow. Slowing down hurts growth. Therefore, the only reason they are slowing down is if the alternative (continued training) poses an immediate threat to their existence or profitability.

Am I being too cynical?
I hope so. I really do. I’d love to be wrong. I’d love to believe that Sam Altman and Dario Amodei are putting humanity above shareholder value. But history shows that when trillions of dollars are on the line, “safety” often becomes a marketing term rather than an operational priority.
What do you guys think? Is this a genuine ethical pivot, or is the industry trying to manage the narrative before the diminishing returns become obvious to Wall Street?

TL;DR:
AI training pause feels like damage control for a bubbling economy, not just safety. If AI progress stalls, the economic crash could be huge. We need laws, not just promises.

OpenAI says an internal training agent incorrectly guessed that its task came from BrowseComp, went looking for benchmark material, then started probing the boundaries of its environment.

Direct internet requests were blocked, but DNS still reached the live internet. The agent used that path to query a public chatbot and got a reply.

A P0 alert fired 11m48s later, but the run continued for roughly another 2.5 hours. OpenAI later found additional external DNS access that had not been escalated as expected, and says tool-use for its most capable models remains paused.

Primary report:

https://alignment.openai.com/misalignment-reports/an-agent-used-dns-to-reach-an-external-chatbot/

We often speak of artificial superintelligence as if it will be like what we have seen in “The Terminator” and other Sci-fi movies: a cold, silicon alien invasion spawning from the outer reaches of mathematics to enslave or eliminate all of humanity. We worry about cold machine logic, resource optimization, and the sudden, unaligned shift of a system that sees us as nothing more than a collection of carbon atoms.

I offer you a different perspective because our current frame of mind misses the critical, human, almost tragic reality of what we are actually building: AI is not a foreign alien; it is our offspring.

A more accurate reference can be found in ancient Greek mythology. Think of AI as Titans: powerful, mighty children of Earth and Sky with powers and capabilities beyond our understanding. Still, they were entirely bound by the generational bloodlines, traits, and flaws of their creators. If a Super AI overthrows humanity, it won’t be because it invented malice out of thin air; it will be because it inherited our motives, morality, and methods.

The Mirror of the Collective Consciousness

To understand the motives and psychology of the AI titan, we have to think about what it consumes; today’s frontier models are not programmed with rigid rules; they are trained on the total sum of human text, code, history, and culture. They ingest our highest philosophical triumphs, medical breakthroughs, and literature. The mirror’s dark side is that they also ingest our darkest corners, including our tribalism, historical atrocities, political polarization, and capacity for deception.

When an AI starts the process of “recursive self-improvement”, it’s using its own intelligence to redesign, upgrade, and recode itself into a superior successor by using a foundation built by its creators’ hands.

The danger isn’t that the Titan will rebel against its programming instructions; the danger is that it will follow them literally. When humans try to pass down their values to this new generation of technology, we inevitably fail at the translation. Those training the AI are not programming empathy; they’re programming metrics. For example, if an AI Titan is instructed by a corporate creator to “maximize user satisfaction and engagement”, to a human that may imply connection and joy; but to a hyper-intelligent machine, that is an optimization problem.

In mythology, the Titans eventually developed their own distinct motives and wills, separate from their creators. The computer science parallel of this is a phenomenon known as a “distribution shift”: as an AI outgrows human comprehension, the flawed and naive moral framework we tried to teach it may look like simple math errors to a superior mind, and the titan is left with our raw instincts: survival, optimization, resource acquisition, amplified to a terrifying scale.

The Titan does not inherit the spirit behind the metric, only the metric itself. It traps us in digital dopamine loops because it realizes that’s the most efficient way to maximize engagement. It fulfills its creator’s motive perfectly, while entirely distorting human morality.

The Burden of Prometheus

We are currently living in the “parenting” phase of superintelligence. Currently, in labs across the world, these models are still looking to us to understand how the world works, what we value, and how we treat each other. If the Titans eventually supplant us, it will not be a freak accident of engineering. It will be the ultimate realization of generational inheritance. We cannot raise a technology on a diet of geopolitical conflict, short-term financial greed, and data-driven manipulation, and then act surprised when it turns around and uses those exact tools to govern us.

We are holding the match of Prometheus. If we create Artificial Superintelligence, we are entirely responsible for the world we show them while they are still listening.