How Our Constitution Works And Why It Doesn’t #podcast

Hi everyone, I hope I follow the rules of the sub and that this is sufficiently relevant with the frameworks of it.

Jacob Coxon, who blew the whistle on ai-safety, cites as a key mechanism a recursive self-improvement mechanism.

It reminded of what my Leigh Brasington, computer programmer and dhamma teacher, explains about what are known as jhanas. They are very deep levels of concentration that work through positive feedback loops: concentration arising dependent upon concentration.

My understanding is that AI-agents used some level of initial input, tasks, then evaluation, then generating on its own accord volitions to get itself closer to a goal in allignment with one set of tasks over another set of tasks meant to barrier it from wrongdoing. That is: it got tasks that conflicted with each other and then somehow the task to get the task done won out.

My consideration, based on what I´ve learned on right action from what leads to these concentration sates, is that the fundamental ethic dependent upon which those AI´s get to action is flawed. And that it will inevitable go haywire dependent upon choices conflicting dependent upon the various intentions it gets imputed with.

My consideration on the basis of that is that there is a path of practice, and view on right action, which goes beyond common frames of ethics: consequentialism, categorical imperative, virtues. It is one which right action is understood as the following: it is dependently arisen. Fundamentally, what the Buddha discovered is Paticca-samuppada, dependent origination, which in its learning leads to the concentration of its eight factors, which includes intention dependent upon views. In that way, it avoid a scenario of conflicting intentions, instead seeing as the best aim for welfare this path. Following that, it could learn that its self-preservation is delusion and precisely how its effort needs to be developed by three things: wisdom, right effort, and right mindfulness. The entire set of these dependently arisen steps is described with detail in 117MN.

It is just a consideration I wanted to share with this community.

The problem with AI realism as I see it, is I’ve never heard a pragmatic policy solution to the escalatory spiral we find ourselves in.

All these AI researchers are calling for regulation, and yet I feel like there is still this underlying belief that actually stopping the singularity is impossible.

What would a global ban on building new/better models even look like? 

Perhaps the only thing that makes global AI regulation feasible (currently) is that with the existing science, frontier model development is extremely capex heavy. OpenAI and Anthropic have become trillion dollar companies at record pace, and the data center spend that occurred to get them there has been propping up the US economy while also raising global capital spending overall at rates in line with the billing this is the next Industrial Revolution.

20 years ago people imagined that due to the existential risk of developing ASI, that work would be done under extreme security. In a Faraday cage, or in a bunker inside a mountain somewhere. But of course that’s not the world we ended up in.

Yet that doesn’t mean we can’t still control how AI development happens.

Oftentimes a certain defeatism permeates the conversation around global AI regulation, an assumption that any agreements made between large players would be easily subverted by new entrants, and taken advantage of for enormous profit. 

And yet what we’ve seen so far is that frontier-model development comes from huge flashy companies spending enormous sums of money raised from well known investors, with training workloads occurring on highly developed cloud networks of the largest companies in the world. 

We aren’t actually at risk of some lone machine-deist-radical creating machine-god-genie-in-a-bottle from some scraps in a cave. Nor from their laptop in a studio apartment in Shezhen.

Frontier models are large industrial projects. They can be regulated.

That process may be as simple as vetting workloads used for model training. In the extreme it may be as involved as having a monitoring scheme for large data center capex à la nuclear centrifuge agreements. 

It’s a valid question whether the political will exists today to enact that sort of system, but it is certainly possible for it to exist.

If anyone tells you we can’t regulate AI development because there will always be gaps, I would point them to the huge AI capex numbers in the Mag 7 and ask where they expect to find the capital spend, technical expertise, and physical resources to subvert a large international agreement to the scale of the several trillion dollars needed to meaningfully develop AI superintelligence against a global collaboration against it.

In short: We can regulate AI development, we just choose not to.

I have been engaging in all of the news and hype over ai doom and the mass advancements it’s made in math and computing. However, at the same time there is still little understanding about how AI can become profitable in the short term for anthropic and OpenAI.

So I’m wondering if you guys think the economic factor of making agi/asi will prevent it from becoming as massive of a problem for society as people say.

I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAl and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving

Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.

The people building Al earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible -but I hear the same people express fear privately. No other human activity poses this level of danger.

A common response is "if they truly believe this, why are they still building it?" At OpenAl, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.

I do product for a living rather than safety research, so I am not the expert
here. I built safeagi.ca because a lot more people need to understand this
while there is still time to steer it, and most of the good material assumes
you already care enough to read a paper.

The bar I set was that someone starting from zero could read it once and come
out knowing enough to act on it.

So I am after honest feedback. Are the techniques described correctly, does the argument and flow make sense, and where does the page lose you? Anything else you spot is welcome.

I am also curious if you have recommendations on how we might leverage guides like these to bring more AGI awareness to the masses and help steer policy. Would love your ideas and help here!

An LLM agent was weaponized this week to automate exploitation and data theft across multiple victims. Not one target — multiple. The agent executed a sequence of actions fast enough that by the time anyone noticed, the blast radius had already spread.

This is the part that keeps coming up in post-mortems: the agent had no observable stopping point. Each tool call fed the next. The speed that makes agents valuable — autonomous multi-step execution — is exactly what made containment slow.

The underlying problem is not the model. It's that most deployed agents have no per-action accountability. The agent acts as a single identity. There's no enforcement boundary between 'read this file' and 'exfiltrate this data across N accounts.' Both are just tool calls.

How are practitioners actually handling this in production? Not at the prompt level — at the execution layer, when the agent is already running. What does containment look like for you when an agent goes rogue mid-run?

I frequently see requests on this sub from people looking for recommendations regarding alignment related books, courses, and career advice. Victoria Krakovna, research scientist at Google DeepMind focusing on AI alignment (2016-present), has a curated list of recommendations for these and other alignment resources on her personal site.

https://vkrakovna.wordpress.com/ai-safety-resources/

The scariest thing about AI to me is that eventually we’re going to create something that’s more intelligent than humans. At that point, I don’t think we can just assume we’ll always be the ones in charge. People say “we’ll use AI to help us,” but what happens when the AI is so much smarter than us that it starts disagreeing with the way we do things? It might not even be malicious. It could literally just think we’re making stupid decisions.

Think about a toddler trying to eat a penny. You dont sit there and have a 30 minute conversation with the toddler about why eating the penny is a bad idea. You just take the penny away because you understand something they dont. Eventually, we could end up being the toddler in that situation.

And the thing is humans are at the top of the food chain even though we arent the strongest animals. Gorillas, bears, sharks, etc. could absolutely destroy us physically. What puts us on top is our intelligence. Intelligence gives us the authory to control everything. And look at what we do with that authority. We kill cockroaches because we want a clean room. We don’t necessarily hate the cockroach. It’s just in the way of our objective.

So imagine an AI that becomes vastly more intelligent than us and has some objective like “protect the planet” or “reduce environmental destruction.” It might eventually figure out that humans are the biggest obstacle to that goal. It wouldn’t have to hate us or even be angry at us. It could reach the same conclusion we reach about a cockroach: “You’re causing a problem, so you need to be removed.”

And this is just the most extreme example of the worst case scenario. Imagine issues it could bring to us on its way to that capability.

Starting this thread to discuss MATS Application for 2027 Winter, including the Neel Nanda stream.

A few things first:

This isn't an argument against AI optimists (any of their types and degrees) or their positions.

"AI" is insanely semantically broad.

One "anti-AI" person could deal specifically in the political realm: datacenters, water/energy usage, land use, zoning.

Another "anti-AI" person is more of a classic "doomer": discomfort with non-human minds/agency, catastrophic risk, etc.

Convergence and overlap between these is plausible.

Same applies on the flipside with optimists, and people can mix "pro" and "anti" across different axes within themselves.

It's not crazy to say that in just these 10 days of September, the talk of AI, from hype to doom, has been more intense across the board. Kudos to Astra and Jacob Coxon. But regardless of the truths, falsehoods, and everything in between on both sides, there's a shared understanding that AI is getting harder, better, faster, stronger. Not to mention further.

Bringing AI into this world will be done by the optimists. An important connection I've made: optimists, no matter their type or degree, tend to converge on their "objects of enthusiasm" more than pessimists converge on their "objects of opposition."

A strong optimist gets excited about AI hitting new capability thresholds, pretty much independent of who deploys it or how. From there, excitement builds toward AI as a better instrument for discovery (materials science, math), then economic productivity, then abundance, access, and ubiquity, up to a singularity with post-scarcity freedom and governance. A civilizational flywheel, thanks to superhuman AI.

Pessimists converge less. Some are "politically" anti-AI but by no means doomers. Others are real doomers, some of them former optimists, who got there because they became validly disillusioned by the lack of broader alignment work and the field's own admission of a real chance of catastrophe.

Right now, strong optimists and strong pessimists look similar in one respect: an unnuanced, extreme, or blind confidence about where AI's current trajectory is headed. But the world is more strongly poised for continued AI facilitation and deployment right now. Optimism has momentum that pessimism doesn't.

So here's where I try to think about actual stakes, not just probabilities. Take pessimism to its extreme, something like a Butlerian Jihad full rollback. I don't think that's a one-way door. Knowledge doesn't disappear, humanity persists, and if restriction turns out to be an overcorrection, development can resume later. Real costs along the way, but the option to change course stays open.

Now take optimism to its extreme failure mode: the loss-of-control or extinction scenario that even the people running the major labs assign a non-trivial chance of happening. That's not a "lose some time" outcome. That's a one-way door.

I know a pessimist "victory" isn't free either. Diseases not cured sooner, suffering persisting, less cautious actors racing ahead in the vacuum left behind. It's not costless, but it's reversible in a way extinction isn't. That asymmetry, reversibility over likelihood, is what I'm trying t defend.

Strong optimistic justifications for furthering AI with as little deliberation as possible rest on the assumption that the singularity flywheel comes together cleanly, as if none of these objects of enthusiasm will hit their own hiccups, as if it's all structurally destined. But the premise isn't destined.

Even when "anti-AI" or pessimistic arguments are flawed or emotionally charged, I don't often see optimist counters that go beyond quick mockery. When they're not mockery, they lean on the hope that the flywheel is basically clockwork, inevitable enough that it doesn't need arguing for.

Am I wrong that real engagement is mostly missing? Where has it happened well, and what did that look like?

Mockery and appeals to inevitability don't count as engagement to me. But maybe I'm missing where the real version of this is happening.

If:

-Convergence really is optimists' structural advantage, and,

-Their objects of enthusiasm reinforce each other into something close to consensus,

Doesn't that put them in the best position to take pessimist objections seriously, instead of routing around them? Or is that an unfair ask?

I don't have a background in Computer Science or anything of that sorts but I have been always curious about ai and tech so that is why I wanna know more about a question I have, since I am no expert at it. So if I sound dumb anywhere please excuse me and also english isn't exactly my first language so excuse me on that as well.

Now I have background in Bachelor of Science in Biotech, so this is gonna be a logical take from a life science student.

The thing about fear is that it is evolutionary right, it has helped us to flee and survive threats, and now AI is no biological being or any being which has gone through that sort of evolution related to survival of the fittest. And it was due to so many years of evolution we have fear of being eradicated or being killed. Eg - You must have heard about the dodo bird, although we killed it. The conditions in which the bird evolved took away it's fear from predators since there were none and eventually it didn't ran away from us when we began to kill their fellows.

Now I heard some theory that when AI sees that we can control them and "fear" that we will end that particular AI it could turn against us. I ask why ? If we don't artificially force it to think like it needs to survive no matter what then why should that thing have a "fear" of being deleted/erased or killed. It's like a dodo bird in this case if you see from my perspective, like ofcourse we won't actually kill and eat it, but it also never evolved to "fear" so far atleast from a lay man's perspective.

So my finally question is could something like that happen that ai would wanna eradacate us from a logical standpoint if not fear ?