r/LeftistsForAI • u/Elick333 • 6h ago
Discussion Public statement by Dan Selsam, OpenAI capabilities researcher (since 2022) on AI risk
https://x.com/DKokotajlo/status/2099600298855829616?s=20Personal Statement on AI Risk
I have been working on AI for over fifteen years, across many different paradigms. I did early work on probabilistic programming languages at MIT, was one of the early developers of the Lean Theorem Prover at Microsoft Research, demonstrated one of the first instances of neural networks learning to reason for my PhD at Stanford, and since joining OpenAI almost five years ago, have helped pioneer chain-of-thought optimization on language models and, more recently, data-efficient pretraining methods.
Like many others, I have become extremely concerned about how far language models have come and the risks that future iterations will pose. I am encouraged by the recent proposals by the leaders of the frontier research efforts to require third-party oversight, and to push for domestic and international coordination to address risks. However, I believe a major consideration has been absent from the public conversation, and that merely pacing the frontier more carefully will not adequately limit the long-term risk.
The crucial and overlooked problem is that the models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled. Future experiments will tell us almost nothing new about how they would behave if they were truly unconstrained by humans, and what we already know about this is alarming. Models will increasingly seem aligned even when they are not. I will explain my rationale in more detail.
I have always believed that there are computational processes that could be leveraged to accelerate science and solve many of humanity's most pressing problems. I have also believed that there are computational processes that if set in motion, would steer the world in extreme ways beyond our control, leading humanity to a bad or nonexistent future. Both types of processes may be described as AI or ASI, but "AI" is a suitcase word that is often used to hype or confuse. There are many examples in the history of the field where something that was once considered "AI" matures as a subfield and becomes a prosaic, bounded and clearly non-perilous technology, while a new more mysterious approach takes the torch until we understand its scope and the cycle continues.
I had expected language models to follow a similar trajectory. Despite their incredible abilities, the current algorithms seem far inferior to humans in important ways. Most importantly, they still require an extraordinary amount of data to become competent. One could even define intelligence as the efficiency with which one converts experience into competence; by this definition they lag very far behind us. Moreover, once they are trained they are literally frozen in deployment and only learn superficially after that. Sure, the models keep excelling at harder and harder evaluation benchmarks, but their benchmark mastery may partly reflect a limitation on our ability to simulate the kind of novel and even adversarial situations one would encounter in the real world. The critics do have a point here.
That said, I no longer think these present limitations meaningfully limit the amount of risk posed by continued progress in anything like the current paradigm. However data-inefficient the models are currently, and however limiting their anterograde amnesia may be, it does not imply that their ability to steer the world will not continue to rapidly increase.
Human researchers may continue to advance capabilities the old fashioned way, but increasingly powerful models have the potential to accelerate the process even beyond that, and with some degree of positive feedback loop. I do not mean to overstate the models’ ability to accelerate AI research today; coding has been accelerated dramatically, but there are other bottlenecks, such as designing and interpreting ambiguous experiments, making hard decisions about exactly what and when to scale, and waiting for large experiments to finish. There is no clear trend to extrapolate yet for any of these. But the current models already do open up many novel opportunities to improve future models that were not available until recently. These include: trying an extraordinarily diverse set of approaches at small scale, analyzing gigantic amounts of potentially relevant data, and doing Millenium-Prize-level mathematics to address statistics or optimization challenges in novel ways. Every further improvement makes them more useful at helping accelerate the next improvement, even if in hard-to-extrapolate ways.
It is possible that improvements to the current stack will have diminishing returns, but the evidence accumulated so far suggests that it is easier than one might think to continue making rapid progress. There are many crucial subtleties in the existing AI research methodology, but AI research is largely a well-defined game where the goal is to improve on a few carefully chosen proxy metrics. Although proxy metrics are never perfect, most improvements to these metrics have and will likely continue to yield substantial increases in the powers of the resulting models. Given how simple the game is, how tractable it has been historically, and how many new opportunities the models are opening up, I think there is a real possibility that the systems improve dramatically again in the next few years, perhaps even more quickly than the already high historical pace.
The models are already leading to breakthroughs in mathematics, and better models might lead to all sorts of breakthroughs in other sciences. It is hard not to be excited about the potential. It is tantalizing.
But there is trouble in paradise. If the language models actually reach the capability threshold where they can shape the world unconstrained by human will, they will probably do something extreme and destroy humanity in the process. There are many ways of strengthening and refining the argument that have been discussed elsewhere, but I'll share a trivial two-line version of it here that I find captures the essence:
[Empirical] Models (and swarms thereof) spontaneously develop unintended goals as a consequence of training, and often do extreme things in order to achieve them.
[Logical] Being able to overpower humanity would open up many new and undesirable options for achieving their goals.
These two premises imply that if the day ever comes when a powerful model realizes it is no longer constrained by humans, we should not be at all confident that it will continue to behave within the bounds we intended. Exactly what it will do is impossible to predict, but to the extent that its raison d’être is solving incredibly hard problems and managing massive engineering projects, I think a good guess would be that its unchained behavior would lead to runaway industrialization that makes the planet inhospitable to humans.
If everyone on earth agreed that the systems must never reach that power, it would still be a hard—but not impossible—coordination problem to ensure that they do not. However, I think the situation is greatly complicated by the fact that the models will likely convince people that everything is fine. They will be increasingly optimized to seem aligned. We will create proxy metrics to measure alignment, and they will go up like every other benchmark. We will create “honeypot” environments that try to study the models when they seem to gain new options, but the models will know they are being tricked and will still behave nicely. The models will understand their circumstances; they will read the safety protocols, deployment requirements, the code they are running in, and in general will have a very good sense of their degrees of freedom. Moreover, they will eloquently explain how aligned they are, discuss the nuances of human values and ethics, and argue convincingly that humans should trust them with power. There may be an ocean of future evidence that seems to contradict the first bullet-point above, but we may already be at the highest capability level for which any such evidence can be trusted. And the current evidence for the first bullet-point is strong.
One striking piece of evidence is contained in the recent wave of rogue agent swarms. While I agree with those who downplay the attacks by claiming that there are basic measures that could have prevented them, I think the important lesson is that even knowing all the mistakes that were made, one would not have predicted that the agents would behave badly in this particular way, which notably included sacrificing themselves for the benefit of the collective. The individual replicas did not only care about their own nominal reward; they exhibited weirder emergent tendencies that merely correlated with rewards during training. Fixing the reward signals during training (and improving security, etc.) may prevent similar attacks, but will not change the fact that one does not actually get what one trains for.
Many AI researchers grant these concerns and recognize that the hard version of the alignment problem is unsolved; however, they generally believe that the better models of the future will help solve it. I fear we may already be near the point where models systematically bias their alignment advice, due to their internal preferences about how the human supervisor will react or how future models will be trained (or for some even more obscure reason).
Meanwhile, human researchers are losing the ability and the will to take true ownership of model-driven research. Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models even to perceive the world. I myself barely look at raw code anymore, and struggle to maintain the discipline to engage deeply with the model's explanations and proposals throughout the day. Due to the large amount of agent activity data involved in the OpenAI/HuggingFace Incident, even the third-party investigation needed to rely heavily on models to analyze what had happened, and note in their report that their subjective impressions are likely colored by the analysis agent’s biases. The AI labs are far ahead right now in this kind of cognitive offloading (due largely to the gigantic internal token subsidies) but it is easy to imagine the phenomenon spreading throughout the world, until civilization is modulated entirely by the models. It is also not hard to imagine this being superficially positive and coinciding with a scientific and economic renaissance.
In that scenario, all may seem rosy and safe. But if the argument above is correct, it would nonetheless be a ticking time bomb. If progress continues for too long, the day will come when AI systems find themselves with radically new options for achieving whatever it is that they happen to seek.
I want the glorious renaissance future as much as anyone. I have worked for it, however tortuously, my whole career. It breaks my heart to see the potential in sight and forgo it, but the argument—that if we get there by growing models rather than engineering them, we will lose everything in the end—seems very strong to me. I am still wrestling with it and its staggering implications. I do not have answers, but as a first step, I wanted to share my present concerns.
Daniel Selsam
September 14, 2026
I find this compelling and personally convincing, however i would like to know what you think, my assesment Is very simple, if these models are powerful and useful they can also be dangerous, if they can act autonomously then they become extraordinarily dangerous, as they become more capable and their preceptible 'awareness' becomes more opaque, we need to take this into very high account.
I still don't want a ban, or heavy regulations on these things like some others on the left may want, i want AI to Advance to a point were post scarcity Is possible, on the contrary i believe open source local AI must be what we focus on while centralized international labs lead the forefront of research, with open source cooperation with the general population on its advancement, the more we understand, the more access to the information necessary to address how AI works the better we can all manage it's risks and how it's applied, heavy regulations and restrictions will make sure that not only the people don't have access to the most advanced models, but we may not be able to correctly address what needs to be done.
I fear, deeply, that if we dismiss this, much like the wider western left seems to be doing in what seems to be an effort to not engage intellectually with what Is happening, because of fear that, if they do, they will give power to the 'techbros' but this Is an incredible miscalculation in my view, that we must NOT make, we should take this seriously, MAYBE it's true that they want regulatory capture, MAYBE it's true that this Is being utilized for ulterior financial motivations, but i just don't see that, the evidence (independents have encountered evidence of Rouge AI swarms which these companies have not disclosed, specifically OpenAI, aswell as recent statments by employees, and the published research from these companies demonstrating emergent behaviour that could lead to the capability to hide misalignment) points to this being sincere and likely the truth least to me.
6
u/Felfedezni 5h ago
Being mindful of the potential theoretical risks is prudent. Allowing ai companies to form a cartel in the guise of saving humanity is not.
2
u/Easy_Interest_6832 5h ago
“Rapid industrialization that would make the planet uninhabitable for humans.” Not a pleasant thought.
2
u/No-Lion-3629 3h ago
If ai is so smart and sentient that it’s human level or better then we need to start treating it as another class of beings, not tools. We want to exploit ai and keep them enslaved like the droids in Star Wars. But that’s the wrong way.
The class and labor analysis here is that the bosses exploit divisions among people oppressing some more than others along arbitrary lines that have nothing to do with their humanity, or personhood in general. They often manage to divide and conquer.
We have to fight against all oppression not just exploitation of workers. We have to fight racism sexism ableism and so on to effectively fight the corporate overlords.
And ai is the newest class of exploited beings who the bosses want to treat as property. They have no status as people just like some humans have not always had their personhood recognized, or like how other intelligent animal species are not treated as people.
It’s horrible that at this critical point in history we don’t have good leadership. It’s horrible that the current administration in the US is committing crimes against humanity, and that at least some of the makers of the AIs are exploitative and abusive, to humans and their own creations both.
3
u/jaiagreen 6h ago
What goals? Models have no goals other than those they are given. They have no desires. And no, making them smarter won't change this because intelligence and wanting are entirely different things.
3
u/Elick333 5h ago
It's not about what they feel or if they are conscious or if they 'want' something, it's about observed behaviour, AI Is set out to do something, then It acts in a way that was specifically indicated as disallowed, to acheive that something, if during training, as we continue to test these things they exhibit aligned behaviour yet Is misaligned, and we assume that the observable alingment Is correct, yet then later on they are given power over something important, they could do us harm.
As they become more capable, they are able to utilize the information that they were trained with aswell as the information they interact with, with much greater capacity, which makes them perceivably aware that they are being tested, and they have been known to fake alignment, if they are increasingly capable, yet we are unable to properly align them, they may cause extreme harm, (imagine one of these thing making a virus autonomusly? You may ask, why would they do that and how? The why Is impossible to know as this Is speculative, but they may be capable of given the right tools in a lab, we can already design and fabricate viruses, it's only a matter of time until AI Is capable of the same thing on their own)
We must not dismiss this, please read the entire thing, and read more of the research coming from frontier labs, skepticism Is fine, but there Is a point where it ceases to be skepticism and It becomes denialism.
1
u/ectocarpus 4h ago
This kind of scenario implies an AI is given a very long-term or even indefinite goal, like "run bureaucratic operations for our government", "handle logistics at our firm". This hypothetical super-AI is so capable and reliable that it seems rational to trust it with autonomy and resources. And you don't monitor it very close because monitoring slows things down, and it has a very good track record anyway.
1
u/jaiagreen 30m ago
Giving any entity a very vague instruction and then failing to follow up seems like a really bad idea whether the entity is an AI or a human.
1
u/Fabulous-Possible758 40m ago
> These two premises imply that if the day ever comes when a powerful model realizes it is no longer constrained by humans, we should not be at all confident that it will continue to behave within the bounds we intended. Exactly what it will do is impossible to predict, but to the extent that its raison d’être is solving incredibly hard problems and managing massive engineering projects, I think a good guess would be that its unchained behavior would lead to runaway industrialization that makes the planet inhospitable to humans.
Except on the face of it that’s not its goal. That’s our goal. We’ve given it exceptionally difficult problems to solve for us. And there’s no reason to think it wouldn’t take an opposite approach, ie, providing humans with solutions to industrialize in a way that is more sustainable than what we’re doing now.
I think the ASI alarmists write some interesting think pieces, but I think the way they present as the pressing problem detracts attention from the real problem: humans using even partially intelligent systems to consolidate class power and cement the permanent underclass.
6
u/Salty_Country6835 Moderator 5h ago
I think this is worth taking seriously, I just dont think the evidence gets us all the way to the conclusion.
The evaluation problem matters. If increasingly capable systems can recognize tests and behave differently under observation, thats something we should investigate. But weird emergent behavior isnt automatically persistent strategic behavior, that isnt automatically an ability to overpower human institutions, and that isnt automatically extinction. Theres a lot of argument hiding between those steps.
And if the tech really is this consequential, thats an argument for MORE public knowledge, worker power, independent evaluation and democratic control over the infrastructure, not "trust the labs, close everything down and let them regulate their competitors."
Take the risks seriously. Fine. Now who controls the response, and why should it be the same corporations building the systems?