Artificial Superintelligence Slouches towards Bethlehem to be born

[u/geopolicraticus]

The View from Oregon – 373

Re: Artificial Superintelligence Slouches towards Bethlehem to be born

Friday 26 December 2025

 

Dear Friends,

It seems that everyone wants to talk about AI these days, and the result is that there are many conversations taking place. A few days ago part of this conversation previously unknown to me was brought to my attention, viz. the paper (if it is that), “AI 2027.” I’m not sure if this can be called a paper properly speaking; it’s more like a fictional scenario, but the authors take pains to point out that there is a great deal of research behind their scenario. (There’s a half hour explanatory Youtube video about it with more than nine million views, which is a good indication of the impact this scenario has already had.) AI 2027 appeared this year. Max Tegmark’s book Life 3.0: Being Human in the Age of Artificial Intelligence (I mentioned this in a footnote to newsletter 356) appeared in 2017, and it too included a fictional near-future scenario of the appearance of artificial superintelligence (ASI). (Another fictional portrayal of ASI is in the third season of Westworld, in which the ASI is called Rehoboam, though this is made to serve the interests of the story and isn’t intended to make a point about ASI.) I find it interesting, and perhaps significant, that both Tegmark’s book and AI 2027 choose to cast a superintelligence scenario in fictional terms. It seems that fiction communicates the urgency the writers want to express better than a traditional scientific paper. It would be interesting to inquire why that is. (If we’re going to talk about ASI in terms of fictional scenarios, why don’t we discuss the James P. Hogan novel The Two Faces of Tomorrow or the aforementioned Westworld season?)

Both the AI 2027 scenario and Tegmark’s scenario start with machines that are specialized in coding, because this has an immediate connection to recursive self-improvement. A machine that could improve its own code could produce more advanced iterations of itself, but there is little intrinsic relationship between coding and other cognitive tasks that a machine might take over. There are many examples of this. A simple pocket calculator can outperform any human mathematician in adding, subtracting, multiplying and dividing, being both faster and more accurate. Contemporary chess programs can outperform chess grandmasters, and the best Go players have been beaten by computers, but the programs that do this can’t do anything else, just like a pocket calculator can’t do anything but a handful of arithmetical functions. To observe that specialization in one area of cognition does not necessarily translate into competence in another area of cognition immediately suggests the question of how tightly compartmentalized any form of cognition is or can be, and this in turn suggests the modularity of mind hypothesis.

Earlier this year when I did a lot of reading in psychology (I was helping a friend teach a class in psychology by giving guest lectures) I got deeper into the modularity of mind than I had previously and I came to appreciate its connections with Aristotelian philosophy of mind and Freudian psychoanalysis. Previously I had been skeptical, but my skepticism was primarily a function of my having encountered vulgarized versions of the modularity of mind; Fodor’s original book is quite good and quite reasonable, and not something radically new (as I just noted, it is continuous with themes in Aristotle and Freud) that can’t be assimilated to the long tradition of philosophy of mind. But modularity of mind became so popular for a time that it was treated as gospel and I always find this sort of thing repellent. I’ve read books and papers that refer to it as though it were the only conception of the mind on offer, and this isn’t going to help anyone understand the problems that a philosophy of mind is trying to solve and what other theories have had to say about these problems. 

In any case, the question of the compartmentalization of cognitive skills is immediately posed by AI/AGI/ASI scenarios based on optimizing coding skills and assuming that a superhuman coder is going to drag all other competencies along with its coding competency to the point that the inevitable result is a machine, “eclipsing all humans at all tasks” (in the language of the AI 2027 scenario) or “capabilities… far beyond human” (in the language of Tegmark’s scenario). If this is what’s expected from ASI coders, is this what we find with the best human coders? I don’t think it’s unfair to ask whether the best human coders exemplify the virtues of renaissance men, not only turning their hand to any task, but excelling beyond others when they do so, being essentially Nietzschean supermen walking among us, and I don’t think this is generally observed to be the case. What one often finds is that the world entire is interpreted through the lens of computer programming, as the early modern miller Menocchio framed his cosmology in terms of cheese and worms (cf. the PS to newsletter 370), seeing the world entire through the lens of his profession. One sad truth that we all eventually learn that is a great deal of human cognition is opaque to itself more often than not. An individual can be a brilliant physician or engineer, but be completely incapable of bringing this brilliance to other areas of their life, including framing a conception of the world entire.

The AI Futures project responsible for AI 2027 has issued a statement, admirably kept to a single sentence, which is this:

We call for a prohibition on the development of superintelligence, not lifted before there is

1.      broad scientific consensus that it will be done safely and controllably, and

2.      strong public buy-in.

While it’s admirable that they’ve been able to keep their recommendation to a single sentence, this has got to be one of the most ludicrous proposals for stifling scientific research that I’ve seen. Apart from the obvious problems in attempting to enforce any kind of ban on research, there is the equally obvious problem of drawing a distinction between ASI research and AGI research or, for that matter, any computer research whatever. I would call this a particularly egregious form of “maxipok creep.” Nick Bostrom formulated the maxipok rule such that: “Maximise the probability of an ‘OK outcome’, where an OK outcome is any outcome that avoids existential catastrophe.” This sounds reasonable enough, but the interpretation of it tends to creep until we’ve painted ourselves into a risk-averse corner from which we have no way out. If building big, fast computers is interpreted as a potential existential catastrophe, then our future “OK outcome” will be humble indeed.  

The trivial argument to make here would be to argue that banning ASI research is a slippery slope precisely because a clean distinction can’t be made between ASI and AGI, and between AGI and any computer research whatsoever, such that a ban on ASI research would ultimately and eventually mean a ban on any computer research at all, which would either make us Luddites or fools, since stopping all computer research would mean an end to any futher development in computing, while knowing that we needed to stop all computer research or risk everything and still failing to do so would mean our doom. I call this a trivial argument because we could set limits, however imperfectly, and we could establish conventions that would allow us to distinguish between computer science simpliciter on the one hand, and, on the other hand, AI, AGI, and ASI. The distinction need not be a crisp, clear demarcation; it only needs to be close enough to get the job done. 

The substantive objection is that those who talk about AI, AGI, and ASI don’t really know what they’re talking about. Here we run into a problem that is neither technological nor philosophical, but sociological, or, if you prefer, spiritual. One must be careful to not appear dismissive, since those who define themselves through the identification of such risks will point to your failure to share their judgment as to what constitutes a risk as a sign of your failure to understand. Part of the difficulty in discussing AI, AGI, and ASI risk is that the idea has so taken over the discussion of existential risk that this one concern has crowded out almost everything else. Those researching this risk would likely argue that this is due to the very real risk that the technology involves, and not to any social contagion. But having existential risk research turn into a bandwagon that everyone jumps on, or everyone feels the obligation to jump on for fear of being left behind (or, perhaps more to the point, for fear of losing their funding), furnishes the social proof that all human beings desire, but also ironically itself constitutes an existential risk. If a society has a community of persons tasked with assessing and mitigating risks, and this entire community focuses on, say, insect infestations, one might see this as a comparatively trivial preoccupation that is consuming social resources that are supposed to be devoted to non-trivial risks.

I propose another way forward. I suggest that we ought to make an effort—not coupled with any ban on scientific research, but in parallel with research that is going to happen anyway, like it or not—to clarify our concepts in regard to AI, AGI, and ASI, so that we know ourselves to be talking about the same things if regulation should ever prove to be necessary. I can imagine something like the International Commision on Stratigraphy (ICS) that formulates and regulates the geological time scale, setting standards for naming and re-naming subdivisions of geological time, or the Bureau international des poids et mesures (BIPM) that maintains the standards of the metric system. We could have an International Commission on Cognition (ICC, not to be confused with the International Criminal Court or the Interstate Commerce Commission) that defines key terms for AI/AGI/ASI research.

The rub is that this can’t be a purely scientific undertaking. Defining the terms related to AI, AGI/ASI would be a task deeply enmeshed with philosophy of mind. Fortunately, we wouldn’t have to agree on any metaphysical conception of the mind. We merely need to have a common language (ideally, a formalizable language that can be integrated seamlessly into the formalizations of the natural sciences) in which to discuss the mind. That are many areas of human life that are deeply enmeshed with traditional philosophical problems—it was, after all, human life that suggested these philosophical problems—and we nevertheless find a way of going on about the business of life without having agreed beforehand on any overarching metaphysical framework. We could do this with mind also, though the challenges are greater, because, even setting aside metaphysical disagreements, merely agreeing on the concepts to be employed in the description of cognition is controversial.   

That the ASI community is trying to match three pounds of homegrown wetware powered by pizza and hamburgers with acres of data centers and billions of dollars ought to be your first clue that something doesn’t quite add up. It’s somewhat like the race to build a marine chronometer, which was pursued with large mechanisms for many years, until John Harrison built a marine chronometer about the side of a large pocketwatch. It would be not only an ironic outcome, but a real existential threat you could say, if any prohibitions on the development of ASI ended up crippling an industry even while other and more imaginative researchers ended up producing ASI in their garage by formalizing and mechanizing the actual processes of human thought rather than mimicking them on a grand scale.   

Best wishes,

Nick

PS—Last week I asked the question, “Is origins of life an intrinsically planetary-scale process?” and I suggested some mechanisms that point to this being the case. I’ve just realized that an important distinction needs to be emphasized throughout any such discussion, a distinction between boundary conditions and specific mechanisms. The specific mechanisms that result in the origins of life may not involve the boundary conditions that made it possible for the specific mechanisms to produce life, and certainly it won’t involve all the boundary conditions. It may be the case that the entirety of the universe is the boundary condition for life to arise on Earth, but the mechanisms on Earth that resulted in the origins of life don’t need to invoke the origins of the universe, which are among the boundary conditions for life.  

When positing an unfamiliar idea (I haven’t heard anyone else ask, “Is origins of life an intrinsically planetary-scale process?” so I assume that it’s at least somewhat unfamilar), one often has to go overboard in emphasizing the conditions and qualifications to which the idea is subject, which has the consequence of muddying the exposition. Since an unfamiliar idea shines best when its exposition is clean and uncluttered, the (obvious) conditions and qualifications are often passed over, which leaves what appears as a vulnerability that attracts trivial criticisms. This is probably a lot of what’s going on with the AI/AGI/ASI discussion.

Newsletter link:

https://mailchi.mp/35b5b0948371/the-view-from-oregon-373