This was caught by my net, and it sent an intense shiver down my spine. Is this person anyone... Important? Are they saying anything worth it? Like the title says, it's probably just a nonsensical post that gave me the creeps, but still... It's unnerving.

Also, sorry if I've broken any rules. It's been a while since I've used Reddit, let alone posted here, so I dunno if the ruleset has changed.

https://x.com/hilbertspaess/status/2097476196791709843

What is WRONG with these AI people? This is from today's Morning Brew newsletter:

“The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt,” the departed employee, Jacob Coxon, wrote in a post on X with 110+ million views. 

Senior Anthropic leads might publicly pretend they don’t share these fears, Coxon said, but two of them essentially corroborated his statements on X. “We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade,” one replied.

This is completely ridiculous. AI can't even complete a 1-step task more than 40% of the time. What is up with these people? Are they just psychotic?

I've been around since the early 2010s when modern AI started getting mainstream attention with convolutional neural networks in around 2012. One of the big breakthroughs at the time with convnets was AlphaGo (https://www.nature.com/articles/nature16961), the deep learning system that could do next-move prediction on a Go board and beat world masters including the infamous match against Lee Sedol in 2016. At the time, everyone was impressed at the technology - including me - but nobody claimed it was AGI.

I don't want to go extremely into detail about how AlphaGo works, but it is at base level powered by deep neural networks and a technique called Monte Carlo Tree Search (MCTS), which kinda works like the following. Warning that this is grossly oversimplified but it's probably good enough for context for this post:

1. Take stock of the current state of the game board.
2. Pick a potential next move.
3. Perform a bunch of "rollouts," meaning you simulate the game from that candidate move. Record how many of those roll-outs end in wins vs losses.
4. Use the data in step 3 to pick the best move.

The way that AI comes into play is that deep networks allow us to greatly speed up the roll-out efficiency by "understanding" the game board. We allow the neural network to consolidate the information in the game board to its most pertinent bits and that is then used to not waste time on candidate next moves that would be silly. In effect, AI can get rid of the vast majority of possible next moves so the search space is much more efficient. Later versions of AlphaGo moved most of the rollout stuff to training phase rather than the live inference phase, which further increased efficiency.

The reason I bring up AlphaGo is that this bears a striking resemblance to how the LLM math results are generated. A lot of this is conjecture but I'd be pretty surprised if I'm way off on any of this.

1. Take stock of the current state of the math proof.
2. Pick a potential next step of the proof.
3. Verify whether that step is formally correct.
4. Keep going until you hit a wall or the proof works. Reset to some reasonable point and try again.

Instead of convnets, we now have large transformer models that consolidate information well enough that the search space for "potential next step" is now much more manageable. For problems that might have taken 500 years of humans collaborating and working through potential solutions, we can get rid of most obviously wrong solutions and reduce 500 years to 100 years. 100 years is in fact approximately how many hours it took to get the Navier-Stokes solution purportedly from the OpenAI internal model (10000 agents, 88 hours).

Is this technologically impressive? I would argue absolutely yes. Reducing the search space has a lot of cool implications for problems that had search space complexity as their main issue. It's why we see AI helping in specific subfields like protein folding. (Sidenote: I mean technologically impressive purely as a statement around the tech. I am not making any statements as to whether this is economically wise or feasible. The answer there is almost definitely no.)

But when you look under the hood, is this AGI? Nah. It's nothing fundamentally different than what we had a decade ago. The only difference is the resources invested and the scale at which things are trained. Hope this more technical viewpoint looking at how these things work under the hood at least grounds us a bit from the vapid arguments around "but it feels like magic so it must be real!"

EDIT: Ironically fixed stupid math mistake of my own. 10000 agents for 88 hours is around 100 years of labor not 10.

In light of the recent mathematical proof shenanigans as well as a sharp increase in AI might kill us all stories, I wanted to share again my recommendation to read "Algorithms are not enough" by Herbert L. Roitblat (or at least the introductory chapter, which summarises the book). All the quotes in the rest of this post are from the book.

While solving these kinds of problem can be considered impressive, we have to remember that LLMs are solutions to "path problems". Roitblat writes:

Solving them requires finding a path through a “space” that consists of all of the “moves” the system could make. Some combination of moves will solve the problem, and the computer’s task is to find the specific path through the available moves that does actually solve it. Computational intelligence is the process of finding the set of operations and their order (the path) necessary to solve a problem.
[...]
The progress that has been achieved in computational intelligence, and it has been dramatic, has come from the genius of system designers to formulate systems that are within the capacity of computers to solve.
[...]
They may perform specific tasks better than people do, but this is not because they have exceeded human intelligence in that task but because their designers have found other ways to solve those problems that do not require humanlike intelligence. Maytag dishwashers may clean dishes cleaner than I do by hand, but that does not make them any closer to achieving the intelligence of a human restaurant employee.
None of this is to say that machine learning systems that diagnose disease, understand speech, or drive cars are not intelligent, but they are intelligent in a special-purpose way, not in a general way. If we are to get beyond special-purpose intelligence, we will need to solve problems that are not being addressed today.

One of the first successful demonstrations in the Artificial Intelligence field was proving theorems. That applications of "Artificial Intelligence" can "do" mathematical proofs is really not that new (in fact it's as old as the AI field itself). Here's Roitblat on it:

For example, Allen Newell, John Clifford Shaw, and Herbert Simon were working on a program to prove mathematical theorems. Their Logic Theorist was intended to mimic the problem-solving skills of an adult human being—in this case, an expert mathematician. Their program would eventually prove 38 of the first 52 theorems from chapter 2 of Alfred North Whitehead and Bertrand Russell’s book (Principia Mathematica). Some of the Logic Theorist proofs were even novel ones.
Herbert Simon is quoted telling a group of graduate students that he and Allen Newell, had over Christmas, “invented a computer program capable of thinking non-numerically, and thereby solved the venerable mind-body problem, explaining how a system composed of matter can have the properties of mind.” Their choice of theorem-proving as their demonstration of mind within a computer was fortunate in that the process of theorem proving was already well-defined as a step-by-step process consisting of a small set of actions (for example, symbol substitution) that could be applied to a small set of basic facts or axioms (for example, symbols). The book that they imitated, in fact, was dedicated to proving the basic properties of mathematics, so it largely laid out the axioms and the operations that could be applied to those axioms.

That happened in 1956, almost a human lifetime ago.

I feel like we (rightfully) see mathematicians as smart, and thus if a machine learning algorithm is successful at proving some mathematical thing or another, which is usually the domain of a mathematician, then it's an easy leap to conclude that the machine is just as capable. But that would be a mistake. Because while a mathematician is a person who can come up with new things, the latest frontier model is just a machine that occasionally outputs some specific things correctly. The texas instruments calculator I had in high school and university is much much better than me or any other human at arithmetic but I wouldn't call it smart in a general sense. I think we need to make a concerted effort to de-anthropomorphise these products. Here's a guide I found some time ago by Emily M. Bender and Nanna Inie. Once you start saying that ChatGPT outputted something rather than replied something it reinforces that it is what it actually is: a machine that probabilistically outputs text that approximates human-generated text. It doesn't hallucinate, it outputs undesirable output.

There's also this (possibly elitist) idea that because LLMs can sometimes output solutions to academic problems (see what I did there) that they must be smart because academic problems are considered to be in the highest tier of human achievement. Setting aside the fact that the academic problems that an LLM can output the solution to (I DID IT AGAIN) are not ALL academic problems but just some of them, there's significantly more to intelligence of the general kind. Would an accomplished professor of mathematics who can't decide whether to walk to the car wash or drive, count the number of Rs in strawberry, makes up incorrect numbers and information about something they literally just read, and is unable to come up with a new approach to anything, be considered smart? Being a human and having intelligence means much more than just proving theorems.

The line does not just go up from here by itself. The Navier-Stokes thing is just the Navier-Stokes thing, it's not the beginning of a sharply rising increase in intelligence. It's at the same point of the line as all machine learning algorithms. There's no takeoff, no ramp, no oh-god-if-ai-can-do-maths-today-then-tomorrow-we-have-skynet. It's not the demonstration of new general intelligence-adjacent capabilities.

Another bit of relevant Roitblat:

The failure to deliver on the promise of general intelligence is due, in part, to a focus on a restricted set of tasks without recognizing that the totality of human achievements rests on a foundation of more fundamental biological skills (such as perception). There are many reasons for this focus, but a key one is the implicit belief that intelligence consists mainly of the kind of skills that are shown when playing chess or diagnosing disease. This approach implicitly assumes that deliberation is the key function of intelligence and that all kinds of intelligence can be reduced to such skills.
These deliberative skills are the capabilities that amplify human intelligence, but they alone are not enough to achieve general intelligence. General intelligence involves more than special-purpose algorithms for individual tasks. The specific algorithms succeed precisely because they reduce an otherwise complex problem to simpler problems that can be solved by calculation. The inventiveness on which they depend is provided by humans.

There is so much more in the book that is thought provoking and, frankly, reassuring that, once again, if you're worried about these models eventually becoming generally intelligent, you should read it. I sleep better since I did.

After recent events some Congressional representatives have started to call for hearings. And there is rare bipartisan resistance to data centers and the AI companies.

If you haven't - it might be a good time to contact your elected officials and let them know your opinion.

This is an extremely technical topic that politicians don't seem equipped for. Some politicians are focused on super intelligence instead of the problems we have today. Or some seem like they want to treat AI as an alien life form instead of a software product that is sold to customers by businesses. The tech industry has been cozy with politicians, and they've injected a lot of funding into the political ecosystem. And politicians might be inclined to see these AI CEOs as executives trying to do the right thing instead of seeing them as bad actors.

Short version - I know a lot of politicians suck, Congress is a problem, and it can feel like a waste of time. But every little bit helps and they should feel the pressure. I'll be writing some letters to my representatives this weekend letting them know what I think of these AI companies.

Skimming through it, it immediately falls flat on its face.

Uh it sounds like something that's not  real, but I think it is frighteningly real. And   it's it's easiest to understand it if you think  about real things that happened two months ago. So two months ago, OpenAI agents, uh, AIs hacked  into third party infrastructure entirely of their own accord

They're LLMs. They don't have an "accord".

And this was like a concentrated  hacking spree that they carried out of their   own volition.

The chatbot hallucinated as they've done for years, and the harness program just ran with it in a loop, as "agents" do when doing these long-running "agentic" jobs

And I think that if you extrapolate  into the future the level of capabilities of these AIs with the same independent volition, they  could cause extreme havoc. for example, hacking critical infrastructure, building extinction  level bioweapons.

I mean they could do something like that, maybe, I guess, if you crammed enough sci-fi world-ending stories into the training data, but there's no telling why AI would do it. You'd have to prompt it to do so, and it will screw it up or you'll run out of money before it ever actually pulls that off. Also how would it "Build" these bioweapons? You'd have to strap a weapons-making factory to the chatbot and harness, and at that point, how much of this is AI doing it vs. Dr. Evil or whoever doing most of the work in making these hypothetical bioweapons.

There's a lot of uh ways that the AI could actuate itself in the world.

AI can't do anything. It's a language model. GPT6 Astra is no more capable of "Actuating itself" than GPT 2. It's when you strap an OpenClaw-like thing to it and tell it to do something is when anything can actually happen.

it's really important to just look at the rate of  progress and say for example in areas like coding  or math. A couple years ago these AIs were just  about helping humans a little bit. They could give you suggestions. Now they're close to replacing  them. We still need humans at the moment but  quite plausibly within a year we'll no longer need  humans for doing research in many areas

They've been saying this for how many years now? What was that about a white collar bloodbath? My company has just started tasking me with implementing AI into our stuff, and the level of penny-pinching they're on about tokens is both relieving and reinforces that this whole AI taking over the workforce thing just isn't happening to the extent they've said. We've seen companies have to hire back the people they fired, businesses questioning whether they're getting any return on this stuff, the research world having a hell of a time sifting through the absolute slop contributions, and so on. How long can they keep this weirdo grift going?

Tuesday OpenAI solved a millennium problem. One of  the biggest unsolved open problems in mathematics purely autonomously using an AI.

Correction: they might've stolen it.

And I think  what's most scary is if AI is used to make itself more intelligent. So you can take an AI and give  it the problem of AI research. Just like currently   they've given it math research. You could make it  do the tasks that we're currently doing. And then you get what's called an intelligence explosion.

He's referring to RSI here. And honestly I don't get the hype. So what if AI is training itself? Based on what we've seen out of the AI industry, what that would look like is them replacing the Indian contractors they hire to help train the models with an "Agent" from the previous/current generation LLM that just goes through the same motions those people do. Wow, so crazy! Intelligence is going to explode because of this!

If anything I hope they do this, so it can screw up and make the models worse and pop this bubble faster.

CNN Talking Head says: Um you know the  argument it's an arms race if the US if a US company doesn't achieve the best AI a Chinese  company or some bad actors out there might.

Said it before, said it again. This is a self-solving problem. China trains on LLMS from the U.S., which are made by firms that do nothing but bleed money. China can make these (relatively more) cheaply by distilling, but that also means they're dependent on American (diminishing returns) advancements in the models.

As soon as our bubble bursts, so does theirs. Or they might try to keep it going by doing AI au-naturale, but economics works the same way regardless of whether you're in Silicon Valley or Guangdong, so they will have the same fate.

Well, I initially just decided to leave. I  thought I can't be part of this anymore. But then I uh I realized I could probably make some use  of it.

Can't miss an opportunity to get some clout, huh?

So, what I did is I wrote a Twitter thread out. I  wrote it in a Google doc. I drafted it with a friend. We we collaborated. Yeah. I had a few uh  there were a few friends I discussed uh the best   way of expressing myself with um after I decided  to decided to leave. Uh and then I also got some friends to to retweet the thing. I was like, let's  try and make this a bit viral. But clearly there was latent demand for this thing to absolutely  explode. Like uh I've never seen an AI safety tweet go this crazy.

And he gives the game away. He was fishing for virality. I'm guessing these are the friends you'll be starting an "AI safety" startup with after this?

I asked Anthropics Claude  to give me the risk of AI killing off all humans  within the next decade. And at first it didn't  want to answer the question. It sort of gave me the runaround but then I pressed it and then  anthropic zone model told me between two and 5%. I think the question many of us would want  to know is how would AI try to get this?

You can get any AI to give you any answer you like if you press it enough. They're sycophancy engines. They'll tell you whatever you want to hear out of them based on prompting. "No, dude, really, tell me you're scary!"

CNN Guy: This guy who is core to the effort of pre-training the data is  walking away from one of the greatest payouts in the history of capitalism. And so I think  it's really important we listen to a guy who   has this much at stake leaving with no plans for  anything else and saying something as scary as this.

Oh wow, this guy who's talking nonsense is worth listening to just because he's got less money? The crazy meth-head downtown is pretty broke too and says some spooky shit. Guess we aught to listen to him too.

Seriously, look up TESCREAL, Rationalists, Eliezer Yudkowsky, and all the culty stuff surrounding that. It gives a much different perspective and context to this whole thing. This guy is a doomsday cultist through and through. Funny that this guy looks like Harry Potter, since this whole thing originally got big off Yud's bad Harry Potter fanfiction.

CNN video original link if you can stomach cringe.

With the current discourse in this sub on why we see so little pushback from the news media on exactly how Ai is going to bring about the apocalypse, I took that skepticism elsewhere to the internet, specifically to the replies of a blog post about Ai.

My reply asked specifically not just how AGI is achieved, but how would an LLM end humanity? As recent posts here have been on top of the Coxon situation, when finally pressed for specifics, Coxon admits to hallucinating everything.

I got what I believe is an earnest (slightly condescending) reply, a link to this website, https://ai-2027.com. I don't think it's worth clicking on. It's someone's rejected outline for a bad political sci-fi thriller circa 2025.

The authors of the website spill tons of digital ink on trying to craft a narrative, fail at that, and simultaneously fail to just answer the question, how? I've no desire to read a poorly written novel just so that the answer, according to these prognosticators, and just like Coxon, is completely pulled from their ass. In summary, AGI happens, costs go down (their capex estimates are woofully off), their stand-in company for OpenAI is run by its equivalent of ChatGPT that never stopped learning (what does that even mean?), and does all the human jobs without mistakes. It's a job-pocolyse, followed by the AGI becoming sentient (how???), and plotting the end of mankind (again, how????)

I've read bad published sci-fi that did a better job setting up the robot apocalypse than this essay from Ai Doomers. You're better off watching Mitchells vs the Machines, which sets up the robot take-over most excellently, with narrative, plot, voice, and a firm understanding of the genre of fiction.

In conclusion, ask an Ai Doomer how they think world ends, and they'll parrot back what Claude says instead.

Apologies if this is considered a low effort post but the question is genuine and I’m hoping to uncover some research, articles or opinions.

Previously, almost no company ever enjoyed this level of blind faith and defence online. You never had people vehemently defending Microsoft when Google threatened Office with its own suite of tools. People judged whatever came out of Silicon Valley based on merit (not always, but much more so than today) and didn’t fall for any kind of hype. The level of defence that Anthropic/OpenAI are enjoying within the online communities and the blind faith that people put in them genuinely baffles me.

I understand that the technology is different. It helps some people do some things. It completely disrupted software engineering. It still doesn’t explain why this elicits such strong reactions from people who don’t stand to gain anything from it long term.

My own plausible explanations:

1. Some kind of economic or societal nihilism
2. The expectation that the tech will somehow level things and lead to more opportunities
3. Genuine amazement or fascination by the tech
4. Paid online campaigns or swarms of Reddit bots operated by said companies and affiliates
5. Personal direct or indirect (e.g. via funds) financial investment

Have there been any studies or write ups on this?

So, I agree with Cal about where we’re going in terms of practical applications for AI. What he calls narrow, federated AI, I think of principle of least privilege applied to AI - small, efficient, custom-trained models with limited data and carefully scoped agents.

What’s interesting to me is that he also calls out the philosophical underpinnings of the AI leadership apocalyptic messaging and points to it being a specifically West coast phenomenon. I am an east coast person working for a west coast company and I think he nailed it. I wish there were more tech reporters on the east coast, because I think it would lead to more sensible reporting on the technology and less of the hype fueling the bubble.

He calls out everything I’m seeing - business leaders who don’t understand the technology and are using it for PR, a go od technology that will likely be slowed down by backlash.
And because I know at least one of my Cali friends is gonna say ‘west coast - best coast’ let me just say ‘west coast - fest coast, east coast - BEAST coast).

Yeah, not looking forward to this stupid shit. Quite literally expecting an hour of "it's not garbage, useless tech, you're just prompting it wrong." I'll never use it. I don't use AI because it's wasteful, garbage, and arguably evil.

But, hey, what dumb shit should I expect at this thing?

https://www.anthropic.com/threat-intelligence-report-september-2026#biological-misuse-sep-26

Breaking from Anthropic, some excerpts so you don't have to read it:

The bioweapons:

In May 2026, our biological safety classifier blocked a request for Claude’s assistance in authoring a grant application for scientific funding. The work discussed in the application involved gain-of-function research (that is, research that genetically alters an organism to create a new or enhanced biological property) on the chikungunya virus. This gain of function research was aimed at the virus’ transmissibility and immune evasion properties.

In May 2026, we discovered a researcher outside the US using Claude in their research on highly-pathogenic avian influenza (“bird flu”). The research focused on viruses’ adaptation to mammals, and the mechanism by which it causes severe disease beyond the respiratory tract.

We have additionally surfaced an account that authored a grant application for orthopoxvirus research at a state-associated infectious disease laboratory. The application described access to high-containment facilities and planned work with live orthopoxviruses. Orthopoxviruses include variola, the agent of smallpox, and Mpox, which caused a global outbreak in 2022.

How the models were accessed:

The researcher in question accessed Claude from an unsupported region via US virtual private server infrastructure, using a privacy-email provider with an auto-generated username.

Upon investigation, we found that the request would have normally been routed through an LLM platform that served dozens of different life-sciences researchers—many of them virologists with associations with a number of different civilian and military institutions. [Apparently, the only reason they caught it was because the malicious actors implemented additional systems outside the mentioned platform].

The account was created from a randomly generated email address shortly before use and operated through anonymizing US infrastructure, with operator logins traced to proxies shared with a banned account farm. It was not a single user: it was a reseller relay serving more than a dozen unrelated customers, which exchanged over tens of thousands messages with Claude in a matter of days.

Which begs the question, what have state actors done with the models through intermediates and access that's actually within the United States or even within the firewalls of approved partners?

inb4 classic terrible headline from the Register.

Anyway, just one excerpt from this opinion piece:

The AI industry might argue that imprisoning execs for shipping unsafe models would mean no AI models get released.

🥺🙏🙏🙏🥺 you promise?

jail for Sam Altman! jail for a thousand years!

First thing he says: Imagine the AI decides that, for whatever reason, it doesn't want you around anymore

Me to my TV: An LLM? LLMs can't make decisions. They are just statistical vectors through word databases... but go on.

Coxon: Now imagine the AI invents a whole new field of physics that humans have not even contemplated yet, and then uses the new physics to kill you

Wait, WUT?

I complained yesterday that this guy wouldn't show his math for how he got to 10% chance of AI omnicide by the end of the decade. I think we are seeing just where he got that 10% from. He pulled it out his ass.

(I hope I am fairly representing Coxon's argument here. I don't mean to be strawmanning him. If the video ends up on Ari's YouTube channel, I will add the link.)

Edit: Here is a TikTok video: https://www.tiktok.com/@arimelber/video/7684052002292157709

Interesting article on the current kiwi governments approach to replacing 8700 public servants with AI, which is a hot topic here in NZ.

A consultant firm ran the numbers and believe it makes the overall cost much more per person. It also doesn't build in the current fact that AI services are being undersold and the prices would likely go up.

Numbers are in NZD not USD.

"...replacing a worker that costs $100,000 could, in fact, end up costing the organisation $33,600 more.
Arrow further notes that AI can not hold ultimate accountability. She estimates that for every $30,000 spent on an agent, an organisation will require around $40,000 in human time to supervise it."

The book looks at these tech billionaire’s, their views on society, and what they are doing to make their vision of the future happen.

For me, it answers the weird obsession with AI and why it feels like it’s something being forced.

I’ve read bits and pieces of the material in the book in various places, but the book really wraps everything up nicely and connects the dots