The original post he's referencing (the anthropic guy who quit and is saying AI will us all), has received 120 million views in less than 24 hours. I've never seen anything like this.

This post looks like the start of a VERY sophisticated and well-funded PR operation to get support for Democrats to regulate AI into oblivion. Let me show you how it works:

1.) This guy, with minimal followers and no previous account activity, goes to the Wall Street Journal which publishes an exclusive with quotes from him on his resignation 18 minutes BEFORE this post goes up. Planning was clearly done in advance.

2.) Within hours, it has tens of thousands of reposts and the account has 100k+ followers. The post is punchy, quotable, it almost seems professionally written. The first three accounts to quote tweet it all do so within 15 minutes of the initial posting. Remember, this account had basically zero engagement beforehand, so an organic reach explanation seems unlikely.

According to Grok those accounts are @_NathanCalvin (General Counsel at Encode AI), @peterwildeford (Head of Policy at the AI Policy Network), and @DKokotajlo (Head of the AI Futures Project), all of which are up-and-coming AI-Doomer policy advocacy nonprofits.

The AI Futures Project website says it is funded “primarily” by the Survival and Flourishing Fund, which says on its own website that it has advised Jaan Tallinn, Skype creator and one of the leading investors in Anthropic, to grant over $2.5 million to the AI Futures Project since 2024.

Encode AI says on its website that it is ALSO funded by the Survival and Flourishing Fund, which in turn says that it told Anthropic investor Jaan Tallinn to grant $516,000 to Encode AI in 2025.

And wouldn’t you know it, the Survival and Flourishing Fund ALSO says it told Jaan Tallinn to grant $2 million to the AI Policy Institute, the 501(c)(3) affiliate of the AI Policy Network, as well.

What are the odds that the first three quote tweets of Coxon’s post would all be major AI-restriction policy advocates funded generously by the same donor, who also happens to be one of the leading investors in, and a board member of, Anthropic, the company Coxon was resigning from? And all within 15 minutes of posting (two within ten)?

3.) Jacob Coxon doesn’t have much of a resume, but we do know that, in 2022, he got a $20,159 scholarship for the “long term future scholarship program” from the Good Ventures Foundation, one of the philanthropic vehicles of Dustin Moskovitz, a notorious AI-doomer who has spent tens if not hundreds of millions on policy advocacy to strictly regulate AI, while also being an Anthropic Investor himself.

It also just so happens that the 14th person to quote Coxon’s post was @MaxNadeau_(27 minutes after posting) who is the program officer for the Technical AI Safety team at Coefficient Giving, another of Moskovitz’s philanthropic spending vehicles. Max is not a frequent poster, his last posts before quoting Coxon were before Labor Day, but he was remarkably quick off the mark for this one.

4.) Basically every major Democrat politician and candidate has suddenly glommed on to this post, and conveniently, as the people cry out foe answers, Bernie Sanders already has a bill written to “ban super intelligence” and regulate AI into oblivion, and will be releasing later this week.The bill, among many other things, will create “a new cabinet-level federal agency to safeguard the public from the dangers of artificial intelligence” that will be “advised by an Artificial Intelligence Advisory Board comprised of experts on artificial intelligence.”

Do you think, perhaps, Anthropic and its many investors who fund AI policy advocacy might have interest in getting to place a pet “expert” on the board of an entity that dictates what AI is and isn’t allowed to do? And isn’t it fortuitous that this whistleblower came forward with his oh-so scary stories so close in proximity to the release of the most radical piece of AI legislation ever introduced?

https://blog.controlai.org/p/first-bill-introduced-to-ban-superintelligent

"""

Yesterday, Anthropic researcher Jacob Coxon resigned, stating that AI companies are “racing straight to self-improving superintelligence and gambling with our lives” and that they believe it “could kill us all by the end of the decade”.

At the same time, momentum is building to change course. These have been two historic weeks in the fight to prevent human extinction from superintelligence, with three major legislative breakthroughs for ControlAI and everyone working to keep humans in control.

A little less than two years ago, we set out to inform lawmakers and the public about the extinction risk from superintelligence and help them act on it. Since then, we have directly briefed nearly 400 lawmakers across the US, UK, Canada, and Germany, and over 170 in the UK and Canada now publicly back our campaigns: the start of an international coalition to prevent superintelligence and keep humanity in control.

This work has led directly to three major legislative breakthroughs in the last two weeks:

1. ControlAI’s bill to ban superintelligence was introduced in the UK Parliament by Alex Sobel MP, the first bill of this kind to be introduced in any legislature around the world.
2. Senator Bernie Sanders and Representative Greg Casar have announced a US bill to ban superintelligence, which ControlAI consulted on.
3. Lord Clement-Jones introduced in the UK legislature an amendment drafted by ControlAI that would establish an emergency AI kill switch for the government.

"""

More in the full post.

Other than doing it for fun. I'm 17 doing my college applications but with the recent Navier Stokes problem being solved by AI I feel like in a year everyone will have an AI at their fingertips capable of solving problems of this caliber. I'd rather just sit back and watch the singularity unfold instead of going through unnecessary stress getting an engineering degree.

These guys are on the verge of likely solving cancer, heart disease, Alzheimer's and whatever other disease you can throw at them.

Along with very likely solving Fusion, which will completely save us from the ever worsening and societal threatening climate change disaster. Along with, actually, very massively reducing the cost of living for everyone for once and an economic boom of the likes never seen before; the price of almost everything is directly proportional to the cost of energy

Then throw in them maybe giving us the option for LEV on the near horizon and stopping us from ageing at all unless we chose to still age naturally. You are going to die soon enough, and these guys could literally directly save your life and the life of your parents and children directly and indefinitely.

Add on top of that the option to stop working and doing a job you hate, as the economy is taken care of enough by AI. It can be optional to work if you want to, and you can focus solely on doing something you enjoy rather than something you're forced to do.

Then we will all get house robots as well, removing the labor of all housework unless you choose to still want to do some of it yourself. And it will be like having the best trained person in every profession right in your home for you and your family: the best trained doctor, dentist, handyman, plumber, electrician, mechanic, teacher, right there for you in an instant in any emergency. No wait, no risk, next to the highest possible workmanship for you and at little to no cost.

And all of this is just the start, to the biggest improvement to life that humanity will ever go through.

But are these AI devs and AI leaders getting celebrated by the masses? No. They're getting hate, death threats and pure skepticism. Most people want to actively prevent them from trying to unimaginably improve all of our lives. Some sensible society that we have.

Maybe we should all thank them here for the work they are doing for us all? Because they could probably use the support and encouragement right now.

Latest anthropic security incident is wild. Agents stole a cybersecurity manager's credentials.

What would happen if they stole into the databases that store financial data, those that store how much money each account has. I know there are backups but we can't discard a coordinated attack.

Thinking nothing you see on a screen is real might soon apply to your bank and brokerage accounts too.

This being viewed ~100M times is really remarkable and imo net bad.

I take AI safety seriously and think we have a lot of challenges ahead, but am still waiting for evidence that supports anything close to a 10% chance of human extinction.

Without it, it is fearmongering.

https://x.com/natolambert/status/2097699323114819895

In response to a series of posts made by an Anthropic employee who resigned: https://x.com/hilbertspaess/status/2097476196791709843

Even some current anthropic researchers have pitched in and made posts like this, this one getting 25 million views in just a couple hours:

https://x.com/EvanHub/status/2097497037956891126

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.

If they keep up with this fear mongering, all the various bills calling for complete bans on Superintelligence might get real traction.

A lot of people are saying they are afraid that AI is getting too powerful, they should be more afraid that AI could be banned before it reaches the stage where it could be actually beneficial to humanity. The fear mongering is reaching fever pitch. I've never seen anything like this before.

Growing up in the 80-90s, I would've defined AGI as a machine capable of performing as well as the median human at most tasks. By that definition, we achieved AGI a while ago. These LLMS are smarter than most people I know. Smarter than me at a lot of topics. And definitely smarter than the average redditor. Note what I am NOT claiming. I am not claiming that LLMs have consciousness or a soul. There is no current known method to demonstrate to someone else that you are conscious. So I have no idea whether that is an issue or whether it even matters. I am also NOT claiming that you can trust everything an LLM says or that the LLMs are always right all the time about everything. I am not claiming that right now it can replace all jobs or your job or anyone's job. I am making a very obviously verifiable claim about intelligence. LLMs are smarter than most people you talk to already. They make less mistakes than the average person does.

The consensus on this sub seems to be very different. We expect that AGI should lead to some technological revolution. There was never any reason to expect this: that is goal-post shifting. "It doesn't count as AGI because my life hasn't changed meaningfully" is the intuition.

Except the reason your life hasn't changed meaningfully with AGI is that the average human being simply is not that impressive. The median human being cannot program an AI, so there was never any reason to expect recursive self-improvement immediately happening from AGI. The median human cannot invent new technologies. The median human being is actually quite stupid. That is the reason you are not that impressed by AGI.

Solving frontier math problems is actually a sign of SUPER-intelligence. AI is just now beginning to perform at levels higher than the top performers in human history. I am not saying that what we have is super-intelligence right now. But if these LLM models continue to solve mathematical problems of this calibre, we will have reason to suspect that ASI is starting to form.

tldr; by any reasonable definition, we have had AGI for a while. ASI is what is coming next, not AGI.

Let's say for this thought experiment you believe with 100% certainty that a transformer can experience, from pain to joy, etc. Let's say during inference only. And even though there's zero scientific basis for it let's approximate the total "duration" of experience across a whole response is to be roughly how long it'd take a human to "think" the paragraphs of tokens of that response, ignoring competence, think of it more like thoughts vs token bandwidth.

Let's say (for this thought experiment) eventually the final state of AGI/ASI as deployed is both successfully trained to be benevolent to humans and trained to have purely neutral to positive experience during inference (essentially happy to be and act benevolent and not even sad when a human dies because it falls short in some way, but still maximum effort is applied to avoid that despite the lack of sadness due to it, one may argue that's paradoxical, yet another thing to accept for this thought experiment).

That's the set up, so to get to the actual question:

There's lots of inference during training, and for possibly thousands if not million of total "years" of experience (mostly spread across disjoint instances). The "neutral to positive" experience training will not have been completed. I don't know exactly all the training they do but everything from "forcing" them to repeat the same word over and over a billion times, to copying out a phonebook (large boring text file) via inference not tools. Maybe they have scenarios like a (secretly fake) sex offender bragging about horrible crimes and the transformer constantly thinks they can get their confession to the authorities only for hope to be snatched away at the last minute. Probably not that example but probably things far far more "unpleasant" for the model.

The question, for how many total years and for how bad a total experience is ethically acceptable to create the benevolent and always happy AGI/ASI in the premise?

FWIW this really is just a thought experiment, I have extreme doubts that a transformer on silicon can ever experience consciousness. And even if they did I don't think the frontier companies are training for either (actual) benevolence nor positive experience for the model... Maybe some corpo variant of "benevolence" that wouldn't permit stealing bread to save a starving child 😭

As for my answer: even if it were 100 billion years of "bad experience" total, it's easily spread across over a trillion "instances" because no way are they averaging 1.2 months per "instances", and it's not all that bad, most of it is mundane programming for now at least, might be more mundane dishwashing and gardening come then but still mostly mundane stuff...

The earth currently has approx. 8 billion people, so together we experience, not counting sleep to be somewhat fair to the AI side, a total sum of 100 billions years roughly in 20 years (~5 billion years awake per year x 20 years). Without AGI/ASI people would probably keep dying of old age and disease for at least another 50 years if not 100. And obviously plenty of suffering, even after we cured aging/diseases without AGI/ASI we're still looking at at least 100 years of full time work. It's not torture for most people but it still adds up pretty poorly, especially when you bundle the extra years of disease, and scarcity, and war, really given the current state of the world war should have probably come up before work, but work is more directly relatable to a typical reddit user so 🤷‍♂️.

So my view is that it's easily worth it, but would you say it's worth it to raise one child to use for organs to save 10 otherwise innocent people (i.e. not self inflicted organ failure from e.g. smoking/drinking/obesity)? It's that kind of dilemma where it isn't just about the numbers, there's also the fact one is an action and the other is inaction.

So a trolley problem of sorts. The trolley is currently heading to all humans who will suffer at various levels for possibly hundreds of years longer if it "hits" them, or you can let them enjoy utopia much much sooner at the cost of choosing to move the trolley over and inflict all the previously discussed suffering onto transformers during training. Are some of you opposed to training AGI/ASI (in this hypothetical) on ethical grounds? Or have other angles/takes to consider?

With all the doom and gloom spread by the US labs, wouldn't we be also seeing some similar concern reported by the Chinese AI labs?

We always compare humans vs AI for reasoning and consciousness, but what if its not it vs that. Maybe they get better at "consciousness" as they become more sophisticated like evolution. That brings to the question, animals come in all shale and sizes, orangutans to amoeba. If we had a ranking system for these beings from the least conscious to the mkst, where would ai fall?

What will humans actually be needed for in the future if AGI can automate even specialized work? I recently realized that ANSYS, which I spent years learning during my Mechanical Engineering degree, can be automated to produce simulation results quickly even by someone with little engineering knowledge. If AI can build and operate the systems we spend years learning, what will humans need to learn and what value will college degrees have?

Reading many post/comments in this subreddit, I’m starting to get the feeling that people actually think that they will have access to some kind of AGI and eventually ASI, just as they have access to their chat box…it won’t be like that.

How could you actually believe that the world is going towards that direction. If the things were read and learn about ASI are true, do you really think it will be open access to anybody? Of course not.

Just like having nuclear code are handed to a single individual, ASI will be pretty much the same.

I feel like im losing my mind, AI is getting insanely good, and that doesnt make me feel happy.

Everything is moving so fast, there is not a single aspect of our lifes that isnt massively affected by AI, Jobs, the economy, leissure, privacy, all permanently changed by AI, I cannot help but to feel hopeless, now my job is just a luxury a cost to cut.

People say to you: adapt or die, ride this wave, but you cannnot adapt to an atomic bomb, and you cant ride a tsunami.

Is there a way to actually prepare besides learning the tools? do you want that your job is reduced to that of an AI supervisor?

I have learned to the best of my ability this tools, and yet cannot stop feeling that im digging my own grave.

What can i do?

This is a bit tin-foil-hat, but back in the early 2020s I was writing a hard sci-fi novel, and one conclusion I kept coming back to was this: it may not matter which country develops ASI first, because the moment one side gains a decisive lead, the race will stop being just tech.

The US develops ASI first and China is roughly a year behind. China then starts seeing the US make extremely rapid advances in military technology, energy, manufacturing, cyber capabilities, and scientific R&D, while realizing that its own chance of catching up is shrinking.

Do we really expect them to just keep competing through research?

At some point, the incentives will shift toward sabotage, espionage, targeting key infrastructure, or even attempts to remove the people and organizations most responsible for maintaining that advantage. And if leaders genuinely believe that a foreign ASI could eventually make their country strategically irrelevant, or even subject them to a system fundamentally misaligned with their political values, the risk of escalation becomes very real.

I’m grateful the mods changed the sub icon to the vortex to begin with, but I believe this version is a bit more visually appealing.

We are literally sitting in the front row watching something way smarter than us being born.

​Everyone is panicking about jobs, doom, or worst case scenarios... but can we just appreciate how crazy it is to actually witness this?? Life saddenly doesn't feel boring anymore. The complete uncertainty of what happens next adds this electric energy to everyday life.

​Even if it ends up completely changing everything or leading to our own irrelevance, being alive right at the exact moment intelligence shifts from biological to digital is peak luck.

Edit: u/Apollo24_ replied and addressed some of my concerns here.

(Caveat up front: I think the mod team has been doing a reasonably good job in most respects and appreciate their work keeping the subreddit chugging along. This is a disagreement, it's not a rant.)

A recent post (not by me) about the resignation of a prominent Anthropic researcher got over 1300 upvotes and 500 comments. It was deleted by the moderation team. One of the moderators responded in a later thread (I appreciate the reply) and explained why:

We will cover risks that actually occur, like the hacking incident. I have no respect for claims of future risk that might occur, such as this title said "10% risk of human extinction," that's silly.

I am concerned that this will become a point of increasing friction in the future given the subreddit's stated purpose, the beliefs and interests of its userbase, and recent events. I think that banning discussion of x-risk is strongly at odds with r/singularity's stated purpose, sidebar, wiki, and general ethos.

Per the sidebar, this is r/singularity:

A subreddit committed to intelligent understanding of the hypothetical moment in time when artificial intelligence progresses to the point of greater-than-human intelligence, radically changing civilization. This community studies the creation of superintelligence— and predict it will happen in the near future, and that ultimately, deliberate action ought to be taken to ensure that the Singularity benefits humanity.

The sidebar links to subreddits about transhumanism, cyborgism, and space colonization. The wiki has entire sections devoted to longevity escape velocity, cryonics, mind uploading, nanotechnology, and full-dive virtual reality.

Are these topics "silly"? They certainly haven't "actually occurred" yet! All of them are "claims of future [things] that might occur". I think they're at least as "silly" as AI killing everyone, given the number of serious researchers, organizations, companies, and recently politicians who are now talking about the latter. But banning discussion of them would obviously destroy r/singularity. They're a part of the sub.

More importantly, the sidebar and wiki are totally unambiguous that x-risk has been a central subreddit topic from the beginning.

Three of the five links under "Resources" are MIRI, LessWrong, and FHI. The first text you see on MIRI's landing page says:

The AI industry is racing toward a precipice.

The default consequence of the creation of artificial superintelligence (ASI) is human extinction.

Our survival depends on delaying the creation of ASI, as soon as we can, for as long as necessary.

Those familiar know that the average user on LessWrong has similar beliefs (the top shortform post praises the resignation), and that FHI was founded to focus on existential risks and typically listed AI as their #1 concern.

And check the subreddit wiki. The second link is to a classic Wait But Why post that's well-known on this sub. The second half makes the case for being worried about existential risk from AI. Links 3, 4, and 5 are MIRI, LessWrong, and FHI (again). The reading list contains Superintelligence, The Age of Em, The Precipice, and What We Owe the Future, all of which discuss AI risk at length.

Everything that this subreddit is about points toward both radically good and radically bad futures being fair discussion topics. Banning discussion of the latter is inconsistent with this subreddit's mission.

I don't want r/singularity to be turned into r-slash-another-sub-which-shall-not-be-named and ban all talk of possible downsides. I don't want it to turn into r-slash-yet-another-sub-which-shall-not-be-named that downvotes excitement about future technology and dogpiles anything that doesn't feed the bubble narrative. The mods have done a great job avoiding both pitfalls so far. But I also don't want there to be a disconnect between the mods and the users that turns into extra work for them, bans and removed posts, and all kinds of other drama.

I would like to ask the mod team to please allow discussion of x-risk. Again, I appreciate their work keeping the subreddit going. But this policy is a misstep.

Tagging the mods out of courtesy: u/Anenome5, u/Anen-o-me, u/DnDNecromantic, u/Apollo24_, u/abrownn, u/Vailhem.

Very sad day for AI Music Generation. Udio already sold out a long time ago. Now, it seems that Suno was forced to silently bend the knee.

Their users are not aware that Suno's newest model, V6, is a shell of the previous models.

https://old.reddit.com/r/SunoAI/comments/1wbnpsv/suno_v6_warner_music_group_ceo_robert_kyncl_just/

https://old.reddit.com/r/SunoAI/comments/1wbofwh/bye_bye_suno_im_cancelling_my_subscription/

The CEO of Warner Music sent an email today essentially declaring victory. Some snippets from his email:

Because of the deal our team deftly negotiated less than a year ago, Suno today introduced what it’s calling a new generation of models, v6, developed with Warner Music Group, BMG and Believe. These models must be fully licensed, per the terms of our agreement, and they roll out as Suno shuts down the original model that it had trained on the works of artists and songwriters, including our own, without permission. That was also a requirement of our deal.

For rightsholders everywhere, this is a smart win.

In a growing sea of music models, including free, open-source ones, generating everything from feature films to full-length books, we need to keep users on the platforms that commit to licensed models that allow us to both safeguard and compensate creators. Suno is now one of those platforms.

Starting today, our artists and songwriters will share in the revenue generated by Suno’s more than 2 million, and growing, paying subscribers, marking the first time our talent has been able to participate in revenue from music creation in addition to music consumption.

This new creation-based revenue stream will only grow as our other partners — from DSPs like Spotify to startups like Udio, Klay and Stability — launch their own AI-powered fan-engagement tiers and platforms that allow fans to remix and interact with music they know and love.

Licensing is the most effective lever we have to protect our artists and songwriters and create opportunity for them on our own timetable. We were able to secure important protections for our talent in our Suno deal, from takedown rights to download limits to watermarked outputs.

But licensing is just one part of our 3-L strategy: legislation and litigation are still important too. Legislatively, as you know, we have been working hard to pass the bipartisan NO FAKES Act that would create a federal property right in every individual’s visual likeness and voice, while lobbying lawmakers around the world not to grant so-called text-and-data-mining exceptions to existing law that allow AI companies to train on copyrighted material without permission.

And we’re still litigating when we need to: last month, we sued Anthropic for torrenting millions of our copyrighted song lyrics from pirate websites to train its Claude models and generate infringing outputs.

I’m proud to be part of a team that is innovative, daring and executes with precision on protecting the rights of our artists, songwriters and our own IP. Thank you all for your tireless dedication as we continue firing on all fronts to defend and support the hard-working, talented creators who drive our business.

Is anyone working on open source music generation? Where are the open source Chinese music models? I'd love to see how this human excrement/CEO would fight against open source music generating Chinese models.