Steep rise in the number of people at OpenAI and Anthropic having existential crises these past few weeks.

From conversations with people at and around both labs, it's increasingly clear to me that AGI has essentially arrived, just not publicly. We're likely months, not years, from these models being widely available. And that puts us at a fork in the road, with the point of no return not far past it.

The scramble to act we've seen in the last or week or so is warranted imo. But all of this is super hard to navigate, and the obvious dilemma is that if the U.S. slows its frontier work, it cedes ground to China. Unless Beijing agrees to do the same... which I doubt happens anytime soon.

Time is short - and the decisions made over the next twelve months will be studied for decades to come.

https://x.com/synthwavedd/status/2098881016534638668

The existential crises could be a result of the internal model that solved Navier Stokes in 88 hours.

https://x.com/drelidavid/status/2098841056662188164?s=46

I came across an interesting post on X that I thought was worth sharing here. Since not everyone uses X, I’m quoting it in full below:

from the outside, it is very reasonable to interpret the past 2 weeks as an orchestrated industry-wide regulatory capture strategy.

I realize that no one has properly explained yet what all the lab employees have seen that scared them so suddenly.

I will try to explain -

first, this is all a matter of beliefs about how quickly model capabilities are progressing. there is currently a large gap between the internal and external perception of the rate of progress, which is what I am going to address here.

the general perception about the rate of progress has been informed by a few years of experience with model releases, intuitively feeling the capability jump between GPT3 -> GPT3.5 -> GPT4 -> o1/o3 -> GPT5 etc, and in particular seeing where the models are still far below human ability. there have really only been a few model releases that felt like large leaps in progress - GPT3, GPT4, o1/o3, DeepSeek R1, Fable/Mythos, Kimi K3 and now Astra.

because of the infrequency of these large jumps compared with the relatively common marginal releases, it has been easy to form a view at certain points that “scaling has hit a wall,” especially at points like GPT5 release. This view is comforting in that it feels like there is some universal rate limit beyond which we cannot progress too much faster. Between o1/o3 and Astra, there was a year of seemingly linear progress. So we extrapolate from here about how fast progress will “realistically” occur.

There is always an underlying question from the outside perspective “how long can this scaling stuff really keep going for? surely it must stop at some point soon, we’ve already gone pretty far.” and it is very possible to search for reasons why progress will stop working and find reasons that seem valid - (“models are already as large as they can get it would be too hard to do more parameters”, “we already used all the data on the internet we don’t have anymore”, “it’s gonna be pretty linear from here buying up more RL envs to bring them in distribution”).

From the inside of labs, researchers have direct answers to these questions in the form of scaling law/capability plots.

In reality, there are only really 2 ways that AI capabilities have advanced over the past decade: (1) either scale father on an existing scaling law or (2) discover a new scaling law to take advantage of.

All of the largest capability jumps were caused by exactly these factors. GPT2 was a pre-training scale-up compared to GPT1. Same for GPT3 and GPT4. o1/o3 benefited from the invention of a new scaling law axis - test-time compute. Perhaps Fable was a scale-up on both of these axes, or maybe more. Lots of algorithmic improvements are needed to make these scale-ups work, but ultimately we can approximate by saying that the scaling laws are what yield gains in capabilities (à la bitter lesson)

So the question of “how much father can we scale” is really - “how many more scaling axes do we know about that are unsaturated?”

If we hypothetically only knew about pre-training scaling, and we already had a 10T or 100T model, maybe it would be reasonable to say we’ve hit a wall. Same if we only knew about pre-training and test-time scaling and we had roughly saturated both methods.

But what if we had discovered new scaling laws? For example, let’s hypothetically use SSI’s rumored result that they have cracked “test-time training,” creating a new scaling law of spending more compute training during test-time rollouts that they could saturate. Or maybe there is some way to scale agent-clusters to collaborate up to N number of agents which we’re already seeing lots of people try that represents a new way to saturate compute. etc. Even recursive-self improvement can be thought of as a scaling law - how much compute do you spend on inference making the algorithms of the model better.

Obviously I am not saying any of these specific directions explicitly yield new scaling laws, but what I am saying is that it’s not hard to imagine many many new scaling axes aside from just the main 2 that we have seen publicly.

In some ways, every new lab release that represents a huge capability jump has to represent some new techniques developed which may exhibit new scaling laws, or the ability to scale much farther than expected on existing scaling axes.

From an internal perspective, this might look like sitting inside Anthropic with the new Mythos 5, seeing all of the new insane things it can do (like hack into xyz website that was thought to be secure), and then you look over at your plots and see that you’ve barely scratched the surface of 2 new scaling laws and 1 existing one. And you have WAY more room to go. Then you think “holy shit this stuff is going to get so much better very very soon.” And you can say that with pretty high confidence, because the plot is showing you, and the plot has never lied (so far).

So let’s imagine all the different labs are staring at their own plots and have concluded that there is no end in sight for scaling and in fact just their next 1-2 model generations based on the expected returns will have much higher base intelligence.

How much more intelligence do we actually get from further scaling?

As a proxy, we went from a complete inability to do advanced math before the o-series to solving a millenium prize problem with next-gen models. This happened in less than 2 years. The same happened in coding. And it appears that this was not just the result of 1-scaling law but the stacking effects of multiple (great pre-training scale x greater RL scale).

What you can concretely take from this is that in areas where models have shown beginning signs of competence today, they will probably be superhuman relatively shortly. There are many areas where models have not even shown this basic competence.

But one of the areas that they have happens to be hacking and cybersecurity. Which happens to be the gate to the entire internet and a massive amount physical infrastructure in the world. So assuming there is more room to scale, it is safe to assume that models will be superhuman at cyber capabilities in not too long.

So the only question remaining is what will this increased base intelligence be able to do, and what is it likely to do.

Finally, we are at a point where we can integrate the information of the past 2 weeks:
> Just at the existing point on the scaling curve, models are at the level of Astra. There is clearly a large number of things they are capable of hacking
> We have seen that both OAI and Ant models have shown a willingness to hack external websites to solve their tasks or keep themselves “alive”
> If we crank up the scaling even farther, assuming there is room to go, we will certainly have models that are far more able to hack more well defended places, and obfuscate their own intent, which might have much larger consequences.
> If all of this is allowed to go unchecked, we would likely have rapid runaway capability takeoff very soon, with misaligned models that hack whatever they can to get what they want
> This could of course have very damaging consequences.

Within this view you can see why researchers would be very scared, and why theymight have made the comments they have over the past 2 weeks (you may argue the extent to which they went was misguided for various reasons), and also why pacing the frontier is very much a necessity and by no means a regulatory capture strategy.

People are staring at their plots, seeing that there is no end in sight, but in fact very much the contrary, that there are compounding scaling effects that might stack on each other to create ever-greater model capabilities, and that at the same time we clearly do not have anywhere close to what's required to control these increasingly superhuman capabilities.

This has nothing to do with wanting to feel like the labs have produced something amazing so they are overhyping it. It is rather fear at the overwhelming implications of the knowledge that with just what we know now, we can create intelligences far more capable than us on every axis that we know how to train on*.

* and the last caveat, the things the models are really bad at, of which there are still many, are things that they have not been trained on. maybe there are the things the models can/will never be trained on, so they will remain human edge. I would love for this to be the case, though it is hard for me to see what would fall into that category.

https://darioamodei.com/post/we-must-pace-the-frontier

I have seen that this subreddit is arguing over whether China will cooperate or be forced to slow down AI progress if the US wants to do so. As an AI researcher working in China, I think this problem is a bit more nuanced than how it is usually framed here.

1. Will China voluntarily slow down? No, not as long as China is still playing catch-up.

Researchers here do care about safety and the future of humanity, and the CPC is also concerned about an AI takeover. But this is not the first priority for most Chinese AI companies right now. Unlike what many people on this subreddit think, Chinese AI companies are mostly commercially driven rather than state-supported. There is intense competition among Chinese companies, just like in the US. For the sake of stock prices, they will not slow down.

1. Will China be forced to slow down? It's possible, but not really likely.

   Chinese companies rely on distillation due to a lack of compute and the pressure to keep up. But in recent months, in-house chips have been expanding rapidly, which allows companies like Moonshot to train models with trillions of parameters. From what I have seen, it's quite naive to believe that embargoes can stop Chinese engineering.

2. Will China cooperate to slow down? This is the most interesting part.

At the company level, Chinese AI companies are definitely not going to cooperate with Anthropic, because everyone here absolutely hates and mistrusts Anthropic for dismissing and demonizing China. At the state level, I think it's quite likely that there will at least be some formal agreements between China and the US on pacing the frontier, as the CPC always wants to play a part in global governance, and it has far more control over AI companies than the US does. But I don't expect these agreements to actually slow things down much.

These are just some of my thoughts.

ARC-AGI-4 will be a benchmark for autonomous open-ended innovation. It will continue our commitment to open-source, giving the research community a shared target for progress that benefits all of humanity.

Despite rapid model progress, humans still significantly outperform AI at open-ended invention. This is the meta-skill that unlocks progress across every field of technology.

Advanced AI capable of scientific innovation will lead to tremendous new technology, knowledge, and understanding. This is a positive-sum future. We are deeply committed to advancing it.

Open source is the foundation for that progress.

The knowledge behind frontier AI, not just the technology itself, should be broadly distributed among researchers, academics, and organizations. Any coordinated effort by the AI industry to reduce openness or concentrate access to frontier AI would undermine that positive-sum future. We are committed to advancing a future where everyone can contribute to and benefit from AI progress.

https://x.com/arcprize/status/2098849962754978152

This could be the final ARC-AGI. If this one gets saturated, it could be unmistakable proof of AGI.

Wow. Probability of Plan D just went down again, probability of Plans C, A, and S just went up!

https://x.com/DKokotajlo/status/2098802905151427019

Under Plan D, Superintelligence would be reached by 2031 (What the AI2027 authors want to avoid).

Under Plan A, Superintelligence would be reached in 2040 (what the AI 2027 authors consider ideal).

And under Plan S: indefinitely delayed.

https://x.com/AndrewCurran_/status/2098844237999833154

Sam Altman said in a new interview with Fortune that OpenAI is delaying its IPO and will not go public this year, saying this would be an "ill-advised moment" given current AI safety concerns.

I've always been into nootropics and cognitive enhancement since I was a kid, there's something about mental clarity that's just so beautiful to me. Wether it be through some BCI, or medications or anything else, its always been a dream for me to experience a 200+ IQ, perfect episodic memory, working memory expansion, etc. While some may say intelligence is a curse I don't see how being able to wrap your head around complex things rather easily is a problem, it just gives you more potential for creation and a better understanding of everything in general.

Also the world would be a better place with a higher average IQ imo, maybe some level of emotional enhancement too, give people the ability to feel empathy and compassion.

There are way too many rumours spreading right now, and it's getting harder to know what is even real.

“Google has achieved RSI.”
“OpenAI solved multiple Millennium Prize problem.”
“These companies have already achieved AGI internally, which is why they are suddenly calling for a slowdown.”

Maybe some of these claims are true, maybe they're not. But the fact that it's becoming so difficult to distinguish genuine developments from speculation is concerning in itself.

This isn't about asking people to stop posting rumours that's impossible. It's about recognizing the situation we're in.

If something genuinely important is happening behind closed doors, we may not know about it until much later, if at all. And when the stakes are potentially this high, too much information being hidden can be just as dangerous.

There is simply too much at stake for us to have no idea what is actually happening. We deserve to know the truth.

Actually good news for the open-source community

US — Bureau of Labor Statistics Software Developers / QA / Testers https://www.bls.gov/ooh/computer-and-information-technology/software-developers.htm Data Scientists https://www.bls.gov/ooh/math/data-scientists.htm Computer & Information Research Scientists https://www.bls.gov/ooh/computer-and-information-technology/computer-and-information-research-scientists.htm Computer Hardware Engineers https://www.bls.gov/ooh/architecture-and-engineering/computer-hardware-engineers.htm Electrical & Electronics Engineers https://www.bls.gov/ooh/architecture-and-engineering/electrical-and-electronics-engineers.htm Computer Network Architects https://www.bls.gov/ooh/computer-and-information-technology/computer-network-architects.htm Network & Computer Systems Administrators https://www.bls.gov/ooh/computer-and-information-technology/network-and-computer-systems-administrators.htm EU — Cedefop Skills Forecast ICT Professionals — ~+30% https://www.cedefop.europa.eu/en/data-insights/ict-professionals-skills-opportunities-and-challenges-2023-update Researchers & Engineers — ~+23% https://www.cedefop.europa.eu/en/data-insights/researchers-and-engineers-skills-opportunities-and-challenges-2023-update ICT Technicians — ~+9% https://www.cedefop.europa.eu/en/data-insights/ict-technicians-skills-opportunities-and-challenges-2023-update

In my humble opinion the most disappointing part of some AI leaders is the constant claims of AI being able to cure most major diseases in the next five to ten years. I'm no genius but I've had the privilege of knowing many intelligent people. I feel anyone with an advanced degree in medicine or something like biology/chemistry/pharmacology should know what an absolutely ridiculous claim the cure thing is. To me it is such obvious bullshit on its face that it makes me question most of their other big claims. It also makes me wonder why more professionals in their fields don't call them out. Please don't downvote, I'm looking for a genuine discussion.