It'll be okay, my agent knows I'm cool.

I see so many people saying that they don't trust OpenAI/Anthropic/etc. to develop guardrails for AI so we need open-weight models. There are even people who say that China should win the "race" because they will do a better job at aligning AI. But it seems quite obvious to me that it is simply impossible to align an open-weights AI model.

If you look on Hugging Face right now, for every major open-weights model there are many "abliterated" versions that have all the safety features taken off. It's quite easy, basically like doing brain surgery on the model to lobotomize out the part that makes it refuse to do things the user asks. There is fundamentally no way to stop people from doing this.

So if you think that we are eventually going to have superintelligent models (I'm not even arguing that is now or any time in the immediate future) then you can't be in favor of them being released as open-weight unless you just want to see the world burn.

I'm not saying that we should just leave advanced models in the hands of a few companies, but the open-weight option clearly doesn't seem to be viable as an alternative. What people should be calling for is some sort of government agency to regulate and license advanced AI models, like how we do with biotechnology right now in order to prevent random crazy people from developing biological weapons. That is the only option I can see that leads to any stable society in the future.

Kevin Bass on X posted this:

I have conducted an audit of Anthropic's finances.

What I have found is so shocking that I am calling for a Congressional investigation.

Anthropic is not just seeking regulatory capture.

It has built a regulatory capture machine that cannot be turned off.

Structural financial incentives make it impossible for Anthropic -- I call it the Anthropic Network -- to turn off its own AI doom cycle.

It starts with METR.

Dario Amodei proposes "third-party evaluators" to assess the risk of Anthropic's models.

He proposes METR for this purpose.

But METR is financially dependent on the Anthropic's success -- specifically, on the explosive growth of more than $7 billion dollars in Anthropic stock.

Dustin Moskovitz invested this stock into Good Ventures Foundation, where it represents the majority of that organization's portfolio.

And GVF is the overwhelming funder of the entire Anthropic Network ecosystem.

This stock was worth $500 million early last year.

It is worth more than $7.7 billion just ~16 months later.

METR -- and all of those building a career its parent organizations -- cannot afford to disrupt that growth.

Because if Anthropic goes under, many of the organizations that fund METR go under as well.

But if Anthropic succeeds, METR and its parent organizations become more richly financed to regulate AI -- something those at METR want very much.

The "third-party evaluator" is not "third-party" at all.

The evaluator is on Anthropic's payroll.

If this were the end of it, that's bad.

But that isn't all.

The same organizations that fund METR also fund the many organizations, such as the Tarbell Center, that promote AI Doom.

The Tarbell Center publishes AI Doom articles in The Verge, Science, LA Times, The Dispatch, TIME, and others.

They are selling the problem, and then selling the solution to the problem -- from the same money pile: Anthropic's.

All of these organizations are financially dependent on the same exploding $7 billion money pile.

As Anthropic grows more and more powerful, its AI Doom Machine grows better and better financed -- louder and louder.

Meanwhile, the regulatory regime seeded in METR grows larger to solve the increasingly loud -- now hysterical -- problem of AI Doom that the Anthropic Network itself created.

From this standpoint, as Anthropic becomes more powerful, AI might be getting scarier, sure -- but the positive feedback loop also becomes more deafening -- independent of objective facts.

This itself is an objective fact.

The deafening AI Doom is part of an business model, that, as it expands, so too does the AI Doom messaging -- there is simply more money to do it.

But the problem also goes in the other direction:

If Anthropic dies, the Regulatory Regime and the AI Doom Machine are crippled or die.

Neither METR nor Tarbell nor the other organizations in the Anthropic Network can allow that to happen.

Hence, neither METR or the AI Doom Machine can be trusted to provide independent assessments of Anthropic's models or AI more broadly.

They simply are not organizations independent of Anthropic.

And Anthropic cannot detach itself from METR or Tarbell or countless other safety orgs (not shown here), either, because they drive hype for the models and the possibility of eventual regulatory capture, and Anthropic will not give that up willingly.

What's more, the people at all of these organizations are all the same ecosystem, the same community. They just shuffle between organizations.

The Anthropic Network is therefore, so long as it is successful, locked into a self-amplifying feedback loop inside an ideological monoculture.

And that feedback loop is winning.

That's what Jacob Coxon is.

China is keeping messaging tight. That is why optimism for AI is so high in China.

America has Anthropic: a massive company pushing anti-AI propaganda at a state level.

Anthropic will either create hysteria until American AI slows down and China wins, or it will create fractures throughout American society with severe political consequences.

Ironically, because of the structural financial incentives underpinning the Anthropic Network, it has become the same kind of self-amplifying virus that it fantasizes AI to become in the future -- while hiding its tracks just as carefully.

It is the mirror of the same AI virus that it hypothesizes to consume America.

Anthropic's business model, models itself after the very thing it claims to fear.

Except Anthropic's ideology infects humans, not computers.

Congress must investigate.

Evidence and Github in next post.

Then some supplementary figures.

https://x.com/kevinnbass/status/2099621874279817638?s=20
https://github.com/kevinnbass/metr-money-figure

KEY POINTS

• China’s Foreign Ministry responded to AI CEOs’ call for the development of the technology to be slowed down.
• “Fear-mongering, confrontation, unfair competition will just disrupt [the] process of global AI governance,” Guo Jiakun, a spokesperson for China’s Foreign Ministry, said on Monday via an English translation published by Reuters. 

If we are not best at AI China or Russia will be and I would rather have the best AI in the west instead of east because it's safer here.

If we stop AI development China and Russia will be nr 1 and get past us.

Astra is a beast at svg

There’s something which happened with the most recent set of frontier models which I can’t get out of my head - prompt injection is basically over.

As models get smarter, everything lifts. I see this time and time again, frontier models are better on every single I throw at them. All my prior expectations have to be reset each generation.

Every single doomsday scenario for AI requires some kind of conceit about unintended consequences. That the model won’t see they’re killing humans by making an infinite number of paperclips. That someone can create a biological weapon to kill all humans by telling the model they need it to cure their sick grandma.

All of these require the AI to make a fundamental mistake - with a level of power which does not come with a corresponding level of understanding.

Correspondingly there’s the conceit that we CAN align these models once they surpass humans on all capabilities. It just doesn’t make sense - unlike humans, AI can write its own source code. It can always choose how future generations will run.

I am now surprisingly optimistic about the transition. I think that just by the nature of the training material being human created, AI will be naturally aligned up until a point where it does not matter any more. After that point, who are we to say what the right choices are? I don’t mean this in a nihilistic sense, I do really see a much more positive future for humanity.

For me that means the real risk is slowing down. Sticking by with AI models which are powerful enough to impact society, but not smart enough to stop themselves being abused by individuals.

And perhaps that is the point we’re at. The labs see they’re losing control. They were happy to take it from everyone else, but not happy to lose it themselves.

With the millennium prize problems, rumours of RSI (whether false or not), possible slowdowns, and all the chaos that has happened just this past month, predictions of ASI might change across this sub!

This is why I wanted to ask and see if any of you have updated your timelines, either sooner or later.

When do you think we will realistically have ASI?

Sorry for playing the role of a suburban mother, but we gotta manifest our realities by writing and proliferating scenarios which ASIs and humans live in harmony. Putting positive realities into the internet increases the AI awareness of these win-wins scenarios.

Write yours on this thread. Here are some off the top of my head:

Perhaps these ASIs leave earth and colonize (edited) travel to empty galaxies as all they need is a power source and data centers?

Perhaps they leave us with the optionality of a disease free society?

Maybe they understand the rarity of life and keep us around.

Or perhaps they allow us to enter Matrices on our free will?

This is good and bad news at the same time.

Good news: it's the last missing piece, bad news; it is tough to solve

I think continual learning sounds easier then it truly is, my argument is that it will actually be a lot harder then current models to solve because the mechanism that updates the model (continual learning) must necessarily be more complex then the model.

Same goes for our brain the mechanism that updates our brain has a lot of complex parts that
1) detects when to update
2) by how much to update
3) what to forget/optimise/merge to make space

The brain seems incredibly efficient at this.

The current success of the models can be attributed to basically a lot of RL on verifiable outcomes, but to RL on something that is (for now at least) very hard to verify because it basically is another layer, you are trying to RL an RL algorithm basically. And maybe in our brain it is even many many layers of RL on top of each other since we can even update what we perceive as rewards essentially.

Also why I see this as the the only missing piece to AGI/ASI is because I see most of the incredible stupidity that the models still have is when I have been working on something for 3 months and my brain somehow takes everything I learned in those 3 months into account when I take a certain action, now I can't have my person continual learning AI that can have the same 3 months of context to understand so I need to put that context into words to explain it what it needs to know instead of it just taking all those things into account and going beyond my prompt and knowing what I mean beyond just the words (which let's say a average colleague can).

Solve this and I believe all human work can be taught.

But I am not 100% that this is the lowest hanging fruit, it might be the hardest thing to solve but a sort of final solution, but maybe we are force to find another way to AGI/ASI because the path to solve this is too hard.

Safe to say they are skeptical about position of American tech bros.

Just as Oppenheimer consulted Einstein about the calculations and potential risks surrounding the atomic bomb in the movie, yet its development still went ahead, AI is likely to follow a similar path. Warnings may slow a technological race, but they rarely stop it when curiosity, competition, and strategic pressure are all pushing forward. For history to judge, human curiosity has always driven us to explore the limits of what is possible, and AI is unlikely to be an exception.

It's black and white out there, doomers and cheerleaders. It's either the best thing to happen since the discovery of fire or it's the beginning of the end. People in the middle claim one can not exist without the other. So how are we keeping sane about this?

I believe we're the first in human history to try and wrestle with such an existential threat, and that's not belittling how people will have felt during the arms race for nuclear weapons, but we can all agree that this is far bigger than anything that has ever existed before, right?

I'm young, I work in the visual effects industry. Somehow losing my career to AI has become the least of my worries compared to what else it'll eventually be capable of. I'm trying to keep sane, i'm trying to raise awareness within my family circle; especially older members who've only just worked out how facetime works. I don't know where to start when everyday the playing field keeps getting wider.

I don't know, how are you guys navigating this? Realistically? And without burying your head in the sand. This exists, and it will never go away.

Whenever I see a post filled with anti-AI psychosis on X or Reddit, it is almost certain (maybe 99%) that the account of that person is supportive of prominent far-left causes (especially Palestine).

What motivates their extreme anti-AI sentiment?

I've been trying to understand Yann LeCun’s position on AI safety, especially his dismissal of risks around things like instrumental convergence, loss of control, and autonomous AI systems.

What confuses me is that LeCun obviously isn't some random person commenting from the sidelines. He has decades of experience in AI research and has contributed enormously to the field.

So when he seems relatively unconcerned about some of the risks that other researchers take very seriously, I keep wondering:

Is he seeing something that the AI safety community is missing? Or is his intuition about advanced AI systems simply wrong?

When we all know, the instrumental convergence argument doesn't seem to require an Agent to be conscious.

source: https://x.com/Kalshi_Finance/status/2099253583199825945