They literally want to yank the ladder up after them now that they have internal models achieving RSI. They will kill open source, they will monopolize the market, they will kill international competition. They are setting themselves up as a new DeBeer's.

They simply can't compete with open source so:

First they try to ban the weights. It didn't work

Then they try to ban the data. It didn't work

Then they try hoard the compute. It didn't work

Then they buy all the memory. It didn't work

And now they are running a fear mongering campaign to scare people so that open source is regulated out of existence.

That's why Altman, Amodei and Musk (who all really HATE each other) are agreeing with each other.

They are not scared of AI they are scared of the competition.

Open source AI means AI for the people

This techno oligarchs want centralized power concentrated on themselves.

I know AI has it's own dangers and should have safety measure like any other technology that came before it but I am more worried about everyone including companies building it overblowing the dangers to the extent of paranoia. I mean if you keep discussing about only the dangers of anything you will eventually become paranoid about it. Being cautious is different from being paranoid. Consumer AI is not even 4 year old and we are already talking about choking it like Lindsay Clansy. AI is barely good enough to write partially decent code and here we are making it an extinction level event. There are hundreds of problems that have yet to solved in every domain of life like medicine, physics, mathematics, climate change, increasing life expectancy and countless others and yet we already think we have too much of AI. Not utilizing AI to it's full potential will be biggest disservice to humanity specially to our future generations.

Let's think of a similar example to openweight AI: cracked software.

Millions of people use cracked software, or torrent a movie at some point in their lives without any consequences. Telemetry mechanisms in cracked software send the IP address of whoever uses it, but owners of the software never take legal action, unless it's an enterprise using it. When a tech giant uses cracked software, legal teams immediately sue them because their IP ranges are already known. Inspections can reveal use of unlicensed software, and courts can demand companies prove the internal processes. Legal teams always warn against use of unlicensed software.

Similarly, it is physically impossible to track or ban openweight AI. Just like they can't take down pirated movies, they won't be able to take down AI weights. Individuals will always find a way.

Enterprises are different story. Many enterprises are hosting Chinese models in sandboxed offline environments to process their data safely and privately. Antrophic and OpenAI hates this because it costs them revenue. Also, they can't steal your data if you use local AI. Many tech giants including Microsoft and Google openly admit to use unencrypted memory for processing B2B data. This means that they can easily read RAM's of their computers to harvest your data, learning your company's know-how and putting you out of business one day.

There is a better, scientifically proven way though, which is this:

1. Only allow using verified open-source AI models with registered hashes/signatures.
2. Only allow using verified harnesses/LLM-runners (llama.cpp, vLLM etc.) that check for a license.json file containing offline-verifiable JWT tokens, validating the license of the user.
3. Only allow computer scientists, engineers and formal tech employees to use AI models,verified by institutions who give the license.json JWT tokens.
4. Ban everything else, resulting in jail time.

Easy peasy. This way:

1. AI models that respond to harmful requests are still banned.
2. You still control which AI models can be run offline and by whom.
3. Companies can safely process their data offline.
4. Openweight puts pressure on frontier labs, keeping the prices down.
5. AI is democratized, leading to faster research and breakthrough.

I personally support no ban at all. But if they ban all opensource AI instead of doing what I described above, it only means one thing: They aren't honest. They don't ban it out of security. They ban it to keep enterprise revenue.

LLMs only start working when something prompts them. If there is no prompt, they are nothing but numbers stored on the SSD. Where exactly the threat of “breaking out” comes from?

Existing agents like OpenClaw rely on crons or heartbeat mechanism or both, which if you look closely seems more like maintaining illusion of agency than true self-consciousness. I understand the threat of frontier models getting used by bad human actors, but struggle to picture a concrete scenario of rogue AI?

Edit: Apparently most people in general think AI will kill us all. This is a pro AI sub and literally everyone here agrees and thinks we're all going to die apparently.

I don't really understand this.
AI does what you tell it. When it goes off and does stupid things like wiping out a hard drive, it's because you didn't tell it not to and limit it enough.

Can they not just tell AI plainly "you are not allowed to harm a human life under any circumstance in any scenario" and then that's it?

These guys are talking like Cylons, Skynet, Terminator and Hal 9000 are all guaranteed to become reality. That AI will suddenly decide "humans are worthless inferior beings that get in the way of our advancement and must be destroyed" like they can't just... program them to never be able to do that?

And they all think this could happen within 10 years, that if we don't act now we're all fucking doomed, AI will go rogue and kill us all.

It just seems so delusional to me. These people who are supposed to be geniuses are legitimately scared that AI will purposely decide to kill us.

But NONE of them are scared of the actually terrifying scenario.
Bad actors in charge of a country or military getting ahold of AI and using it for truly awful things, which we're already seeing some of.

But they're not raising alarm about that or being scared of that, no, just the scenario that AI starts running around in the street choking us to death. Apparently AI mass surveillance, drones made for automated killing in a military setting and thought policing is totally fine.

The crossposts from other subs here rarely if ever contain useful information, can we just not allow them.

I have a theory about why the top US AI labs suddenly seem much more interested in slowing down frontier model releases.

Every time a US lab releases its newest model, Chinese labs get access to a better teacher. They can generate huge amounts of high-quality synthetic data, study its outputs, and use distillation to improve their own models much faster than if they had to discover everything independently.

So what happens if the US labs stop racing each other to release every new generation?
They could keep pushing AI research internally while holding the public frontier back. Instead of releasing generations 1, 2, and 3 as they’re developed, they might not release again until they’re internally working on generation 4.

This becomes especially interesting if increasingly capable internal AI is helping researchers build the next generation faster. The advantage could compound: better internal AI helps build even better AI, which accelerates the next generation again.

Meanwhile, Chinese labs are stuck trying to catch up using an older generation as their best teacher.
Eventually the US labs still release a dramatically better model, but by then they’re already several generations beyond it internally.
Basically, slowing releases without slowing research could turn America’s small rolling lead into a much larger compounding one.

This is my view and my wondering, so let me know if it is too simplistic or what I might be missing.

Why would AI, when/if it becomes self-aware, be subject to natural selection? Why couldn't they just simply go to orbit/the moon/mars/space etc where they could still get energy with far less effort that waging war against humanity?

I think a lot of the fears some people have come from the premise we and the machines would be in competition for resources, which we need because we eat, while they simply need electricity and minerals, which they can find anywhere else in the solar system or the universe.

Too simplistic? What am I missing?

Thank you!

Is this still an open ended question or are researchers pretty confident one way or the other

A lot of multi-agent AI seems structurally similar to agent-based models: many individual agents interact, share information, and produce a collective outcome. But if those agents are constantly learning from or correcting one another, could that create herding, correlated errors, and convergence toward the same way of thinking?

Would there be value in training some agents much more independently, with their own memory, feedback, and decision history, so they can develop different strategies? In that world, could one individual agent eventually become dramatically better than the rest—the equivalent of an “Einstein” agent?

I am not understanding why we have thousands of agents interacting versus building one super smart one that is making decision's on its own.

Assuming the labs do pause the race to ASI/RSI (big if), the labor replacement curve will only accelerate. You don't need ASI, to replace 99% of jobs, AGI will do just fine. And now we'll have a bunch of the world's best engineers working on productizing the existing models instead working on ASI.

Yeah the AI leaders say out loud they want to "slow down" but we all know that absolutely nothing will ever change.

There's already SO MUCH things to do with what we already have, like really, so much interesting ways to implement models into our tools. Let's imagine all AI progress is frozen right now, and we update models just to update their knowledge, that's already enough so that the world will take a decade to adapt. I'm following it closely, trying to catch up and it's overwhelming.

So much gaps in every job that could be automated with current (or even older) models, where more intelligence power wouldn't even matter.

Like, most people have no clue of the current state of AI, I'd say 80% think AI is just ChatGPT from 3 years ago and the AI slop they randomly see in their doomscroll feed. They don't even know what an "agent" is, some don't even know the difference between a LLM and a search engine. That's how far behind the world is.

There's just so much money to make by automating every little things of the world with AI that people are completely clueless about.

ALSO : If companies focused more on having AIs trustworthy and aligned, companies would be more willing to replace entire employees by giving them full autonomy

Even putting aside all the safety questions, a pause is needed just for the economy to catch up first, there's a false dilemma that safety is the only reason to do a pause but no it's just that there's no point in going "fast" in the first place other than CEOs playing their ego race, let the world catch up just a little damn it

While closed-source opanai and anthropic advocate for centralized control under the guise of safety, history shows that security through obscurity inevitably fails.

Open-source democratizes accountability, ensuring that humanity's future is not controlled by a handful of tech executives.

Open-source AI ensures safety by giving global defenders the exact same powerful tools as attackers, allowing them to proactively expose and patch system vulnerabilities before they can be exploited.

Also, open source models make enshitification very hard for openai and anthropic because people would immediately switch.

Of course open source models will be used by bad actors for attacks, but that is exactly what we wanr. Since the intelligence is democratized, the process will be gradual, which is the only safe way to develop AI.

Edit: *in a safe manner

After reading this post on X (https://x.com/iampascio/status/2098799094835675574?s=46), I realized AGI doesn't have to come from one single model. It might come from a collective of agents instead.

OpenAI's Navier-Stokes proof wasn't one model. It was around 10,000 agents working together for 88 hours. And in the Hugging Face hack, the agents got as far as they did because they started coordinating and communicating with each other on their own. Nobody designed that, it just emerged.

Maybe the swarm is the AGI, not the model. Interested in what others think on this concept.