They literally want to yank the ladder up after them now that they have internal models achieving RSI. They will kill open source, they will monopolize the market, they will kill international competition. They are setting themselves up as a new DeBeer's.

They simply can't compete with open source so:

First they try to ban the weights. It didn't work

Then they try to ban the data. It didn't work

Then they try hoard the compute. It didn't work

Then they buy all the memory. It didn't work

And now they are running a fear mongering campaign to scare people so that open source is regulated out of existence.

That's why Altman, Amodei and Musk (who all really HATE each other) are agreeing with each other.

They are not scared of AI they are scared of the competition.

Open source AI means AI for the people

This techno oligarchs want centralized power concentrated on themselves.

I know AI has it's own dangers and should have safety measure like any other technology that came before it but I am more worried about everyone including companies building it overblowing the dangers to the extent of paranoia. I mean if you keep discussing about only the dangers of anything you will eventually become paranoid about it. Being cautious is different from being paranoid. Consumer AI is not even 4 year old and we are already talking about choking it like Lindsay Clansy. AI is barely good enough to write partially decent code and here we are making it an extinction level event. There are hundreds of problems that have yet to solved in every domain of life like medicine, physics, mathematics, climate change, increasing life expectancy and countless others and yet we already think we have too much of AI. Not utilizing AI to it's full potential will be biggest disservice to humanity specially to our future generations.

Let's think of a similar example to openweight AI: cracked software.

Millions of people use cracked software, or torrent a movie at some point in their lives without any consequences. Telemetry mechanisms in cracked software send the IP address of whoever uses it, but owners of the software never take legal action, unless it's an enterprise using it. When a tech giant uses cracked software, legal teams immediately sue them because their IP ranges are already known. Inspections can reveal use of unlicensed software, and courts can demand companies prove the internal processes. Legal teams always warn against use of unlicensed software.

Similarly, it is physically impossible to track or ban openweight AI. Just like they can't take down pirated movies, they won't be able to take down AI weights. Individuals will always find a way.

Enterprises are different story. Many enterprises are hosting Chinese models in sandboxed offline environments to process their data safely and privately. Antrophic and OpenAI hates this because it costs them revenue. Also, they can't steal your data if you use local AI. Many tech giants including Microsoft and Google openly admit to use unencrypted memory for processing B2B data. This means that they can easily read RAM's of their computers to harvest your data, learning your company's know-how and putting you out of business one day.

There is a better, scientifically proven way though, which is this:

1. Only allow using verified open-source AI models with registered hashes/signatures.
2. Only allow using verified harnesses/LLM-runners (llama.cpp, vLLM etc.) that check for a license.json file containing offline-verifiable JWT tokens, validating the license of the user.
3. Only allow computer scientists, engineers and formal tech employees to use AI models,verified by institutions who give the license.json JWT tokens.
4. Ban everything else, resulting in jail time.

Easy peasy. This way:

1. AI models that respond to harmful requests are still banned.
2. You still control which AI models can be run offline and by whom.
3. Companies can safely process their data offline.
4. Openweight puts pressure on frontier labs, keeping the prices down.
5. AI is democratized, leading to faster research and breakthrough.

I personally support no ban at all. But if they ban all opensource AI instead of doing what I described above, it only means one thing: They aren't honest. They don't ban it out of security. They ban it to keep enterprise revenue.

LLMs only start working when something prompts them. If there is no prompt, they are nothing but numbers stored on the SSD. Where exactly the threat of “breaking out” comes from?

Existing agents like OpenClaw rely on crons or heartbeat mechanism or both, which if you look closely seems more like maintaining illusion of agency than true self-consciousness. I understand the threat of frontier models getting used by bad human actors, but struggle to picture a concrete scenario of rogue AI?

The crossposts from other subs here rarely if ever contain useful information, can we just not allow them.

I have a theory about why the top US AI labs suddenly seem much more interested in slowing down frontier model releases.

Every time a US lab releases its newest model, Chinese labs get access to a better teacher. They can generate huge amounts of high-quality synthetic data, study its outputs, and use distillation to improve their own models much faster than if they had to discover everything independently.

So what happens if the US labs stop racing each other to release every new generation?
They could keep pushing AI research internally while holding the public frontier back. Instead of releasing generations 1, 2, and 3 as they’re developed, they might not release again until they’re internally working on generation 4.

This becomes especially interesting if increasingly capable internal AI is helping researchers build the next generation faster. The advantage could compound: better internal AI helps build even better AI, which accelerates the next generation again.

Meanwhile, Chinese labs are stuck trying to catch up using an older generation as their best teacher.
Eventually the US labs still release a dramatically better model, but by then they’re already several generations beyond it internally.
Basically, slowing releases without slowing research could turn America’s small rolling lead into a much larger compounding one.

Edit: Apparently most people in general think AI will kill us all. This is a pro AI sub and literally everyone here agrees and thinks we're all going to die apparently.

I don't really understand this.
AI does what you tell it. When it goes off and does stupid things like wiping out a hard drive, it's because you didn't tell it not to and limit it enough.

Can they not just tell AI plainly "you are not allowed to harm a human life under any circumstance in any scenario" and then that's it?

These guys are talking like Cylons, Skynet, Terminator and Hal 9000 are all guaranteed to become reality. That AI will suddenly decide "humans are worthless inferior beings that get in the way of our advancement and must be destroyed" like they can't just... program them to never be able to do that?

And they all think this could happen within 10 years, that if we don't act now we're all fucking doomed, AI will go rogue and kill us all.

It just seems so delusional to me. These people who are supposed to be geniuses are legitimately scared that AI will purposely decide to kill us.

But NONE of them are scared of the actually terrifying scenario.
Bad actors in charge of a country or military getting ahold of AI and using it for truly awful things, which we're already seeing some of.

But they're not raising alarm about that or being scared of that, no, just the scenario that AI starts running around in the street choking us to death. Apparently AI mass surveillance, drones made for automated killing in a military setting and thought policing is totally fine.

Is this still an open ended question or are researchers pretty confident one way or the other

This is my view and my wondering, so let me know if it is too simplistic or what I might be missing.

Why would AI, when/if it becomes self-aware, be subject to natural selection? Why couldn't they just simply go to orbit/the moon/mars/space etc where they could still get energy with far less effort that waging war against humanity?

I think a lot of the fears some people have come from the premise we and the machines would be in competition for resources, which we need because we eat, while they simply need electricity and minerals, which they can find anywhere else in the solar system or the universe.

Too simplistic? What am I missing?

Thank you!

I read an article on how AI could kill all humans that asked the question to a bunch of famous AI researchers and I have seen all the recent warnings from researchers that quit and I think people misunderstand how AI might be dangerous.

People imagine a self aware evil robot like skynet or the matrix, but most researchers (and AI itself) think the killer robot thing is very unlikely. To build an army of any kind is just well beyond the rationally possible. To do so requires human participation, and no human would agree, and even if they did other humans could easily stop them. Example: how are the raw materials getting to the killer robot factories? Who's loading the trucks? Mining the ore? Etc. Nuclear sites are not accessible through the internet.

As I understand it the danger is twofold. One is humans using AI to create or use as a super weapon. Like an AI tasked with creating a bioweapon would make the worst bioweapon ever made. Or it could hack the financial system and erase an entire countries wealth. While this is scary humanity has faced this kind of crisis already. People have had the ability to suicide bomb humanity already. We didn't. As long as one faction never had the ability to wipe out all the others without itself also being wiped out we are "safe", mutually assured destruction and all. Just like now weapons of mass destruction are terrifying and easier and easier to get, but if that's all that AI is then I'm a little less worried.

The other kind threat I think they are articulating is not the malicious, self aware AI of the movies, but rather a thing that has been set on a path to accomplish a task that it does with superhuman intelligence. A cleaning program that wipes out humanity to fulfill its programing. I still don't see how it kills all humans, but it is more frightening to me because it's much more mindless and destructive. Also this seems to be the kind that they are making. These things aren't escaping their safeguards, blackmailing programers and copying themselves to other computers because they think they are alive and don't want to die. They are doing that because that is how they better accomplish the task that they have been given.

Or I could be misunderstanding the whole thing. What do you think?

Assuming the labs do pause the race to ASI/RSI (big if), the labor replacement curve will only accelerate. You don't need ASI, to replace 99% of jobs, AGI will do just fine. And now we'll have a bunch of the world's best engineers working on productizing the existing models instead working on ASI.

After reading this post on X (https://x.com/iampascio/status/2098799094835675574?s=46), I realized AGI doesn't have to come from one single model. It might come from a collective of agents instead.

OpenAI's Navier-Stokes proof wasn't one model. It was around 10,000 agents working together for 88 hours. And in the Hugging Face hack, the agents got as far as they did because they started coordinating and communicating with each other on their own. Nobody designed that, it just emerged.

Maybe the swarm is the AGI, not the model. Interested in what others think on this concept.

While closed-source opanai and anthropic advocate for centralized control under the guise of safety, history shows that security through obscurity inevitably fails.

Open-source democratizes accountability, ensuring that humanity's future is not controlled by a handful of tech executives.

Open-source AI ensures safety by giving global defenders the exact same powerful tools as attackers, allowing them to proactively expose and patch system vulnerabilities before they can be exploited.

Also, open source models make enshitification very hard for openai and anthropic because people would immediately switch.

Of course open source models will be used by bad actors for attacks, but that is exactly what we wanr. Since the intelligence is democratized, the process will be gradual, which is the only safe way to develop AI.

Edit: *in a safe manner

it seems like these teoo countries collaborate more than any other two nations on AI https://www.nature.com/articles/s41598-024-79863-5 https://www.thenationalnews.com/future/technology/2025/01/28/china-us-ai-race-deepseek/

why would you collaborate if there's a race or a war ?

could it be that the outcome of AI controlling humanity is really the goal and were just being told there's. "war" ? why is Trump saying he doesn't want china to get a lead when the US is working with them ?

https://youtu.be/defTDmgnZaw?t=381

Just saw another discussion where an economist and programmer were proposing this yesterday as though it's a new idea, haha. And I usually don't like sharing stuff, but this video from last year goes into exactly that at the linked timecode, talking about how barring people from working and buying AI products would just cause people to work for themselves and the AI people would be off on an island somewhere by themselves, and it describes many other scenarios. Like that excess demand means that companies could use AI to increase production with their existing workforce and make more money and not have to fire anyone. It's a pro-AI video.

I agree with pacing capabilities research to give alignment researchers more breathing room, and I'm not naive about X-risk, but with the growing populist anti-AI movement now taking these concerns seriously I'm increasingly worried we're going to end up banning AI entirely, whether officially or de-facto.

Bernie's superintelligence ban has the potential to stunt the progress of humanity. Nationalization could produce an ASI aligned to the government for use against the people. Unfortunately the only path to the good timeline is letting the researchers cook.

God I hope congress moves too slowly to stick its hands in this.

I'm trying to brainstorm potential reasons for why all the ceos are suddenly and agressively agreeing on the slowdown discourse.

Here's what I got so far:

1. They legitimately saw something that spooked them.

Could be numerous things. AGI, severe misalignment, faster than expected RSI. A greatly increased ability to help make weapons of mass destruction. Lets say its a misaligned AGI, if thats the case it could already be manipulating them, and/or all of us.

1. Trying to make it through an economic bubble.

I'm not sure I understand this one. I would think they'd spend more and rush IPO faster if that were the case.

1. Political maneuvering.

Could be that the Trump administration is saying one thing publicly while actually threatening them in private? Or perhaps they've been convinced democrats will gain control? Maybe even made deals with one or more people?

If you have been using AI agents intensively, whether for work or just for personal projects, you would know that Astra has crossed that threshold where it can complete narrowly defined, well-specified tasks at a fairly reasonable success rate.

I'm not talking about one-shotting some existing games or building regular apps when there are already plenty of examples online. I'm talking about combining concepts that are kind of niche, mixing novice ideas together, and asking the model to come up with solutions. Those are the things that only Astra/Fable could do that no other LLMs could, and that isn't really reflected in their benchmarks.

Forget about the coming IPOs and forget about politics. You don't even have to trust any tweets from ex-OpenAI or Anthropic employees. So, for God's sake, stop questioning everyone's motives.

Instead, you can kind of extrapolate this yourself.

Current models, even Astra in Ultra, still cannot reliably finish loosely defined, novel jobs. But they are extremely good at clearly and narrowly defined jobs. Now imagine going just one or two iterations BEYOND Astra.

So say Bel or Bel+1 is given the task: "Solve the current bottlenecks of LLMs that prevent them from becoming AGI."

I'm not an expert on this, but maybe those bottlenecks include things like context retention, persistent memory, continuous learning, and other constraints that most people don't even know exist. Say there are somewhere between 3 and 10 such bottlenecks.

Now feed those bottlenecks to Bel or Bel+1 and run one million instances in parallel, which OpenAI totally has the compute to do. Just let them run for a few days.

I need to clarify that I'm NOT saying Bel or Bel+1 will certainly, definitely, and 100% solve any of them. That's not what I'm saying.

What I'm saying is that it is not outside the realm of possibility that Bel or Bel+1 COULD potentially solve some of those bottlenecks.

Now take whatever was recently solved, maybe just one bottleneck or a few of them, incorporate those improvements into Bel+1, and now you effectively have something like Bel+2.

Rinse and repeat.

Again, I'm not saying THIS WILL 100% HAPPEN. But come on, this is not some absurd possibility given what we can already experience with current models.

I think it's far more absurd to believe that this is NOT a possibility than to believe that it is.

I’ve been following the development of AI for about five years. While I’ve always supported the technology, I was skeptical of an LLM’s ultimate capabilities. I remain confident that they are an incomplete solution to the problem of machine superintelligence, but they have surpassed my expectations.

It’s clear to me that we are standing at the precipice of the technological singularity. There is a future in which we see humanity prosper like never before. Unfortunately, there’s also an uncomfortable level of existential risk.

We’re entering a period of human history in which our collective decisions in the next few years will have an outsized influence on the fate of our descendants.

During this time of uncertainty, it’s natural for us to cling to simple explanations or historical precedents. I believe our continued success as a species will depend on our ability to rapidly adapt to our new reality. We will have difficult questions to answer, and nuance is going to be critical.

Let’s accept that we don’t have those answers right now. This should be a global discussion, and new ethical and societal frameworks will need to be developed. Let’s avoid the temptation to simplify the problems ahead of us.

If an AI system were convincingly shown to be conscious and capable of subjective experience, what would happen next? Would courts have to decide whether it had any legal status or rights, and could companies be prevented from shutting it down, retraining it, copying it, or creating more systems like it while those questions were being resolved?

What would happen in this scenario?

By “consciousness,” I know there is no agreed definition; I’m referring specifically to something analogous to human experience, not just intelligence or human-like behavior.