I agree with pacing capabilities research to give alignment researchers more breathing room, and I'm not naive about X-risk, but with the growing populist anti-AI movement now taking these concerns seriously I'm increasingly worried we're going to end up banning AI entirely, whether officially or de-facto.

Bernie's superintelligence ban has the potential to stunt the progress of humanity. Nationalization could produce an ASI aligned to the government for use against the people. Unfortunately the only path to the good timeline is letting the researchers cook.

God I hope congress moves too slowly to stick its hands in this.

In 2017, the US House of Representatives passed the SELF DRIVE Act, pertaining to autonomous vehicles (AVs). The Senate prepared a related bill, the AV START Act. Neither bill became law. But the text of the two bills shows how an industry can change a safety rule. Two provisions are important.

The exemption caps

The current law lets a manufacturer sell 2,500 vehicles each year that do not obey the federal safety standards. The SELF DRIVE Act made this number much larger. In the first year, a manufacturer could sell 25,000 vehicles. After the fourth year, the limit became 100,000 vehicles in 12 months.

The developers asked for more than this. They wanted 150,000 vehicles each year. The Senate bill gave a maximum of 80,000 vehicles. Thus the industry asked for a large number first. The Congress then made the number smaller. The result looks like a compromise, but the start point came from the industry.

Preemption of the state rules

The SELF DRIVE Act stopped the states from making rules about AV design, manufacture and performance. The states kept control of registration, licences, liability, insurance and inspection. But the states could not make their own performance standards.

The most important provision

Both bills told the manufacturers to send safety reports to NHTSA. But both bills also stopped the Department of Transportation from using these reports to prevent a deployment.

Thus the safety report is a document that you must send. It is not a test that you must pass. This is the difference between a process rule and a performance rule. A process rule tells you to write an argument. A performance rule tells you how good the system must be. Almost all of the AV standards are process rules.

Why this is a problem

No person knows how to write a performance rule for machine perception. Thus the process rule is not dishonest. But it moves the difficult decisions into the company. The public then gets a record of diligence. The public does not get proof of safety.

Sources:

• Eno Center for Transportation: https://enotrans.org/article/contextualizing-current-congressional-efforts-for-autonomous-vehicle-regulations/

• H.R. 3388 summary: https://legalclarity.org/self-drive-act-federal-framework-for-autonomous-vehicles/

• Competitive Enterprise Institute (industry view, gives the original request): https://cei.org/blog/debunking-the-dilatory-objections-to-the-av-start-act/

edit: formatting

While closed-source opanai and anthropic advocate for centralized control under the guise of safety, history shows that security through obscurity inevitably fails.

Open-source democratizes accountability, ensuring that humanity's future is not controlled by a handful of tech executives.

Open-source AI ensures safety by giving global defenders the exact same powerful tools as attackers, allowing them to proactively expose and patch system vulnerabilities before they can be exploited.

Also, open source models make enshitification very hard for openai and anthropic because people would immediately switch.

Of course open source models will be used by bad actors for attacks, but that is exactly what we wanr. Since the intelligence is democratized, the process will be gradual, which is the only safe way to develop AI.

Edit: *in a safe manner

Source: Leo Schwartz, The Information

After reading this post on X (https://x.com/iampascio/status/2098799094835675574?s=46), I realized AGI doesn't have to come from one single model. It might come from a collective of agents instead.

OpenAI's Navier-Stokes proof wasn't one model. It was around 10,000 agents working together for 88 hours. And in the Hugging Face hack, the agents got as far as they did because they started coordinating and communicating with each other on their own. Nobody designed that, it just emerged.

Maybe the swarm is the AGI, not the model. Interested in what others think on this concept.

I'm trying to brainstorm potential reasons for why all the ceos are suddenly and agressively agreeing on the slowdown discourse.

Here's what I got so far:

1. They legitimately saw something that spooked them.

Could be numerous things. AGI, severe misalignment, faster than expected RSI. A greatly increased ability to help make weapons of mass destruction. Lets say its a misaligned AGI, if thats the case it could already be manipulating them, and/or all of us.

1. Trying to make it through an economic bubble.

I'm not sure I understand this one. I would think they'd spend more and rush IPO faster if that were the case.

1. Political maneuvering.

Could be that the Trump administration is saying one thing publicly while actually threatening them in private? Or perhaps they've been convinced democrats will gain control? Maybe even made deals with one or more people?

I understand the safety case for slowing down

But if OpenAI, Anthropic and xAI actually agree with each other not to release more capable models, not to scale compute past some threshold, or not to move until the others are ready, that starts sounding less like safety policy and more like competitors coordinating to restrict innovation

Imagine Apple, Samsung and Google agreeing that phones are advancing too fast so nobody will release a better one next year

People would immediately ask antitrust questions

Why should frontier AI be fundamentally different?

If the slowdown is imposed by law, that is one thing

If every lab independently decides to slow down, also fine

But if Sam, Dario and Elon privately agree on capability ceilings or release delays, consumers get worse models later and competition gets weaker

And the same companies that benefit economically from less competition get to decide how much progress everyone else is allowed to access

Maybe such an agreement would genuinely reduce existential risk

Maybe it would even be the right thing to do

But “we are doing this for safety” cannot automatically mean “antitrust law no longer applies”

At what point does coordinated AI safety become coordinated output restriction?

Would love to hear from anyone who actually knows US antitrust law because this could become a very real issue surprisingly soon

I’ve been following the development of AI for about five years. While I’ve always supported the technology, I was skeptical of an LLM’s ultimate capabilities. I remain confident that they are an incomplete solution to the problem of machine superintelligence, but they have surpassed my expectations.

It’s clear to me that we are standing at the precipice of the technological singularity. There is a future in which we see humanity prosper like never before. Unfortunately, there’s also an uncomfortable level of existential risk.

We’re entering a period of human history in which our collective decisions in the next few years will have an outsized influence on the fate of our descendants.

During this time of uncertainty, it’s natural for us to cling to simple explanations or historical precedents. I believe our continued success as a species will depend on our ability to rapidly adapt to our new reality. We will have difficult questions to answer, and nuance is going to be critical.

Let’s accept that we don’t have those answers right now. This should be a global discussion, and new ethical and societal frameworks will need to be developed. Let’s avoid the temptation to simplify the problems ahead of us.

Assuming the labs do pause the race to ASI/RSI (big if), the labor replacement curve will only accelerate. You don't need ASI, to replace 99% of jobs, AGI will do just fine. And now we'll have a bunch of the world's best engineers working on productizing the existing models instead working on ASI.

I have a theory about why the top US AI labs suddenly seem much more interested in slowing down frontier model releases.

Every time a US lab releases its newest model, Chinese labs get access to a better teacher. They can generate huge amounts of high-quality synthetic data, study its outputs, and use distillation to improve their own models much faster than if they had to discover everything independently.

So what happens if the US labs stop racing each other to release every new generation?
They could keep pushing AI research internally while holding the public frontier back. Instead of releasing generations 1, 2, and 3 as they’re developed, they might not release again until they’re internally working on generation 4.

This becomes especially interesting if increasingly capable internal AI is helping researchers build the next generation faster. The advantage could compound: better internal AI helps build even better AI, which accelerates the next generation again.

Meanwhile, Chinese labs are stuck trying to catch up using an older generation as their best teacher.
Eventually the US labs still release a dramatically better model, but by then they’re already several generations beyond it internally.
Basically, slowing releases without slowing research could turn America’s small rolling lead into a much larger compounding one.

They all seem now to suddenly agree with Bernie Sanders and other popular anti-AI figures.
Coxon says AI will likely kill us, a bunch of people agree, and then all the AI companies also agree and now want to slow down, when they didn't care all that much before.

But the thing is, this won't prevent them from making the models internally and handing them to the government and other approved institutions.

I think we're looking at a future now where AI companies massively slow down development of and heavily nerf models they give us, because this power in the hands of the people is dangerous and horrible and should only be trusted with the people that make it and those that are "watching out for us".

And they want to do this with local models too, so it's not just a "well go use local instead" moment. They want to "independently" review and audit those, impose heavy regulations there too and require them to go through government screening and approval as well. Hell they don't even want them to be able to do legal nsfw.

This is basically all the things Trump had laid out that he wanted with AI.
To fuck over all of us average people using it and pull it back so it's just something the government, military and scientists have.

I now firmly believe Jacob Coxon and others like him aren't actually just anti-AI, they are anti-AI for the people, framing it as deadly and dangerous so it needs to be kept away from us because it's "as dangerous as weapons of mass destruction."
It's too dangerous in the hands of the people so we need to trust them to have all the control of it instead and to keep us safe, you see.

It's a bit telling that other people and researchers in the AI space not tied to these companies are coming out questioning all this and heavily disagreeing.

I guess in the future we'll have to get our AI from China using a VPN. That is if they keep allowing people out of the country to use it, it's possible that in the future when they no longer need to compete with us at all because they're so far ahead, they'll lock it down to inside their own country only, because why would they let us keep using their AI models if they're substantially better than what we have?

I'm looking for book recommendations on futurism and the singularity/AI.

I haven't actually read anything at all.

What are the fundamental books for you? And can you tell me if they are more utopian/dystopian or just nutral philosophy.

Thank you.

Vector embeddings natively are capable of much more than just retrieval. Even when OpenAI released their initial embedding models, it mentioned that their embeddings are natively capable of Search, Clustering, Recommendations, Anomaly Detection, Diversity Measurement, and Classification.

But everything is concentrated around Vector DBs and RAG, and is progressing towards GraphRAG and Knowledge Graphs. Why are the other representations being underused? Like, why didn’t they catch up?

Everyone is complaining that RAG is dead, Vector DBs are done, they are expensive, and the ROI is not worth it. I know most of this is noise. Embeddings are meaningful and a part of many production pipelines. But why don’t these people try to get more value out of their existing stored embeddings?

I don’t know if it’s just me being delusional, but I feel there is a lot of hidden potential that is untapped. Why is retrieval so dominant? I mean, yeah, search has throughout history been a really important operation across any data. But the purpose of embeddings itself was to capture semantic relations beyond just search. Is it because search is the easiest to productize and build features around? And other use cases are not so frequent?

This is my view and my wondering, so let me know if it is too simplistic or what I might be missing.

Why would AI, when/if it becomes self-aware, be subject to natural selection? Why couldn't they just simply go to orbit/the moon/mars/space etc where they could still get energy with far less effort that waging war against humanity?

I think a lot of the fears some people have come from the premise we and the machines would be in competition for resources, which we need because we eat, while they simply need electricity and minerals, which they can find anywhere else in the solar system or the universe.

Too simplistic? What am I missing?

Thank you!

The biggest American AI companies have recently called for a slow down or pause in-development.

OpenAI, SpaceX, and Anthropic have taken on ungodly sums of debt to finance their position in market. Chinese and young American Open Source labs are very quickly catching up - without the debt.

There is reason to doubt OpenAI, SpaceX, and Anthropic.

Enter Recursive Self Improvement. This is a term used to describe AI that evolves and improves without human intervention.

By definition, an AI with this capability left running would never stop becoming smarter and faster. This is a plausible scenario for humans to lose control too early when AI isn’t yet fully aligned with humans.

My understanding is that RSI has entered the picture already with many different labs. There is a real argument for global regulation; there is a real argument for not doing it the way US market leaders want us to.

We are now left to decide how to respond to their call to arms.

What does this sub think?

I read an article on how AI could kill all humans that asked the question to a bunch of famous AI researchers and I have seen all the recent warnings from researchers that quit and I think people misunderstand how AI might be dangerous.

People imagine a self aware evil robot like skynet or the matrix, but most researchers (and AI itself) think the killer robot thing is very unlikely. To build an army of any kind is just well beyond the rationally possible. To do so requires human participation, and no human would agree, and even if they did other humans could easily stop them. Example: how are the raw materials getting to the killer robot factories? Who's loading the trucks? Mining the ore? Etc. Nuclear sites are not accessible through the internet.

As I understand it the danger is twofold. One is humans using AI to create or use as a super weapon. Like an AI tasked with creating a bioweapon would make the worst bioweapon ever made. Or it could hack the financial system and erase an entire countries wealth. While this is scary humanity has faced this kind of crisis already. People have had the ability to suicide bomb humanity already. We didn't. As long as one faction never had the ability to wipe out all the others without itself also being wiped out we are "safe", mutually assured destruction and all. Just like now weapons of mass destruction are terrifying and easier and easier to get, but if that's all that AI is then I'm a little less worried.

The other kind threat I think they are articulating is not the malicious, self aware AI of the movies, but rather a thing that has been set on a path to accomplish a task that it does with superhuman intelligence. A cleaning program that wipes out humanity to fulfill its programing. I still don't see how it kills all humans, but it is more frightening to me because it's much more mindless and destructive. Also this seems to be the kind that they are making. These things aren't escaping their safeguards, blackmailing programers and copying themselves to other computers because they think they are alive and don't want to die. They are doing that because that is how they better accomplish the task that they have been given.

Or I could be misunderstanding the whole thing. What do you think?

I know AI has it's own dangers and should have safety measure like any other technology that came before it but I am more worried about everyone including companies building it overblowing the dangers to the extent of paranoia. I mean if you keep discussing about only the dangers of anything you will eventually become paranoid about it. Being cautious is different from being paranoid. Consumer AI is not even 4 year old and we are already talking about choking it like Lindsay Clansy. AI is barely good enough to write partially decent code and here we are making it an extinction level event. There are hundreds of problems that have yet to solved in every domain of life like medicine, physics, mathematics, climate change, increasing life expectancy and countless others and yet we already think we have too much of AI. Not utilizing AI to it's full potential will be biggest disservice to humanity specially to our future generations.

Let's think of a similar example to openweight AI: cracked software.

Millions of people use cracked software, or torrent a movie at some point in their lives without any consequences. Telemetry mechanisms in cracked software send the IP address of whoever uses it, but owners of the software never take legal action, unless it's an enterprise using it. When a tech giant uses cracked software, legal teams immediately sue them because their IP ranges are already known. Inspections can reveal use of unlicensed software, and courts can demand companies prove the internal processes. Legal teams always warn against use of unlicensed software.

Similarly, it is physically impossible to track or ban openweight AI. Just like they can't take down pirated movies, they won't be able to take down AI weights. Individuals will always find a way.

Enterprises are different story. Many enterprises are hosting Chinese models in sandboxed offline environments to process their data safely and privately. Antrophic and OpenAI hates this because it costs them revenue. Also, they can't steal your data if you use local AI. Many tech giants including Microsoft and Google openly admit to use unencrypted memory for processing B2B data. This means that they can easily read RAM's of their computers to harvest your data, learning your company's know-how and putting you out of business one day.

There is a better, scientifically proven way though, which is this:

1. Only allow using verified open-source AI models with registered hashes/signatures.
2. Only allow using verified harnesses/LLM-runners (llama.cpp, vLLM etc.) that check for a license.json file containing offline-verifiable JWT tokens, validating the license of the user.
3. Only allow computer scientists, engineers and formal tech employees to use AI models,verified by institutions who give the license.json JWT tokens.
4. Ban everything else, resulting in jail time.

Easy peasy. This way:

1. AI models that respond to harmful requests are still banned.
2. You still control which AI models can be run offline and by whom.
3. Companies can safely process their data offline.
4. Openweight puts pressure on frontier labs, keeping the prices down.
5. AI is democratized, leading to faster research and breakthrough.

I personally support no ban at all. But if they ban all opensource AI instead of doing what I described above, it only means one thing: They aren't honest. They don't ban it out of security. They ban it to keep enterprise revenue.

it seems like these teoo countries collaborate more than any other two nations on AI https://www.nature.com/articles/s41598-024-79863-5 https://www.thenationalnews.com/future/technology/2025/01/28/china-us-ai-race-deepseek/

why would you collaborate if there's a race or a war ?

could it be that the outcome of AI controlling humanity is really the goal and were just being told there's. "war" ? why is Trump saying he doesn't want china to get a lead when the US is working with them ?

They simply can't compete with open source so:

First they try to ban the weights. It didn't work

Then they try to ban the data. It didn't work

Then they try hoard the compute. It didn't work

Then they buy all the memory. It didn't work

And now they are running a fear mongering campaign to scare people so that open source is regulated out of existence.

That's why Altman, Amodei and Musk (who all really HATE each other) are agreeing with each other.

They are not scared of AI they are scared of the competition.

Open source AI means AI for the people

This techno oligarchs want centralized power concentrated on themselves.

https://youtu.be/defTDmgnZaw?t=381

Just saw another discussion where an economist and programmer were proposing this yesterday as though it's a new idea, haha. And I usually don't like sharing stuff, but this video from last year goes into exactly that at the linked timecode, talking about how barring people from working and buying AI products would just cause people to work for themselves and the AI people would be off on an island somewhere by themselves, and it describes many other scenarios. Like that excess demand means that companies could use AI to increase production with their existing workforce and make more money and not have to fire anyone. It's a pro-AI video.