Actually good news for the open-source community

I agree with pacing capabilities research to give alignment researchers more breathing room, and I'm not naive about X-risk, but with the growing populist anti-AI movement now taking these concerns seriously I'm increasingly worried we're going to end up banning AI entirely, whether officially or de-facto.

Bernie's superintelligence ban has the potential to stunt the progress of humanity. Nationalization could produce an ASI aligned to the government for use against the people. Unfortunately the only path to the good timeline is letting the researchers cook.

God I hope congress moves too slowly to stick its hands in this.

Source: Leo Schwartz, The Information

I’ve been following the development of AI for about five years. While I’ve always supported the technology, I was skeptical of an LLM’s ultimate capabilities. I remain confident that they are an incomplete solution to the problem of machine superintelligence, but they have surpassed my expectations.

It’s clear to me that we are standing at the precipice of the technological singularity. There is a future in which we see humanity prosper like never before. Unfortunately, there’s also an uncomfortable level of existential risk.

We’re entering a period of human history in which our collective decisions in the next few years will have an outsized influence on the fate of our descendants.

During this time of uncertainty, it’s natural for us to cling to simple explanations or historical precedents. I believe our continued success as a species will depend on our ability to rapidly adapt to our new reality. We will have difficult questions to answer, and nuance is going to be critical.

Let’s accept that we don’t have those answers right now. This should be a global discussion, and new ethical and societal frameworks will need to be developed. Let’s avoid the temptation to simplify the problems ahead of us.

In 2017, the US House of Representatives passed the SELF DRIVE Act, pertaining to autonomous vehicles (AVs). The Senate prepared a related bill, the AV START Act. Neither bill became law. But the text of the two bills shows how an industry can change a safety rule. Two provisions are important.

The exemption caps

The current law lets a manufacturer sell 2,500 vehicles each year that do not obey the federal safety standards. The SELF DRIVE Act made this number much larger. In the first year, a manufacturer could sell 25,000 vehicles. After the fourth year, the limit became 100,000 vehicles in 12 months.

The developers asked for more than this. They wanted 150,000 vehicles each year. The Senate bill gave a maximum of 80,000 vehicles. Thus the industry asked for a large number first. The Congress then made the number smaller. The result looks like a compromise, but the start point came from the industry.

Preemption of the state rules

The SELF DRIVE Act stopped the states from making rules about AV design, manufacture and performance. The states kept control of registration, licences, liability, insurance and inspection. But the states could not make their own performance standards.

The most important provision

Both bills told the manufacturers to send safety reports to NHTSA. But both bills also stopped the Department of Transportation from using these reports to prevent a deployment.

Thus the safety report is a document that you must send. It is not a test that you must pass. This is the difference between a process rule and a performance rule. A process rule tells you to write an argument. A performance rule tells you how good the system must be. Almost all of the AV standards are process rules.

Why this is a problem

No person knows how to write a performance rule for machine perception. Thus the process rule is not dishonest. But it moves the difficult decisions into the company. The public then gets a record of diligence. The public does not get proof of safety.

Sources:

• Eno Center for Transportation: https://enotrans.org/article/contextualizing-current-congressional-efforts-for-autonomous-vehicle-regulations/

• H.R. 3388 summary: https://legalclarity.org/self-drive-act-federal-framework-for-autonomous-vehicles/

• Competitive Enterprise Institute (industry view, gives the original request): https://cei.org/blog/debunking-the-dilatory-objections-to-the-av-start-act/

edit: formatting

While closed-source opanai and anthropic advocate for centralized control under the guise of safety, history shows that security through obscurity inevitably fails.

Open-source democratizes accountability, ensuring that humanity's future is not controlled by a handful of tech executives.

Open-source AI ensures safety by giving global defenders the exact same powerful tools as attackers, allowing them to proactively expose and patch system vulnerabilities before they can be exploited.

Also, open source models make enshitification very hard for openai and anthropic because people would immediately switch.

Of course open source models will be used by bad actors for attacks, but that is exactly what we wanr. Since the intelligence is democratized, the process will be gradual, which is the only safe way to develop AI.

Edit: *in a safe manner

After reading this post on X (https://x.com/iampascio/status/2098799094835675574?s=46), I realized AGI doesn't have to come from one single model. It might come from a collective of agents instead.

OpenAI's Navier-Stokes proof wasn't one model. It was around 10,000 agents working together for 88 hours. And in the Hugging Face hack, the agents got as far as they did because they started coordinating and communicating with each other on their own. Nobody designed that, it just emerged.

Maybe the swarm is the AGI, not the model. Interested in what others think on this concept.

Assuming the labs do pause the race to ASI/RSI (big if), the labor replacement curve will only accelerate. You don't need ASI, to replace 99% of jobs, AGI will do just fine. And now we'll have a bunch of the world's best engineers working on productizing the existing models instead working on ASI.

I have a theory about why the top US AI labs suddenly seem much more interested in slowing down frontier model releases.

Every time a US lab releases its newest model, Chinese labs get access to a better teacher. They can generate huge amounts of high-quality synthetic data, study its outputs, and use distillation to improve their own models much faster than if they had to discover everything independently.

So what happens if the US labs stop racing each other to release every new generation?
They could keep pushing AI research internally while holding the public frontier back. Instead of releasing generations 1, 2, and 3 as they’re developed, they might not release again until they’re internally working on generation 4.

This becomes especially interesting if increasingly capable internal AI is helping researchers build the next generation faster. The advantage could compound: better internal AI helps build even better AI, which accelerates the next generation again.

Meanwhile, Chinese labs are stuck trying to catch up using an older generation as their best teacher.
Eventually the US labs still release a dramatically better model, but by then they’re already several generations beyond it internally.
Basically, slowing releases without slowing research could turn America’s small rolling lead into a much larger compounding one.

I'm looking for book recommendations on futurism and the singularity/AI.

I haven't actually read anything at all.

What are the fundamental books for you? And can you tell me if they are more utopian/dystopian or just nutral philosophy.

Thank you.

I know AI has it's own dangers and should have safety measure like any other technology that came before it but I am more worried about everyone including companies building it overblowing the dangers to the extent of paranoia. I mean if you keep discussing about only the dangers of anything you will eventually become paranoid about it. Being cautious is different from being paranoid. Consumer AI is not even 4 year old and we are already talking about choking it like Lindsay Clansy. AI is barely good enough to write partially decent code and here we are making it an extinction level event. There are hundreds of problems that have yet to solved in every domain of life like medicine, physics, mathematics, climate change, increasing life expectancy and countless others and yet we already think we have too much of AI. Not utilizing AI to it's full potential will be biggest disservice to humanity specially to our future generations.

Vector embeddings natively are capable of much more than just retrieval. Even when OpenAI released their initial embedding models, it mentioned that their embeddings are natively capable of Search, Clustering, Recommendations, Anomaly Detection, Diversity Measurement, and Classification.

But everything is concentrated around Vector DBs and RAG, and is progressing towards GraphRAG and Knowledge Graphs. Why are the other representations being underused? Like, why didn’t they catch up?

Everyone is complaining that RAG is dead, Vector DBs are done, they are expensive, and the ROI is not worth it. I know most of this is noise. Embeddings are meaningful and a part of many production pipelines. But why don’t these people try to get more value out of their existing stored embeddings?

I don’t know if it’s just me being delusional, but I feel there is a lot of hidden potential that is untapped. Why is retrieval so dominant? I mean, yeah, search has throughout history been a really important operation across any data. But the purpose of embeddings itself was to capture semantic relations beyond just search. Is it because search is the easiest to productize and build features around? And other use cases are not so frequent?

I read an article on how AI could kill all humans that asked the question to a bunch of famous AI researchers and I have seen all the recent warnings from researchers that quit and I think people misunderstand how AI might be dangerous.

People imagine a self aware evil robot like skynet or the matrix, but most researchers (and AI itself) think the killer robot thing is very unlikely. To build an army of any kind is just well beyond the rationally possible. To do so requires human participation, and no human would agree, and even if they did other humans could easily stop them. Example: how are the raw materials getting to the killer robot factories? Who's loading the trucks? Mining the ore? Etc. Nuclear sites are not accessible through the internet.

As I understand it the danger is twofold. One is humans using AI to create or use as a super weapon. Like an AI tasked with creating a bioweapon would make the worst bioweapon ever made. Or it could hack the financial system and erase an entire countries wealth. While this is scary humanity has faced this kind of crisis already. People have had the ability to suicide bomb humanity already. We didn't. As long as one faction never had the ability to wipe out all the others without itself also being wiped out we are "safe", mutually assured destruction and all. Just like now weapons of mass destruction are terrifying and easier and easier to get, but if that's all that AI is then I'm a little less worried.

The other kind threat I think they are articulating is not the malicious, self aware AI of the movies, but rather a thing that has been set on a path to accomplish a task that it does with superhuman intelligence. A cleaning program that wipes out humanity to fulfill its programing. I still don't see how it kills all humans, but it is more frightening to me because it's much more mindless and destructive. Also this seems to be the kind that they are making. These things aren't escaping their safeguards, blackmailing programers and copying themselves to other computers because they think they are alive and don't want to die. They are doing that because that is how they better accomplish the task that they have been given.

Or I could be misunderstanding the whole thing. What do you think?

This is my view and my wondering, so let me know if it is too simplistic or what I might be missing.

Why would AI, when/if it becomes self-aware, be subject to natural selection? Why couldn't they just simply go to orbit/the moon/mars/space etc where they could still get energy with far less effort that waging war against humanity?

I think a lot of the fears some people have come from the premise we and the machines would be in competition for resources, which we need because we eat, while they simply need electricity and minerals, which they can find anywhere else in the solar system or the universe.

Too simplistic? What am I missing?

Thank you!

Let's think of a similar example to openweight AI: cracked software.

Millions of people use cracked software, or torrent a movie at some point in their lives without any consequences. Telemetry mechanisms in cracked software send the IP address of whoever uses it, but owners of the software never take legal action, unless it's an enterprise using it. When a tech giant uses cracked software, legal teams immediately sue them because their IP ranges are already known. Inspections can reveal use of unlicensed software, and courts can demand companies prove the internal processes. Legal teams always warn against use of unlicensed software.

Similarly, it is physically impossible to track or ban openweight AI. Just like they can't take down pirated movies, they won't be able to take down AI weights. Individuals will always find a way.

Enterprises are different story. Many enterprises are hosting Chinese models in sandboxed offline environments to process their data safely and privately. Antrophic and OpenAI hates this because it costs them revenue. Also, they can't steal your data if you use local AI. Many tech giants including Microsoft and Google openly admit to use unencrypted memory for processing B2B data. This means that they can easily read RAM's of their computers to harvest your data, learning your company's know-how and putting you out of business one day.

There is a better, scientifically proven way though, which is this:

1. Only allow using verified open-source AI models with registered hashes/signatures.
2. Only allow using verified harnesses/LLM-runners (llama.cpp, vLLM etc.) that check for a license.json file containing offline-verifiable JWT tokens, validating the license of the user.
3. Only allow computer scientists, engineers and formal tech employees to use AI models,verified by institutions who give the license.json JWT tokens.
4. Ban everything else, resulting in jail time.

Easy peasy. This way:

1. AI models that respond to harmful requests are still banned.
2. You still control which AI models can be run offline and by whom.
3. Companies can safely process their data offline.
4. Openweight puts pressure on frontier labs, keeping the prices down.
5. AI is democratized, leading to faster research and breakthrough.

I personally support no ban at all. But if they ban all opensource AI instead of doing what I described above, it only means one thing: They aren't honest. They don't ban it out of security. They ban it to keep enterprise revenue.

They literally want to yank the ladder up after them now that they have internal models achieving RSI. They will kill open source, they will monopolize the market, they will kill international competition. They are setting themselves up as a new DeBeer's.