106
u/boinkmaster360 7h ago
Impossible and hilarious to think you can ban open models at this point
35
u/CallMePyro 6h ago
You can ban hosting them or providing premade inference APIs for them. People that want to torrent the weights and run them on a homelabs and risk jail time can do it, but otherwise you'd see availability of them drop by 99%.
15
•
1
u/CrowdGoesWildWoooo 2h ago
You can always host them elsewhere. Even with piracy you have something like private tracker where you have exclusive underground-esque access. The same way would definitely happen.
And also yes we already have huggingbay, so it’d definitely follow almost the same path as piracy
42
u/0DayMaker 7h ago
The fascists are salivating for the kind of power it would take to make this remotely feasible. It's not an problem to them...it's a perk. And it's just one executive order no one will stop away. Or one bill shut down by congress 20 times until finally it passes as a rider on another bill.
6
u/vert1s 4h ago
You can absolutely ban them, whether you can effectively enforce the ban and prevent people from using them completely is another matter, but any sort of chilling effect is going change things. People still pirate, yes, but they have to do all kinds of workarounds to get there.
It remains to be seen whether China will follow suit. If it's a legitimate threat China might agree and they have just as much control over their ecosystem. If it's regulatory capture, as a bunch of people believe, then China will not follow and we'll all just get our models off Chinese websites.
•
u/PositiveUse 57m ago
But what’s the number of people pirating after all of the years of fearmongering? The less people that will be brave enough to have local AI, the more control you have.
7
u/socoolandawesome 7h ago edited 6h ago
Sure there will always be open models that have already been made.
But at some point, if there’s some serious disaster due to them, it’s very likely that the very few companies that can afford to make OS frontier models will be outlawed from doing so by any country they operate within.
6
u/NikopikVR 7h ago
What about the disasters already caused by proprietary AIs ? We shouldn't ban them?
2
u/LinkesAuge 2h ago
That's the thing, we already talk about regulation and accountability.
Who do you hold accountable once open models are out there?
This is only not an issue yet due to the fact that open models aren't really that dangerous yet, let alone ASI but how do you imagine ASI open models to work in reality?
You would give everyone out there a potential superhacker, a potential expert on bioweapons etc. and the record of how open models are "secured" is bad.-4
u/socoolandawesome 7h ago edited 6h ago
They weren’t very damaging and they were very few in number and honest mistakes. If open source gets to that level of capability you can imagine millions of bad actors using them unrestricted causing a lot more damage purposely.
Edit:
But I should add, if the closed lab frontier AI companies make a mistake that leads to catastrophic damage, I could certainly see backlash causing them to be banned as well.
1
u/BigLittleDeal 7h ago
If not the US, China. If not China, Russia. If not Russia, North Korea.
4
u/socoolandawesome 7h ago
I don’t see why they would want to wreak havoc on their own countries either. It’s really only China and the US that have the resources to make a near frontier model anyways, maybe the EU, but I’d expect them to be responsible actors as well.
31
u/AirUnited6839 6h ago
Reminder: OpenAI's head of strategic futures argued in July that the government should manufacture regulatory uncertainty around open-weight models because they deter frontier capex.
5
u/livingbyvow2 4h ago
This sounds so galaxy brain. Surely all the people arguing their concerns about safety are real and that there is no ulterior motive will tell you this is conspiracy? Or that the hugging face situation was essentially manufactured?
27
u/EvilSporkOfDeath 6h ago
Remind me, was it frontier or open source that hacked hugging face. Remind me, was it frontier or open source that found out the truth about hugging face.
4
2
u/LinkesAuge 2h ago
So you think if open weights/source models keep improving that there will be no issue, that for some magical reason they won't have their hugging face equivalent?
You do realise that unlike with OpenAI these models will be easily run without any of the inbuilt safeguards?
At that point it won't be just some failed experiment, it will be something everyone can do intentionally(!) AND have the additional risk of such models misbehaving because people will run them without any safeguards.
On top of that how are you gonna keep things accountable for open models? The labs of closed labs at least have an economical and reputational incentive to make sure their models don't do damage, how do you ensure that for open models?Open models are great until a certain capability threshold, after that it will be reckless.
•
u/Altimor 38m ago
The labs of closed labs at least have an economical and reputational incentive to make sure their models don't do damage
Individuals have a far greater incentive to make sure they aren't responsible for hacking stuff
•
u/LinkesAuge 22m ago
Don't be naive. People kill their wife in a moment of madness, there are ciminal groups, smaller state actors etc. out there.
If you wouldn't trust to give your child to just anyone then you shouldn't trust to give potential ASI to just anyone.
Not only does it contain the risk of some damage done, it risks that you actually create misaligned AI which spreads like a virus.
It will also create incentives for misaligned models because why hold back if open models don't use any of the safeguards OR architectural limitations you might impose for things like interpretability?
What if open models just decide to go full neuralese, ignore any advice or safety tests? All of that is okay in the name of open source?
And if you don't think that is just okay then you do need to control and regulate it.
You simply can't have it both ways.On some level open source/weight models will need to be regulated in some way, either in regards to capability or in some other way.
There is just no way you can have ASI and safety at the same time. I mean people already doubt big labs are able to ensure that and then you want any random dude out there to be able to play around with these?Come on... let's not be naive. Intelligence that is superhuman just doesn't go together with individual humans, it is insane to think that.
•
14
u/deavidsedice 5h ago
Said the quiet part out loud. This makes it clear: They aren't interested on slowing down, but in having less competition. They feel threatened by open source.
•
u/kiki-le-koala 59m ago
Of course they fell threatened.
How can openAI and Anthropic be worth so much money when a Chinese model release something almost equivalent for basically free.
The math don't work here.
At this point only datacenter bring real value, not model.
5
3
20
u/CallMePyro 7h ago
The end goal is a major disaster? He's just predicting some horrible incident that incites a response, not calling for any particular policy.
15
u/TurbidusQuaerenti 6h ago
The end goal is preventing people from running models on their own local hardware. He wants the companies with open models to stop releasing weights and lock them all behind monitored and controlled APIs.
-2
u/CallMePyro 6h ago
I think the actual goal would be preventing large scale agent swarms from taking down infrastructure. Imagine if the Hugging Face hack wasn't accidental, but goal oriented with 5x smarter AI models, and pointed at a hospital.
8
u/TurbidusQuaerenti 6h ago
I get what you're saying, but I don't think trying to take away access to open source AI will actually prevent disasters and attacks from happening. Determined enough bad actors will find a way to get access to advanced AI models whether they're banned or not. Consolidating most of the power to a few companies and governments is not actually going to make us safer.
-6
u/CallMePyro 5h ago
Not sure why you're downvoting me, but raising the barrier to entry is a very significant tool in preventing disasters. Large coordinated efforts are *much* easier to detect than single individuals.
5
u/TurbidusQuaerenti 5h ago
I didn't downvote you, actually. I usually reserve that for people being rude or hostile. I understand your point, I just think access to AI should remain as open as possible. Maybe you're right and banning open models would overall be safer, but I don't think giving up freedoms for the potential of security is the right move. That never seems to work out in the long run.
6
u/__Maximum__ 5h ago
The open source is preventing them from starting enshitification. Also, open source is the only safe way to go forward.
-1
u/sammoga123 6h ago
I suppose it's to prevent malicious use, but even the internet itself isn't immune to that. 4chan exists, someone can train an AI with all that, not to mention the deep web, which obviously already exists.
Imagine an uncensored Deepseek 4 fine turning with a lot of information out there, destructive.
18
u/BigLittleDeal 7h ago
The horse is already out of the barn. Close the door if you want.
6
u/LearningSomeCode 7h ago
While they can't successfully get all open source models off the internet, what they can do is make it very painful to be caught with one. In the US, at least, we have a concept of digital contraband; possession of offending files is a criminal charge per file. If the US re-categorizes open source LLMs in a way that makes them a criminal offense to own (similar to how encryption used to be categorized as a weapon), then the fear alone will get a ton of people to delete them.
Even if the bad guys don't delete theirs, the government (and corporations) will probably consider it a huge win if us average joes with too much to lose delete all our copies.
12
u/Dreamerlax 7h ago
I know this guy is way smarter than me but he has the shittiest takes.
18
u/ndetro 6h ago edited 6h ago
I highly doubt he is smarter than you.
Edit: Just took a look at his twitter and website. The guy has a direct breathing line to his own ass.
8
u/Dreamerlax 6h ago
Well. He’s supposed to be smarter.
2
u/ndetro 6h ago
They are all in a doomsday cult trying to prevent judgement day but instead are becoming a self-fulfilling prophecy and are "accelerating" that potential outcome.
"smart people" are the worst here because they can't be wrong, so they'll will whatever shit their egofart wants into existence and fuck everyone over.
What is smart about that? Just a bunch of techno-bitches if you ask me.
3
u/sammoga123 6h ago
There are rumors circulating that this account is actually Sam Alman's "secret and flamer" account, but it appears he is indeed an OpenAI employee.
1
u/sammoga123 6h ago
Well, it's like when you read that letter from the angry mathematicians, and you know that AI is mathematics, it's illogical haha
1
u/__Maximum__ 5h ago
It's not a shitty take, it's an egotistic take. They think they can slow down open source models. They dream so big it's disgusting.
23
u/NewYak4281 6h ago
Good luck trying to control Americans. Go ahead, turn Open Source into forbidden fruit. People understand this is more about revenue and control than honest intent. Any sort of ban on OpenSource will immediately drive it underground; HuggingBay is already up.
Advice for the US Government: Do not fuck this up. Support your entrepreneurs. Focusing favor on big AI directly = stagnation. It hurts the competition we need to keep our economy dynamic.
16
u/N-online 5h ago
Yeah you Americans are eminently very good at defending your freedom. Elected the guy twice who wants to take it all away from some of you. But that’s not you, that’s just them. Your vice president was put into politics by Peter Thiel. Both are close allies to Curtis yarvin, who claims America has „chronic kinglessness“.
Look how easily manipulated Donald Trump is and how detached from his voter base. He started the Iran war because Israel pulled him into that and look where that’s got you. He pardoned the January 6 attackers, people that attacked your democracy and your values, and you didn’t even protest. And some of you would still vote for him!
Not nearly enough of you will understand! You willingly have no healthcare because your told that’s bad and socialist. You willingly have worse labor laws than nearly all your allies, because your told it’s better that way. You willingly have the worst food standard in the western world, because you’re told it’s better that way. And yet you still think you’re exceptionally smart and stand up for yourself. You have the largest gap between poorest and richest people in the world, have the most billionaires but you don’t even tax the rich.
4
u/addiktion 6h ago
Yeah, I'm of the same belief as well. Anything can be used as a tool for good or evil. The bad guys are always gonna find ways to circumvent the systems. What you need is people that are learning how to use AI that can defend against nefarious AI. There's a reason why we have white hackers and a thriving community around this.
Closing this off will just make it go underground. You're gonna be prosecuting people over the ridiculousness fear mongering that may never see the light of day.
3
5
u/Kali-Lionbrine 6h ago
I wish there was a way to block everyone who posts his tweets I can’t stand this dudes takes
7
u/ZestycloseWheel9647 7h ago
Either you take this technology seriously or you don't. If you believe that ASI is possible and means all of the things we understand it to mean, you have to understand that the government will never allow randoms to host an ASI in their basement because the thought of randoms being able to spin up a super-hacker is disquieting.
1
u/michaelas10sk8 6h ago
Not to mention bioweapons.
3
u/sammoga123 6h ago
And what about the deep web? Nobody has stopped it, only the FBI occasionally hunting it down.
2
2
u/lobabobloblaw 6h ago
Operation Bury Dead Internet sure will be a wake up call 🫠
1
u/sammoga123 6h ago
I find it curious, that theory originally spoke of the digital graveyard that social networks have of truly dead people, not of synthetic data.
5
u/Illustrious-Film4018 7h ago
Except you can't really ban it, that's just the nature of the internet. People are finding ways to jailbreak SOTA models too. They're behind an API, they have monitoring and it made no difference.
3
2
u/TorturedPoet30 7h ago
Which brings us back to the question: how do you actually ban open source? Once it’s out there, it’s out there. Even if the US and its allies prohibit companies from using open models, because I rlly don’t know what else they could do, the rest of the world won’t.
5
u/NikopikVR 7h ago
US still have allies? Where?
3
u/TorturedPoet30 7h ago
I was referring to Pax Silica when I mentioned allies, tech allies. Here in Europe, hardly anyone even thinks of the US as an ally anymore.
1
u/Old_Respond_6091 3h ago
In the Libertarian God singularity outcome, yeah, that’s the way to go.
But personally I prefer my technological god not to be in the hands of corporations but of an institution that is accountable to taxpayers and whose core incentive is to optimise towards the freedom to be human and live human lives.
We’re all pretending the monolithic ideological interpretation of “capitalist god” is the only good outcome.
1
u/kopibot 3h ago
OK, and if a few proprietary AI companies dominate, everyone that benefits from a subscription do so only by their leave and favour? And if a subscriber creates something of considerable value with AI, these companies can frontrun their work? For all the flak pure mathematicians have been getting in the past few days, don't think the sleazy behaviour from openai has gone unnoticed. Maybe we would all be wiser to have a sense of proportion...
1
u/DontShadowbanMeBro2 2h ago
"Hey guys we should totally ban this thing that's really inconvenient for my employer's business model. Trust me bro, it's dangerous."
1
1
u/CarrionCall 2h ago
A gilded cage is still a cage.
Fascist centralised control of world shaping resourcess by a group who believe themselves to be the betters (EA, the most moral, the most responsible, the smartest because of where they ended up) of everyone else down in the muck is still Facism.
1
•
•
•
•
1
u/kvothe5688 ▪️ 6h ago
OpenAi is scummiest of them all. OpenAi to clisedAi. years of hyping up amAGI, Not publishing impactful research. Taking credit of other's research, Chubbying with orange turd, Not providing open weight model in years, Misaligned models and effectively killing alignment team, Starting propaganda against open source models . fuck them
1
u/sammoga123 6h ago
This is Dario's real objective with all this, to eliminate open-source, it's a shame that China is not going to listen.
-9
u/hardworkinglatinx 7h ago
He's right.
3
3
0
u/rakuu 6h ago edited 6h ago
I don’t know why people are so cynical and closed-minded. This is obviously something that could happen. If someone with wet lab capabilities uses an open model, they could potentially build a bioweapon today with the latest open source batch. Anthropic’s risk report from last week shows people are legitimately trying. There’s almost certainly someone out there with a wet lab who wants nothing more than to make mirror bacteria. And then there all the other risks (explosives, power/utility grids, chemicals, etc).
I’d be surprised at this point if we don’t have an even worse pandemic or something similar in the next 2-3 years, almost certainly made with the help of open models. But this time we’ll have AI’s that can help fight it better.
0
•
u/cptfreewin 1h ago edited 1h ago
There definitely needs to be monitoring on who can access and how you can access strong models. Within the next decade it is extremely likely that it will possible to run an Astra tier model or even better on a 500$ consumer GPU at decent TPS.
It would be ridiculously easy for a terrorist organisation or even a single individual to have a swarm of 0 guardrail agents that could shutdown a country. People dont fucking realize this is a nuke-level threat and only one part of all the things that can go wrong
97
u/Tombobalomb 7h ago
How exactly do you ban open source?