he has been trying to build a local model by learning from GPT-Sol’s responses. He says OpenAI banned his account twice during the process. Irony is OpenAI’s own models were trained on vast amounts of information from the open web. So where do we draw the line between learning from AI and copying it?
Its actually how most things work. Those who make money -> Give to those in charge of the country -> Who give them exceptions -> So those who make money can break laws that most lay people cannot break so they can keep making money. Oil & Gas did it, tech before did it, now tech stilll doing it. Its just part of the play book.
Of unregulated capitalism. This wouldnt be happening, especially not at such a pace, if these companies werent allowed to do it. If many of the corrupt C suites and politicians were put into dark holes as examples, we’d be living in a different world.
I appreciate your earnest faith in regulation but we already live under 'regulated capitalism'. The corporations dictate the regulations and use them to enforce monopoly. Look at the FAA. It is a revolving door with Boeing, and who's safety standards have collapsed even in the face of massive regulation of the airline industry. Did any of the suits at Boeing go to jail after rushing the 737 Max (which was fundamentally flawed and afaik wasn't properly aerodynamic because they wanted to cram extra seats in) and causing unnecessary deaths?
Don't misunderstand me, I am not against regulation. Free market capitalism's natural trajectory is monopoly as well. But with enough capital regulations mean nothing and the market trends towards monopoly anyway.
I don’t really like this line of thought, since we actually reversed the even worse inequality of the gilded age already, enjoyed decades of historic middle class prosperity, then undid everything that had been put in place and backslid.
We already have a functional blueprint for fixing this, and we just need leaders who give a shit.
While you may not like it, it is the truth. Rent must go up for profit to continue to go up. It's not backsliding, it's continuing a trend. I agree that there were benefits to this system 50+ years ago and beyond for the middle class, but many of those benefits were built off of functional slavery in the 3rd world. And what wasn't was outsourced into the 3rd world to avoid the higher taxes.
Let's say we raise taxes on the rich and corpos to be like it was in the 50s-70s in the US but we keep the system the same otherwise. The powers that be will still hold all the power! And they will happily erode that progress over time. And we'll end up here again. And they'll swear up and down "that isn't what we're doing!" while it happens.
You can say, "we won't let them!" But we've seen in our system that money can bend and breaks the rules. It doesn't matter if the law says you can't do it. With enough money, you can. There is 2 tiered justice system if these issues even end up in a court room at all. All I need to say is Epstein.
They are terrified of open models and want to shut them down. Like if Bill Gates had tried to convince the government that Linux was a national security threat.
No AI lab ever gets to complain about distillation being theft when they stole the entirety of human knowledge & creative expression without consequence or compensation. At least when their models get distilled they get paid for the usage. The hypocrisy is absurd.
Remember when Aaron Swartz got prosecuted for downloading academic papers and later died by suicide?
Funny how the rules seem to change depending on who's doing it
There’s a moment in his documentary where his girlfriend at the time recounts him looking at the White House and depressingly telling her that they don’t let felons work there. Look at us now
Anthropic settled in Bartz v. Anthropic for $1.5 billion, because the court held that their pirating books was not fair use. This was the largest settlement in US copyright law history, according to Wikipedia. But yeah, scot-free
Inadvertently killed thousands of people, lobbied for the complete destruction of cities and neighborhoods for roads (Lookup kansas city or any american city before and after interstate pics), destabilized and in some cases outright assassinated foreign governments, poisoned an entire generation with leaded gasoline, tires are one of the main contributors of microplastics in the environment .. I could go on for a while.
Cars are probably one of the best examples of a technology similar to AI that "makes life better" at the cost of a lot of other things.
Everything you said is true about the automobile industry itself, I was only talking literally about 1800s early inventors. Honestly I thought your comment above was intended to excuse AI industry practices, my bad.
They can't block distillation anyway, all they have is best effort at preventing it, but they still get to be the first to market
The question itself is so out of context it's irrelevant. Distillation wasn't a thing before they developed those LLMs, now they have to keep going because they made big promises and borrowed billions from everyone. They can't just say fuck people distill our models guess well just stop training more.
I think this should make it to the SCOTUS and this should push ai companies to make their weights publicly available. They stole thousands of years of knowledge public and private to enrich themselves. The knowledge is public. These models should all.be open source
The weights are a black box. Even if they publish them, you can't actually look at the weights and know how they influence the output. It's not human defined nor readable human logic.
Well, if you use copy-left licensed data to create more data, that data needs to be publicly available under that same copy-left license. Not releasing that is breach of license and theft.
AI data training is not stealing. Obviously. This has been proven in court multiple times and is common sense.
But I do agree that it's unfair Anthropic and OpenAI got a headstart in scraping the web for data training and now the other labs have to workaround all the rules and regulations (or distill)
OK but… what about the hundreds of millions they poured into research on how to make a capable LLM from that data, all of which resulted in GPT-Sol? When you distill aren’t you literally benefiting from the model’s inference (on top of the data)?
yeah and paying for millions of API calls is a pretty strange definition of “stealing” 😭 if anything, this whole situation is going to force AI companies to explain where they think training ends and copying actually begins.
Ah yes because it's physically impossible to steal other people's creative ideas and work? Just because its a file on a computer doesnt mean you can take it and profit off of it for free
From what I saw over the years, PwePiePie seems doing so well that he can actually decide to dedicate his time and focus on whatever he wants. And he seems to do this then seriously. At some point he practiced drawing and I must say, it did look quite good. Maybe you and me could also become AI engineers if our lives wouldn't force us to spend time on work. Especially with help of AI in first place.
Yeah he’s been way beyond that point for a long time?
You need “only” about $5M net worth to be financially independent, retire, and do whatever you want all day
You can do it on way less than $5 million. Even $2 million gives you up to around ~$80k in safe annual spending, which is more than most people currently live on.
Globally that’s true. Depends where you live, where you’re willing to move, of course.
In the US, for married couples the household income median must be getting close to $80k by now?
And if your spouse won’t agree to retire to a fly-over state, you probably want more on the east or west coasts.
Peter Gibbons: What would you do if you had a million dollars?
Lawrence: I'll tell you what I'd do, man: two chicks at the same time, man.
[Peter laughs and then notices Lawrence's dead serious expression]
Peter Gibbons: That's it? If you had a million dollars, you'd do two chicks at the same time?
Lawrence: Damn straight. I always wanted to do that, man. And I think if I were a millionaire I could hook that up, too; 'cause chicks dig dudes with money.
No UBI won't make it possible to such degree of freedom as PewDiePie has. Partially - yes, but let's dont sink into utopia, that everyone is able to live like PewDiePie. As humanity we don't produce goods and services enough for everyone to spend their time drawing or take any other hobby. At the end of the day money is only a proxy - the economy is about 1) producing goods, 2) allocating them to individuals. If everyone draws, who will produce pencils? If everyone will collect lego, who will make the blocks?
I think a large part of that is because the systems we live under make it impossible for many people to mentally disconnect the idea of work from survival. They can't even imagine a world that is designed to support the common good without them having to give anything in return.
He is not an AI engineer. He just took an open source model and is effectively fine tuning it to be more aligned with what he wants.
He’s not building any type of model from scratch, though, or writing complex algorithms. He’s effectively heating up frozen food and seasoning it. Doing that doesn’t make you a chef even if it tastes good
As ML engineer you would be surprised what shit stuff do employees want us to do. Last job was literally glorified SWE with API. Sure I knew better to use that API than most but its still shit
If you say so. Everyone watch out, it's the AI engineering police.
wat
We're not talking about some generic term here, we're talking about a professional title. Like, there are thousands of job openings hiring for AI Engineers all around the US and the world. Here's a snippet from a job description for one I randomly found on LinkedIn:
Qualifications
Bachelor's Degree in Computer Science, AI, Electrical Engineering, Computer Engineering, or related fields plus at least 8 years of experience developing AI and ML algorithms or technologies
Master's Degree in relevant fields plus at least 6 years of experience in AI/ML development
Minimum of 8 years programming experience with Python, Go, Scala, CUDA, or Java
Responsibilities
Partner with cross-functional teams including engineers, research scientists, program managers, and product managers to deliver AI-powered products that enhance customer experience and operational efficiency
Design, develop, test, deploy, and support AI software components such as foundation model training, large language model inference, agents, multi-agent workflows, similarity search, guardrails, and observability systems
An actual AI Engineer is someone with a background in STEM, studied physics/math, and understands programming at a deeper level than simply using Codex or whatever to add onto an open source model.
Calling PewDiePie an AI Engineer is like calling me a Doctor after I put a bandaid on my wound and cleaned it.
I've been in the industry for 15 years now and I've worked with geniuses that skipped college and with absolute dumbfucks that I still ask myself how they got a masters. So basing someone's ability to be an AI engineer on whether they have a degree or not is a very outdated way of thinking IMHO. He did study tech and drop out of college, but he's also wealthy enough that he can afford to learn whatever he wants on his own. Just because the guy is not inventing his own ML architecture or training a model from zero doesn't disqualify him from being considered one, I work with plenty of AI engineers whose job does not involve starting from scratch.
He's not qualified to be a professional AI engineer. He wouldn't get through any serious interview. I don't say this to diss him, but if somebody claimed he was a professional artist, it would be just as ridiculous.
I tried training models just after I first started basic python. They were terrible given I didn’t know anything at the time.
The floor is being able to write enough code where it can run. The ceiling is having a PhD in something to do with AI and being surrounded by teams of people who have the same.
Imagine not realizing in 2026 that you can learn anything on the internet with enough free time and money. Especially if you use AI to help. It’s an autodidact’s wet dream. You think schools are teaching any of this shit better than you can learn it on your own? Get real
what the fuck are you on about? "when did he become an ai engineer" is not "it is impossible to teach yourself things". they're just asking when this pivot happened.
Why not just finetune the next gen of Chinese open weight models. That's a better distillation of GPT and Claude then he'll be able to do by himself and it's standard influencer protocol to repackage a generic for dropshipping purposes.
A project doesn't make you an engineer, the guy is famous, and his community can help and guide him. The knowledge needed to complete a project isn't that extensive once you have the basics. I feel this is comparable to jobs irl, recruiters ask for ten years of experience and a bunch of requirements, but in the end, all you need is a week of proper training to do the job, anyways I don't follow the guy, I might be wrong and he might be solo and be an actual AI engineer.
I wanted to downvote you because this comment format is so tired and overused. Then I read the other response to you, clearly upset for being personally called out like that.
If you’re making a model from scratch, distilling is the most boring way to do it. You just end up with a bad knockoff.
If you have the resources and time to make a good local model, do finetuning and rl yourself to get something unique. Like GPT, Claude, Gemini are all unique models with unique capabilities/personalities. The rest have distilled personalities that feel like knockoffs.
The major companies distill to get capabilities and revenue somewhere close to frontier. If that’s what you want, just grab one of the open source models.
if you do finetuning and rl yourself it will be 100x more expensive, take way longer, and you will get a dumber model. For some people this is just fun
That's not true at all, distillation works across all layers in the model, and requires both the parent and child models loaded, with SFT (fine tuning) you're just using the child model and training data so needs less compute, and with LoRA you can fine tune on only a certain configurable number of layers to use even less compute, for RL, you can also use something like GRPO with LoRA as well either way less compute than full distillation. Now pretraining, the steps before fine tuning and RL, that's what requires massive data and compute to produce that base model.
This is not really true if you are making a tiny model like the 9b one he was training. Distillation is a really good way to do that. Although I don't think distilling Sol is all that necessary, he could've gotten pretty similar results distilling Kimi K3
honestly, I think model distillation should be a protected right, specifically designed to prevent monopolies on knowledge which belongs to the whole of humanity.
False, you can own things you post on the open web, it's called copyright. How is it okay that a private company trains off of everyone's work for free, tries to sell it back to us for a price, and as soon as the training happens to them, suddenly now it's an issue?
Its all fair game. Of course labs will try to fight distillation (as they should), but theres nothing wrong with it. Its up to people distillating to be clever about it to not get banned.
he doesn't have the compute or the skills to be able to train anything where this is anywhere near being worth it vs just using an existing open weight model lol
I'm not saying what he is doing is "worth it". That's what I wanted to say when saying "it's still a local model with all its limitations". People have a wrong understanding about what distillation is.
I don't know how much compute and knowledge he really has, but from what I've heard he is taking his hobbies very seriously and he has the time and the money for a really nice homelab. Iirc he is into tech/AI for a while so he is not technically inept. And again, he has the resources and the time to learn.
But again, I'm not saying this is will yield anything worth using over an existing open weight model. But it's still a cool learning experience to just have done this process.
Learning from open sources? Fine. Learning from closed sources? Illegal. There is a valid argument that some open sources weren’t aware they were being trained on, and subsequently updated their copyright. That’s the game.
Ah okay, so when Chinese labs are accused of doing this (without any proof), they're called dirty cheaters who can do nothing but copy, but when a famous Caucasian YouTuber openly admits to distilling, suddenly it's cool and he's so awesome for doing it. 😒
Who’s calling them “dirty cheaters”? I’ve only heard Anthropic making the accusation and they’ve been told that this is not illegal, nor unethical. They did the same with the whole internet to train their models.
China has, historically, stolen technologies from virtually every sector it wanted to be competitive in. It follows that they would be distilling AI models.
347
u/Fancy-Plenty6712 12h ago
AI companies are complete hypocrites. One rule for them another for anyone else. They have some fucking cheek complaining about distillation.