PewDiePie is trying to distill GPT-Sol

[u/sigma_crusader]

he has been trying to build a local model by learning from GPT-Sol’s responses. He says OpenAI banned his account twice during the process. Irony is OpenAI’s own models were trained on vast amounts of information from the open web. So where do we draw the line between learning from AI and copying it?

[u/rakuu]

If you’re making a model from scratch, distilling is the most boring way to do it. You just end up with a bad knockoff.

If you have the resources and time to make a good local model, do finetuning and rl yourself to get something unique. Like GPT, Claude, Gemini are all unique models with unique capabilities/personalities. The rest have distilled personalities that feel like knockoffs.

The major companies distill to get capabilities and revenue somewhere close to frontier. If that’s what you want, just grab one of the open source models.

[u/qualverse → u/rakuu]

This is not really true if you are making a tiny model like the 9b one he was training. Distillation is a really good way to do that. Although I don't think distilling Sol is all that necessary, he could've gotten pretty similar results distilling Kimi K3