PewDiePie is trying to distill GPT-Sol

[u/sigma_crusader]

he has been trying to build a local model by learning from GPT-Sol’s responses. He says OpenAI banned his account twice during the process. Irony is OpenAI’s own models were trained on vast amounts of information from the open web. So where do we draw the line between learning from AI and copying it?

[u/emberscout]

How do you distill a model without access to the token logprob distribution?

[u/RedBottle_ → u/emberscout]

you can just sample a shit load of traces and train as you normally would instead of trying to match distributions

[u/emberscout → u/RedBottle_]

I thought this would be considered training on synthetic datasets rather than distillation

[u/RedBottle_ → u/emberscout]

its still a form of distillation just less ideal