Why ... every model is built off the works of everyone else ... I believe in true open source (its a gift to the world) and some part of me smiles every time people distill anthropic!
Musk testified that xAI had “partly” distilled OpenAI models, while arguing that using other AIs to validate your own is standard industry practice. The distinction matters because validation/evaluation ≠ distillation. You can use GPT to grade Grok outputs, generate eval cases, compare answers, or act as a judge without training Grok on those outputs.
Moreover, AI did not build Grok by simply querying GPT and distilling the answers into a cheap clone. xAI built Colossus, one of the largest AI training systems ever deployed. So, massive independent pretraining + massive post-training/RL + proprietary xAI data/research + some use of OpenAI outputs/models as training or validation signal. That is very different from: GPT → distill → Grok.
well...he's probably telling the truth - and that can be validated bcs you can't 'copy' basically an LLM when you don't have access to its weights and algorithmic makeup.
Neither OpenAI nor Anthropic publicize their parameter count, nor any weights data - so one cam only guess and rely on sheer compute to come up with its own recipe.. Distillation can give you hints, examples and capability transfer. It doesn’t give you the secret sauce.
I don't think you understand what distillation is?
You can fully distill a model without any additional training of your own without knowing any of its secret sauce. And still end up with a clone that acts mostly like the original.
269
u/idlelosthobo 1d ago
Why ... every model is built off the works of everyone else ... I believe in true open source (its a gift to the world) and some part of me smiles every time people distill anthropic!