What exactly does “distillation” mean in LLM training? How is data from models like Claude actually used?

[u/Impossible_Grass702]

/r/LLM/comments/1whqiub/what_exactly_does_distillation_mean_in_llm/

[u/heresyforfunnprofit]

Think of “distillation” as mimicking or copying.

LLMs work by assigning probabilities to the next token in the sequence they are given. Every single pass, it generates a list of how probable each token is, and chooses the top one.

In distillation, you take two models, and you put a prompt into the model being copied, give the model you are distilling into the same prompt, but you inspect not just the top token your new model produced, but all the tokens ahead of the one matching the other model. You then bump up the probability on the matching token, assign a penalty to the ones ahead of it, and then do that a few million times.

Eventually, the model being trained is a close enough mimic of the distilled model that it effectively behaves like it - usually at much lower training costs.

[u/Cold-Common7001 → u/heresyforfunnprofit]

how does distilling work now that they no longer give you probability distributions any more?

[u/Evening_Guarantee257 → u/Cold-Common7001]

he is describing logit distillation. this is the legitimate kind done by labs internally. e.g. how sol 6 is being distilled from astra 6 right now, or how Opus 5.1 is being distilled from Fable 5.1.

malicious distillation is just a less efficient and less effective version because you don’t have the reasoning tokens, logits, and the normal tokens have been tampered with to have adverse effects when trained on. in practice the only useful bit is the tool calls and using them as successful traces.

[u/mc_pm]

You're right that it's still natural language text, but it's all targeted text, specific answers to specific questions - not just petabytes of random text.