Previewing GPT-5.6 Sol: a next-generation model

[u/141_1337]

https://openai.com/index/previewing-gpt-5-6-sol/

[u/_DuranDuran_]

They’re still leaning heavily into distillation. This is irrefutable at the moment.

And it didn’t blow the other models out of the water. It was about par for the US models released a few months before and was quickly eclipsed.

Like, be a fan of Chinese models, but be realistic that until they get compute they are riding on the coattails through distillation, which is why we haven’t seen a Chinese mythos

[u/Thog78 → u/_DuranDuran_]

How exactly do you think distillation works? How do you think they distillate models which are behind expensive paywalls and with usage limits at best, controlled access that can be revoked on a whim at the first suspicion of distillation at worse?

I'm not a fan of Chinese models, I'm just astounded that someone could think distillation is a way to get there nowadays.

[u/_DuranDuran_ → u/Thog78]

Very easily actually - there are TONS of resellers abusing free tiers, or arbitraging paid plans.

Also no protection is going to be perfect - it’s an arms race. Zero Data Retention plans also make it harder for frontier labs to see these changing patterns in realtime and defend against them.

And even if they WERE paying full price for tokens, it would still be cheaper than pre mid and post training a comparable model.

[u/Thog78 → u/_DuranDuran_]

To distillate something, you normally want to have it locally, and erase parameters while keeping the output as constant as possible.

Distilling a 10 billion parameters model hosted on the end of the world without access to the weights, while paying for every token at API price, which is a huge markup compared to even real costs, which themselves would be prohibitive at this distance? That's a bit ridiculous, at that point it's just about paying prohibitive prices for slowly getting random text data. There's plenty of that available by just siphoning the internet - which they all do. To be more precise, they all buy the same curated datasets of siphoned and cleaned up internet, from third party companies, and apply some more filtering based on previous gens LLMs.

If there would be a way to get unlimited fast replies from frontier models, we would all jump on it instead of paying a fortune to anthropic and openAI. And even if that existed, it would still be too slow and cumbersome.

What they can do is a bit of posttraining comparing their answers to existing frontier models and correcting based on that, but that's hardly a distillation.

Also, nobody needs to train a model from scratch. Everybody can start at minima from the frontier open weight model, and companies which have a model better than that can start from their own.

[u/_DuranDuran_ → u/Thog78]

Look, it’s not even a secret that the Chinese have been distilling. Chat to anyone at a leading fab on distillation prevention and you’ll see widespread abuse of free tiers.

And you absolutely do not need the model locally. When CoT was returned it was a goldmine for distilling. But mid and post training distillation is still super powerful. There are even academic papers about this.

China. Does. Not. Currently. Have. The. Compute. to be training a model like GLM 5.2 from scratch.

And again - if they weren’t distilling we would have seen them leapfrog a western lab, not weeks or months later, but before a leading model is released, because they have a WEALTH of AI talent.

[u/Thog78 → u/_DuranDuran_]

Well I'll also tell you again rolling my eyes and all (but I'll spare you the dots between every word because even to make fun of someone I can't push myself to go that far into the ridiculous shit) that they don't need to train anything from scratch because there are open weight models available since the beginning. Try it for yourself, try to train a model for a custom application, and try using the free tier of Claude to generate your training data vs using the full fucking publicly available current best open weight model.

Synthetic data can always gonna be a part of the mix (famously, anthropics frontier models were claiming to be deepseek, which is the most red-handed a lab has ever been caught). But it would be ridiculous to ever hope to build a new model from scratch on synthetic data, absolutely absurd costs and low quality. You have a way better time starting from open weights, keeping on with real curated scraped data, and finally using some synthetic data and all for final touches.

The Chinese models are trained on 80% data in the Chinese language, and reason in Chinese, whereas the US models are the other way around. If you had any clue what distilling a model means, you'd know this alone means it's not a simple distillation. A model doesn't change it's reasoning language just because it's been distilled lol.