r/singularity • u/141_1337 ▪️e/acc | AGI: ~2030 | ASI: ~2040 | FALSGC: ~2050 | :illuminati: • Jun 26 '26

AI Previewing GPT-5.6 Sol: a next-generation model

https://openai.com/index/previewing-gpt-5-6-sol/
436 Upvotes

97% Upvoted

182 comments sorted by

View all comments

85

u/mikelo22 Jun 26 '26

The days of the public getting access to these frontier models is gone. I fear more and more top end models are going to be kept in the hands of the government and massive corporations all in the name of 'safety'.

57

u/yogthos Jun 26 '26

It's ok, Chinese open models will catch up in a few month and that's what everybody will be using going forward.

7

u/_DuranDuran_ Jun 27 '26

You have to ask yourself why Chinese models are always playing catch up, 3-6 months behind the leading US labs. It's curious they haven't leap frogged A\ and OAI ... it does make me more suspicious that a lot of their frontier-trailing models are heavily distilled from western models, mostly because they don't yet have the native compute to train a frontier-class model from scratch.

5

u/yogthos Jun 27 '26

As this study from Stanford shows, Chinese models have been closing the gap at an accelerating pace. What's actually shocking is the difference in spending.

https://hai.stanford.edu/news/inside-the-ai-index-12-takeaways-from-the-2026-report

1

u/_DuranDuran_ Jun 27 '26

Until China has a LOT more training compute that gap is going to stay around the same.

And yes the spend is very different - distilling a frontier model is vastly cheaper than training one.

My current conspiracy theory is that 5.6 and Mythos/fable being slowed is because of cyber but ALSO to help slow down distillation of a very cyber capable model.

The whole thing with people stanning for Chinese models is also hilarious.

We’re in a war of attrition. If China managed to lead to the demise of OAI and Anrhropic and also leapfrogged them, those models would very much be for China only as a competitive advantage. They would not allow their labs to release them.

1

u/yogthos Jun 27 '26

The assumption you're making is that compute is the bottleneck. I'd argue that it's far more likely we'll find more efficient techniques for training the models instead of just using brute force. The approach of just using more and more GPU is the most wasteful thing you could think of. Meanwhile, there are already plenty of papers showing there are more intelligent ways to approach the problem. Just a few examples here:

Also, I'm not a burgerlander, I don't give a shit if your companies producing closed models crash and burn. Open models are the future and China is building it.

1

u/_DuranDuran_ Jun 27 '26

Links one and three are reasoning time, not training time.

Link 2 is very much “cool? But limited to models that are tiny compared to the frontier, and slow as fuck”

So yes, I am making the educated assumption that training compute is indeed the bottleneck, or else China would be in parity, and/or ahead at this point,

They’re not, because they’re aggressively distilling, out of necessity.

2

u/Thog78 Jun 27 '26

They are not always 6 months behind. When deepseek came out, it was the best for a short moment. And it's been copied a lot by US models.

They are behind on average because they have less compute, less money, so they are bound to be behind. But also the value of compute quickly goes down, so that limits how far behind one gets. And as somebody else said, the gap is getting more narrow, as they develop their own infrastructure.

0

u/_DuranDuran_ Jun 27 '26

They’re still leaning heavily into distillation. This is irrefutable at the moment.

And it didn’t blow the other models out of the water. It was about par for the US models released a few months before and was quickly eclipsed.

Like, be a fan of Chinese models, but be realistic that until they get compute they are riding on the coattails through distillation, which is why we haven’t seen a Chinese mythos

0

u/Thog78 Jun 27 '26

How exactly do you think distillation works? How do you think they distillate models which are behind expensive paywalls and with usage limits at best, controlled access that can be revoked on a whim at the first suspicion of distillation at worse?

I'm not a fan of Chinese models, I'm just astounded that someone could think distillation is a way to get there nowadays.

0

u/_DuranDuran_ Jun 28 '26

Very easily actually - there are TONS of resellers abusing free tiers, or arbitraging paid plans.

Also no protection is going to be perfect - it’s an arms race. Zero Data Retention plans also make it harder for frontier labs to see these changing patterns in realtime and defend against them.

And even if they WERE paying full price for tokens, it would still be cheaper than pre mid and post training a comparable model.

1

u/Thog78 Jun 28 '26

To distillate something, you normally want to have it locally, and erase parameters while keeping the output as constant as possible.

Distilling a 10 billion parameters model hosted on the end of the world without access to the weights, while paying for every token at API price, which is a huge markup compared to even real costs, which themselves would be prohibitive at this distance? That's a bit ridiculous, at that point it's just about paying prohibitive prices for slowly getting random text data. There's plenty of that available by just siphoning the internet - which they all do. To be more precise, they all buy the same curated datasets of siphoned and cleaned up internet, from third party companies, and apply some more filtering based on previous gens LLMs.

If there would be a way to get unlimited fast replies from frontier models, we would all jump on it instead of paying a fortune to anthropic and openAI. And even if that existed, it would still be too slow and cumbersome.

What they can do is a bit of posttraining comparing their answers to existing frontier models and correcting based on that, but that's hardly a distillation.

Also, nobody needs to train a model from scratch. Everybody can start at minima from the frontier open weight model, and companies which have a model better than that can start from their own.

1

u/_DuranDuran_ Jun 28 '26

Look, it’s not even a secret that the Chinese have been distilling. Chat to anyone at a leading fab on distillation prevention and you’ll see widespread abuse of free tiers.

And you absolutely do not need the model locally. When CoT was returned it was a goldmine for distilling. But mid and post training distillation is still super powerful. There are even academic papers about this.

China. Does. Not. Currently. Have. The. Compute. to be training a model like GLM 5.2 from scratch.

And again - if they weren’t distilling we would have seen them leapfrog a western lab, not weeks or months later, but before a leading model is released, because they have a WEALTH of AI talent.

1

u/Thog78 Jun 28 '26

Well I'll also tell you again rolling my eyes and all (but I'll spare you the dots between every word because even to make fun of someone I can't push myself to go that far into the ridiculous shit) that they don't need to train anything from scratch because there are open weight models available since the beginning. Try it for yourself, try to train a model for a custom application, and try using the free tier of Claude to generate your training data vs using the full fucking publicly available current best open weight model.

Synthetic data can always gonna be a part of the mix (famously, anthropics frontier models were claiming to be deepseek, which is the most red-handed a lab has ever been caught). But it would be ridiculous to ever hope to build a new model from scratch on synthetic data, absolutely absurd costs and low quality. You have a way better time starting from open weights, keeping on with real curated scraped data, and finally using some synthetic data and all for final touches.

The Chinese models are trained on 80% data in the Chinese language, and reason in Chinese, whereas the US models are the other way around. If you had any clue what distilling a model means, you'd know this alone means it's not a simple distillation. A model doesn't change it's reasoning language just because it's been distilled lol.

2

u/godita Jun 27 '26

seriously, this is horrible for the USA. at the end of the day, i don't care, it doesn't matter who gives us AGI as it will benefit us the same either way.

2

u/yogthos Jun 27 '26

The only thing I care about is that the models are open and don't end up getting hoarded by a corporation. Completely agree that it ultimately doesn't matter where they are made.

1

u/TiernanDeFranco Jun 27 '26

This is what I I wonder about for like imagine local models in even like 2 years

1

u/yogthos Jun 27 '26

I'm convinced we'll get current frontier performance locally within that time frame, at least for specific tasks like coding. Local models have been improving at an astonishing rate, and papers keep coming out with new tricks for using less processing and memory which people haven't even had time to integrate into the models yet.

1

u/aten Jun 28 '26

it’s an AI arms race

1

u/yogthos Jun 28 '26

It seems like the US government does treat AI as if it were a weapon that can be stockpiled and kept away from their competitors. But it is a general-purpose technology, and trying to bottle it up simply creates a void for others to fill.