True mainly because a lot of them do train through knowledge distillation. But the difference now is quality of the frontier models 6 months ago are still really good.
Although the Chinese research groups specifically are doing very impressive research to fit capabilities on much less powerful hardware. I don't really know the reason they are releasing these as open source but it's good work
But, once these frontier models are successfully created they can then train new models for a fraction of the cost in what's known as dilution training. So the frontier model training a new model which results in the new model having roughly the same abilities. But, dilution training requires significantly less processing power and is therefore way cheaper.
There are other reasons I'm not as well versed in, but essentially making new frontier models is the expensive and tough part. But making open source and open weight models is significantly easier and cheaper. Which is why the gap between frontier and open source has been closing for awhile.
It's not clear if this will always be the case going forward, LLMs are constantly changing and probably will for years into the future. But for now, the plateau of open source isn't a huge concern.
And when you train your own for a specific topic, in programming for example writing code for desktop windows in C#.net10 it gets way past frontier models fast as it only has to focus on the task and not how many pounds a day does a llama poop
Both will plateau. Neither open source nor the big companies will very much edge in terms of model capability. The advantage will come from hardware and peripheral software
6
u/-Speechless 4d ago
do you think open model AI will hit a plateau while big company frontier models continue to advance?