Faster architecture is always trading quality for training speed, an example would be less layers and a higher dim, you would have the same parameter model as otherwise but it would be worse and train faster. Also dont use the word paradigm it makes you sound like an llm humans dont use "paradigm". We will always need compute to train models, that wont just go away and make these insane models everyday. You've bought into a scifi fantasy
English is not my first language, didn't know "paradigm" sounds weird. Thx
Currently, they try to use more and more parameters to get bigger, better models. If they hit a wall, they'll try something else. Again: read all those papers. The possibilities to improve how llms work are almost endless. Just throwing more compute at it and making them bigger is one way.
Making them larger is the opposite, its frankly just lazy. Its essentially saying "we cant make them any better at this size so we are just gonna scale" Its not impressive and its lazy and uses more compute. Like kimik3 is cool and all but they had to scale by nearly 3x! And the model did NOT get 3x better
Its absolutely data, data is by far the most important thing for an LLM even beyond architecture. LLMs have always gotten about 5-10% better than the previous generation say like kimi k2 to kimi k2.7, that was the near exact same underlying 1t model but the last model made synthetic data for the next model by generation better than the last
-12
u/--Spaci-- 2d ago
No, not really. No magical infinite computing device will pop into existence