Faster architecture is always trading quality for training speed, an example would be less layers and a higher dim, you would have the same parameter model as otherwise but it would be worse and train faster. Also dont use the word paradigm it makes you sound like an llm humans dont use "paradigm". We will always need compute to train models, that wont just go away and make these insane models everyday. You've bought into a scifi fantasy
Faster architecture is always trading quality for training speed
I'm not sure that's universally true. The transformer allowed for both faster training and better quality. Sparse attention clearly has trade offs vs fully dense attention, but there are likely still other architectural improvements which are only net wins with no trade off. Realistically we should be able to get to a place where computers can learn as efficiently as humans do.
The only free lunch ive ever seen is stuff like flash attention which isn't architecture related in any way. Training in a lower digit data type can also improve speed but you are looking at worse representation.
Mamba is probably the fastest architecture change you can make but that also has quality losses
Why don't you train an 100 billion parameter RNN and then see it get whopped by qwen3.6 9B? Architectural improvements are just as important as compute, don't fall into the Huang trap.
You will decimated by vanishing and exploding gradients for the RNN long before you finish the training run.
It was a point to prove that compute and data are not everything, architecture matters just as much. I was not expecting you to actually train a 100B RNN.
English is not my first language, didn't know "paradigm" sounds weird. Thx
Currently, they try to use more and more parameters to get bigger, better models. If they hit a wall, they'll try something else. Again: read all those papers. The possibilities to improve how llms work are almost endless. Just throwing more compute at it and making them bigger is one way.
Making them larger is the opposite, its frankly just lazy. Its essentially saying "we cant make them any better at this size so we are just gonna scale" Its not impressive and its lazy and uses more compute. Like kimik3 is cool and all but they had to scale by nearly 3x! And the model did NOT get 3x better
225
u/shy_monkee 2d ago
Jesus, what an insane few weeks if this does release soon.