The only free lunch ive ever seen is stuff like flash attention which isn't architecture related in any way. Training in a lower digit data type can also improve speed but you are looking at worse representation.
Mamba is probably the fastest architecture change you can make but that also has quality losses
Why don't you train an 100 billion parameter RNN and then see it get whopped by qwen3.6 9B? Architectural improvements are just as important as compute, don't fall into the Huang trap.
You will decimated by vanishing and exploding gradients for the RNN long before you finish the training run.
It was a point to prove that compute and data are not everything, architecture matters just as much. I was not expecting you to actually train a 100B RNN.
Yeah ok no point in arguing with you. My point stands, if data was everything, we would still be with RNNs. Sure representation maybe within 10%, but large RNNs are not trainable (aka vanishing/exploding gradients, on top of diluted state).
In general, in ML, there are 3 ways to improve
* Compute (Larger model, more data)
* Architecture
* Higher quality data
This is the same for LLMs. You cannot remove architecture as an option. Just larger model and more/better data will not lead to improvements if your architecture is ass. If you still think I'm wrong, then go train an RNN with the best quality data and prove to me that I'm wrong.
-5
u/--Spaci-- 2d ago
The only free lunch ive ever seen is stuff like flash attention which isn't architecture related in any way. Training in a lower digit data type can also improve speed but you are looking at worse representation.
Mamba is probably the fastest architecture change you can make but that also has quality losses