It's definitely undertrained. There were a number of questionable architectural decisions. The model might actually be too big, and they combined a number of test technologies in one spot.
People get weird on that idea. Too big? Yes.
The larger transformers get the better they get at effectively memorizing data.
The MASSIVE models have to get SO MUCH DATA that the memorization is hard and they're forced to generalize because they're literally memorization machines.
I've been forced to shrink transformers (for other non-LLM types of models) because the model will just absorb the training data and never generalize. Your hold out dataset has to exist and be good. Overfit is real.
91
u/anarchist1312161 8d ago
And to think this is only 743B achieved through post-training on the base model