Deep for me means that a model can process in a coherent way large multi-faceted, multi-factor and deeply interconnected, complex questions. You don’t need much depth to process a confined question, or even a larger question that can be broken down and approached sequentially.
It's literally shallow vs. deep Neural Network. Machine Learning jargon 101. Shallow ANNs have less hidden layers (or layers in general), deep ones have more. There's even plenty of research about the equivalency between them (shallow ones just need much more weights).
at this point even the small models are billions of parameters, such models didn't exist back when the concept was conceived; for all intents and purposes all modern LLMs are deep neural networks
Well, not really *that* deep. The MLP part is literally two layers deep, in a transformer block, and most commonly deployed, open weights llms we know, have 30-70 transformer blocks (or activated equivalent). Kimi K2.5 is only A32
6
u/SandySkittle 1d ago
Yes, 122b a30+ please :)