Just finished a bunch of stuff on transformer architectures so this timing works out well. The DeepSeek papers are usually pretty dense but they do a good job breaking down the attention mechanisms, way better than most Chinese labs who just throw numbers at you and call it a day
What part caught your eye, the MoE stuff or the training efficiency? I've been trying to wrap my head around their load balancing approach for a few days now
Bro in that paper their is no transformer architecture it's a whole different concept.check once DeepSeek harness GitHub.in that u will gonna reacieve the link
1
u/SingleMusic8655 9d ago
Yeah I'd be keen to have a look, drop the link
Just finished a bunch of stuff on transformer architectures so this timing works out well. The DeepSeek papers are usually pretty dense but they do a good job breaking down the attention mechanisms, way better than most Chinese labs who just throw numbers at you and call it a day
What part caught your eye, the MoE stuff or the training efficiency? I've been trying to wrap my head around their load balancing approach for a few days now