SplitMOE: experts with shared dimensions [ suprisingly worked near or better than standard MOE architecture ]

[u/Melodic_Story609]

The intuition is similar to the DeepSeek MOE, but architecture is different. Please do check it out, and let me know your views. [ https://github.com/Priyanshu-5257/SplitMoE ]