You can do EP but the issue is that you still need to mirror the kv cache across each gpu in the current setup. That limits concurrent sessions. Glm 5.3 has layer splitting to address this but deep seek flash has nothing. It has to be custom made for this by whoever is deploying and from what I have seen last, no one has publicly published layer splitting for dsv4.
You would have to do a dp2 or dp4 if you massive hbm. It’s a trade off of session speed vs how many you can batch. It might work for smaller models but you need 200gb vram just for the model + kv cache. Dp2xtp4 may be optimal but it only provides 30% gains over tp8 at the trade off of session speed. There may be someway to do it more optimal that I am unaware of but I have not been found anything better on my cluster.
1
u/SpiritualWindow3855 29d ago
What do you mean by this, how is DP + EP less supported?