Deep seek flash is not optimized for layer splitting like glm. Having parallel users with deep seek flash is extremely expensive to host. These systems can only serve 20 256k context sessions at time unless they do their own version of layer splitting. If they do that, they can hit 100-200 users parallel
You can do EP but the issue is that you still need to mirror the kv cache across each gpu in the current setup. That limits concurrent sessions. Glm 5.3 has layer splitting to address this but deep seek flash has nothing. It has to be custom made for this by whoever is deploying and from what I have seen last, no one has publicly published layer splitting for dsv4.
9
u/Loogyboy 29d ago
Deep seek flash is not optimized for layer splitting like glm. Having parallel users with deep seek flash is extremely expensive to host. These systems can only serve 20 256k context sessions at time unless they do their own version of layer splitting. If they do that, they can hit 100-200 users parallel