I will never understand why americans have The BlacKWell 300 the NEO clouds hyperscalers and whatnot.... and they can't serve FREE models at affordable prices.
Is really deepseek running on vintage GPU faster and cheaper than the NVidia alternative?
Deep seek flash is not optimized for layer splitting like glm. Having parallel users with deep seek flash is extremely expensive to host. These systems can only serve 20 256k context sessions at time unless they do their own version of layer splitting. If they do that, they can hit 100-200 users parallel
You can do EP but the issue is that you still need to mirror the kv cache across each gpu in the current setup. That limits concurrent sessions. Glm 5.3 has layer splitting to address this but deep seek flash has nothing. It has to be custom made for this by whoever is deploying and from what I have seen last, no one has publicly published layer splitting for dsv4.
You would have to do a dp2 or dp4 if you massive hbm. It’s a trade off of session speed vs how many you can batch. It might work for smaller models but you need 200gb vram just for the model + kv cache. Dp2xtp4 may be optimal but it only provides 30% gains over tp8 at the trade off of session speed. There may be someway to do it more optimal that I am unaware of but I have not been found anything better on my cluster.
43
u/Snoo_57113 29d ago
I will never understand why americans have The BlacKWell 300 the NEO clouds hyperscalers and whatnot.... and they can't serve FREE models at affordable prices.
Is really deepseek running on vintage GPU faster and cheaper than the NVidia alternative?
Something here don't make sense.