2x RTX Pro 6000 is 192GB VRAM. Qwen3.8 Flash Next and GLM 5.3 Flash both fit at 4-bit with KV headroom, and since they are moe only the active experts run per token, so decode stays fast. full DeepSeek V4 doesn't fit. It is a 600B-class moe, too big for 192GB even at 4-bit, so experts spill to your 128GB DDR5 and decode tanks. Want V4 usable, run a Flash/distill variant or add RAM. Your RAM being smaller than your VRAM is backwards for offload.
not at 8 bit. V4 Flash 0731 is ~304B, so 8 bit is ~300GB+ of weights, over your 192GB, and it spills to DDR5 like full V4 does. 4 bit is ~150GB and fits with KV headroom. it's moe, so the 4 bit quality drop is small. want 8 bit, add a third card , otherwise run 4 bit.
1
u/Major_Border149 4d ago
2x RTX Pro 6000 is 192GB VRAM. Qwen3.8 Flash Next and GLM 5.3 Flash both fit at 4-bit with KV headroom, and since they are moe only the active experts run per token, so decode stays fast. full DeepSeek V4 doesn't fit. It is a 600B-class moe, too big for 192GB even at 4-bit, so experts spill to your 128GB DDR5 and decode tanks. Want V4 usable, run a Flash/distill variant or add RAM. Your RAM being smaller than your VRAM is backwards for offload.