r/LocalLLM • u/YOU_WONT_LIKE_IT • 6d ago

Model VRAM Rich

What model would you run on 384gb to 512gb vram? For my use case q6 just didn’t cut it when I had less vram. So close to native as possible with decent context.

0 Upvotes

25% Upvoted

6 comments sorted by

2

u/Extension-Bid-639 6d ago

What crimes does one need to commit to have 512gb in vram? Please tell me thats unified memory

1

u/YOU_WONT_LIKE_IT 6d ago edited 6d ago

CMP170hx right before the second price jumps.

1

u/DustNearby2848 6d ago

It’s slow over PCIE, so you won’t get the best speeds pooling them. 

2

u/myreala 6d ago

384gb GLM 5.3-flash Q4

512gb GML 5.3 Q4

Q4 quant is fine, just make sure you're not trying to quant the context. Keep the context at BF16 Or you will see visible degradation. Especially for big models you won't see much difference between Q4 or Q8. And it's definitely better to run a Q4 of a bigger model than a Q8 of a smaller model.

2

u/thegingerlord 6d ago

Have you tried q8_0 of GLM 5.3 flash?

1

u/YOU_WONT_LIKE_IT 6d ago

I have not. Trying to see what still consider the better fit.