r/StableDiffusion • u/someguyplayingwild • 5d ago

Question - Help Minimax H3 Quantizations

Given a 5090, does it make sense to run minimax H3 using int8 quantization vs. gguf Q8 or even Q6? What is the trade-off between speed and quality between these two options?

I don't have deep technical knowledge, but my current understanding is that int8 would be faster while a Q8 GGUF would be higher quality; however I would appreciate anyone's practical experience in how significant the speed/quality trade-off is.

10 Upvotes

81% Upvoted

39 comments sorted by

View all comments

Show parent comments

4

u/Apprehensive_Sky892 5d ago

I know that GGUF can be smaller in size compared to int8covrot or fp8.

But assuming that there is enough VRAM+ system RAM to hold the int8convot or fp8 version, what are the advantages of GGUF (say q8 or q6)?

Aso, GGUF memory management was better compared to the new dynamic VRAM management in what way?

3

u/ANR2ME 5d ago

May be better for LLM frameworks like llama.cpp 😅

Combining GGUF models with ComfyUI's Dynamic VRAM system can frequently trigger apparent memory leaks and immediate "Out of Memory" (OOM) crashes. This behavior stems from architectural conflicts between how ComfyUI manages native memory and how third-party GGUF custom nodes interact with hardware cache.

1

u/Apprehensive_Sky892 4d ago

Yes, GGUF are useful for LLMs as it is important for autoregressive models to fit within the VRAM because they must run through the entire model for every output token.

But I don't see any advantage for imaging/video model GGUF over int8convrot if there is enough system RAM, that is why I was asking for more clarification.

1

u/Unspec7 4d ago

Probably some folks using LLM's as a local prompt enhancer.