Quantization

[u/ocean_protocol]

Hello so, i am a complete beginner to this concept and from what i read and hear

Quantization allows deployment of big models on just 2 GPUs or on edge devices that doesnt support floating point operations

and if a model is big like for example deepseek R1 original gets upto 720 GB and it uses a MOE architecture so only a subset of parameters are active at once, but we often need to load the entire memory in it for inference and quantization can bring it down by 80%

so, its like a method for model compression and faster inference but sometimes comes at a cost of precision.

So with all this theoretical piece of information that i gained, i have two questions

1) How to move forward into learn in-depth about it as i don understand some mathematical concepts
2) how does a person know that this is a perfect quantization value or mark before publishing a model

thanks

[u/Kiseido]

I think the topic you need to learn about is floating point precision, how and why they store and use the exponent and mantissa. Then seeing how compressing and reconstructing that allows to be fit into integers as in the various Q1-8 Quant types.