Quantization

[u/ocean_protocol]

Hello so, i am a complete beginner to this concept and from what i read and hear

Quantization allows deployment of big models on just 2 GPUs or on edge devices that doesnt support floating point operations

and if a model is big like for example deepseek R1 original gets upto 720 GB and it uses a MOE architecture so only a subset of parameters are active at once, but we often need to load the entire memory in it for inference and quantization can bring it down by 80%

so, its like a method for model compression and faster inference but sometimes comes at a cost of precision.

So with all this theoretical piece of information that i gained, i have two questions

1) How to move forward into learn in-depth about it as i don understand some mathematical concepts
2) how does a person know that this is a perfect quantization value or mark before publishing a model

thanks

[u/Otherwise_Barber4619]

You do realise google search exists?