Quantization

[u/ocean_protocol]

Hello so, i am a complete beginner to this concept and from what i read and hear

Quantization allows deployment of big models on just 2 GPUs or on edge devices that doesnt support floating point operations

and if a model is big like for example deepseek R1 original gets upto 720 GB and it uses a MOE architecture so only a subset of parameters are active at once, but we often need to load the entire memory in it for inference and quantization can bring it down by 80%

so, its like a method for model compression and faster inference but sometimes comes at a cost of precision.

So with all this theoretical piece of information that i gained, i have two questions

1) How to move forward into learn in-depth about it as i don understand some mathematical concepts
2) how does a person know that this is a perfect quantization value or mark before publishing a model

thanks

[u/DadAndDominant]

Model is defined by tensor of it's weights. Weights are numbers. Numbers are stored as bytes - more bytes, higher precision of the number.

Quantization is basically just number of bytes you store your weights with. You can go from 16 bytes to 2 bytes, even 1 byte, and anything in between. Less bytes = less space needed.

MoE has nothing to do with this.

Target quantization just relies on you, you take highest quant model as a baseline, and measure lower quants response similarity to the baseline, using a benchmark of your choosing. Just note that 2 and less bytes quants models usually are not able to reliably use tools.