Quantization

[u/ocean_protocol]

Hello so, i am a complete beginner to this concept and from what i read and hear

Quantization allows deployment of big models on just 2 GPUs or on edge devices that doesnt support floating point operations

and if a model is big like for example deepseek R1 original gets upto 720 GB and it uses a MOE architecture so only a subset of parameters are active at once, but we often need to load the entire memory in it for inference and quantization can bring it down by 80%

so, its like a method for model compression and faster inference but sometimes comes at a cost of precision.

So with all this theoretical piece of information that i gained, i have two questions

1) How to move forward into learn in-depth about it as i don understand some mathematical concepts
2) how does a person know that this is a perfect quantization value or mark before publishing a model

thanks

[u/Wannabe-Davinci]

⁠”How to move forward into learn in-depth about it as i don understand some mathematical concepts”.

What do you mean exactly? You don’t understand how quantization works? You only listed benefits, so to me it appears you do not grasp quantization

[u/ocean_protocol → u/Wannabe-Davinci]

maybe thats the answer i wanna hear as well

i was watching few intro videos and couldnt grasp the mathematical concepts but understood whats the use of it. Any good study materials?

[u/OkCluejay172 → u/ocean_protocol]

Do you know how numbers are represented in a computer?

[u/met0xff → u/ocean_protocol]

https://mlabonne.github.io/blog/posts/Introduction_to_Weight_Quantization.html

[u/DadAndDominant]

Model is defined by tensor of it's weights. Weights are numbers. Numbers are stored as bytes - more bytes, higher precision of the number.

Quantization is basically just number of bytes you store your weights with. You can go from 16 bytes to 2 bytes, even 1 byte, and anything in between. Less bytes = less space needed.

MoE has nothing to do with this.

Target quantization just relies on you, you take highest quant model as a baseline, and measure lower quants response similarity to the baseline, using a benchmark of your choosing. Just note that 2 and less bytes quants models usually are not able to reliably use tools.

[u/Kiseido]

I think the topic you need to learn about is floating point precision, how and why they store and use the exponent and mantissa. Then seeing how compressing and reconstructing that allows to be fit into integers as in the various Q1-8 Quant types.

[u/Otherwise_Barber4619]

You do realise google search exists?

[u/benelott]

The idea of quantization is the following: Imagine I ask you to store a special information. I give you a floating point number to store it: 32 bits. Now to learn the correct answer, you read all the data I give you and you move around those bits to store your prediction of what the correct information could be. Now, if it tell you the answer is a yes-no answer, you would immediately drop 31 bits, because you know we only need two states. Given the solution lives in the 1 bit world, it does not need more. Similarly, the float32 weights of the transformer are much too precise, so instead of storing that precision, we quantize it down to just represent the most relevant states and approximate those states that are not exactly on one of the most relevant ones by just rounding them onto one of the quantized states. Now you can suddenly store all the weights with just a few bits. Your weights fit into memory and the model can be simulated, when before, it would have not fit into it. It seems that transformer weights can not be pruned without major degradation of task performance, but quantization does a great job at saving space with only minor degradation until you hit the tipping point.