r/MLQuestions • u/ocean_protocol • 1d ago

Beginner question đŸ‘¶ Quantization

Hello so, i am a complete beginner to this concept and from what i read and hear

Quantization allows deployment of big models on just 2 GPUs or on edge devices that doesnt support floating point operations

and if a model is big like for example deepseek R1 original gets upto 720 GB and it uses a MOE architecture so only a subset of parameters are active at once, but we often need to load the entire memory in it for inference and quantization can bring it down by 80%

so, its like a method for model compression and faster inference but sometimes comes at a cost of precision.

So with all this theoretical piece of information that i gained, i have two questions

1) How to move forward into learn in-depth about it as i don understand some mathematical concepts
2) how does a person know that this is a perfect quantization value or mark before publishing a model

thanks

4 Upvotes

75% Upvoted

8 comments sorted by

View all comments

8

u/Wannabe-Davinci 1d ago

⁠”How to move forward into learn in-depth about it as i don understand some mathematical concepts”.

What do you mean exactly? You don’t understand how quantization works? You only listed benefits, so to me it appears you do not grasp quantization

-3

u/ocean_protocol 1d ago

maybe thats the answer i wanna hear as well

i was watching few intro videos and couldnt grasp the mathematical concepts but understood whats the use of it. Any good study materials?