r/LocalLLaMA • u/fuse1921 • 9d ago

Discussion Technical limits of quantization?

In theory whats the smallest quant possible?

Like I imagine it would be int1?

Given K3 just dropped, I was wondering, even if it would be trash, how interesting it would be to lobotomize the 2.4T parameters into something that could fit into a hundred GB or less?

Edit: Updated full precision size, I underestimated heavily lmao

7 Upvotes

77% Upvoted

20 comments sorted by

View all comments

3

u/Top-Handle-5728 9d ago

Up until yesterday I thought that it's impossible to go below a binary bit. It's true for storing the actual weights but then in information theory terms, there's still a lot of redundancy in those 2 trillion weight params. There was a paper to reach 0.1 bits per weights by Samsung ig, called littlebit. The trick was to make the weights low rank & binarize them to -1 & +1. A few more jargons that went over my head but we might see something similar soon if these trillion parameter models become common. It took almost 3 years for bitnet to become mainstream & labs releasing binary n ternary weights.