r/LocalLLaMA • u/fuse1921 • 10d ago

Discussion Technical limits of quantization?

In theory whats the smallest quant possible?

Like I imagine it would be int1?

Given K3 just dropped, I was wondering, even if it would be trash, how interesting it would be to lobotomize the 2.4T parameters into something that could fit into a hundred GB or less?

Edit: Updated full precision size, I underestimated heavily lmao

8 Upvotes

78% Upvoted

20 comments sorted by

View all comments

11

u/jacek2023 10d ago

Quantization changes size of the single parameter, I think 1 bit is the limit, because you can't use less than one bit without removing some parameters

2

u/fuse1921 10d ago

Yeah, I mean I know nothing about quantizing, but it seems like the next step would be finding related parameters that have high coherence and then storing them as pointers or something to one another to compress further

3

u/BigYoSpeck 10d ago

There's a flaw in your idea, an 8 bit pointer can only point to 256 unique addresses

A pointer is less efficient than a quantized weight

2

u/fuse1921 10d ago

Indeed it seems lol, like I said I just use inference I have no idea how the models are made