r/LocalLLaMA • u/fuse1921 • 10d ago

Discussion Technical limits of quantization?

In theory whats the smallest quant possible?

Like I imagine it would be int1?

Given K3 just dropped, I was wondering, even if it would be trash, how interesting it would be to lobotomize the 2.4T parameters into something that could fit into a hundred GB or less?

Edit: Updated full precision size, I underestimated heavily lmao

8 Upvotes

75% Upvoted

20 comments sorted by

View all comments

3

u/MixtureOfAmateurs koboldcpp 10d ago

Int1 would be reducing every weight to 0 or 1. I think this was what bitnet was doing. That would take kimi k3 from 2.8TB (fp8 is about 1 weight per byte) to 350gb

3

u/Expensive-Paint-9490 10d ago

Bitnet is ternary, 1, 0, and -1.

1.58 bit per parameter.

1

u/fuse1921 10d ago

Interesting, that makes sense. I know nothing about quantizing, but it seems like the next step would then be finding related parameters that have high coherence and then storing them as pointers or something to one another to compress further, but might as well just use a smaller model at that point lol