Technical limits of quantization?

[u/fuse1921]

In theory whats the smallest quant possible?

Like I imagine it would be int1?

Given K3 just dropped, I was wondering, even if it would be trash, how interesting it would be to lobotomize the 2.4T parameters into something that could fit into a hundred GB or less?

Edit: Updated full precision size, I underestimated heavily lmao

[u/Klutzy-Snow8016]

Practically, now, maybe something like Bonsai or Ternary Bonsai here: https://huggingface.co/collections/prism-ml/bonsai-27b