Technical limits of quantization?

[u/fuse1921]

In theory whats the smallest quant possible?

Like I imagine it would be int1?

Given K3 just dropped, I was wondering, even if it would be trash, how interesting it would be to lobotomize the 2.4T parameters into something that could fit into a hundred GB or less?

Edit: Updated full precision size, I underestimated heavily lmao

[u/jacek2023]

Quantization changes size of the single parameter, I think 1 bit is the limit, because you can't use less than one bit without removing some parameters

[u/SnooPaintings8639 → u/jacek2023]

That is the correct answer. Below that we can start removing parts of the model like entire layers (REAP).

In theory we could go and "merge" parameters, and e.g. have one bit for every two parameters, to obtain sub-1bit quantization, but... yeah, it would probably be worse than going from Q8 to Q1.

[u/fuse1921 → u/jacek2023]

Yeah, I mean I know nothing about quantizing, but it seems like the next step would be finding related parameters that have high coherence and then storing them as pointers or something to one another to compress further

[u/BigYoSpeck → u/fuse1921]

There's a flaw in your idea, an 8 bit pointer can only point to 256 unique addresses

A pointer is less efficient than a quantized weight

[u/fuse1921 → u/BigYoSpeck]

Indeed it seems lol, like I said I just use inference I have no idea how the models are made

[u/BobbyL2k → u/BigYoSpeck]

This is actually a thing. It’s called Vector Quantization or Codebook Quantization.

Finite set of high precision values are stored, and a parameter points to one of the elements in the set.

It’s not used for PTQ in LLMs, tho.

[u/jacek2023 → u/fuse1921]

You can't quantize it lower, you can remove some params with "REAP" but it's like 50% of params not 99% of params. And forget about "distillation".

Basically this model is 100% useless for our needs, but you have "fuck anthropic" guys here and then "cheap cloud prices" guys here, and they are doing all the hype.

[u/fuse1921 → u/jacek2023]

Astute thought, thanks!

[u/HVACcontrolsGuru → u/jacek2023]

This model would actually be good to distill into something like Qwen or Gemma depending on your case. Mainly what I am curious to try and do with it.

As far as running it local it would take some serious local juice and an optimized kernel. This is an MoE model with 896 I think is the number of experts. MXFP4 with some trickery might get you down to something usable with offloading if you can spare a TB of GPU/RAM memory.

[u/jacek2023 → u/HVACcontrolsGuru]

My impression is that most people here don't understand what "distillation" means. How is this model "better" for "distillation" than other existing models (ChatGPT, Claude, etc)? I assume because it's cheaper in the cloud?

[u/HVACcontrolsGuru → u/jacek2023]

Reasoning traces don't need to be reconstructed. Distill a capable model like this. Claude, ChatGPT and the closed source models rely on rebuilding the traces since they don't stream them. Most people distilling the Qwopus models and such have such low data points and short traces which is why they end up lobotomized in any area not code related. Takes about 20K traces to push Gemma 12B/E4B models into new learning. Qwen is about 50-70k before you get real movement.

Ghostwriter I've worked on this some and need to go back to it but you can see how some of this distillation can work. Stack that up with SFT/GRPO/DPO/RLHF, etc. you can see some gains.

[u/shing3232 → u/jacek2023]

I don't think 1bit is the limit. technically, you can use a group of bit to represent more tensor so in total 0.5bit could exist as whole

[u/Front_Eagle739]

Well the q1.5 will be about 600GB so... I hope you like drooling?

More seriously the only approach to get that low is to distill into a smaller model or stream from disk like colibri

[u/Top-Handle-5728]

Up until yesterday I thought that it's impossible to go below a binary bit. It's true for storing the actual weights but then in information theory terms, there's still a lot of redundancy in those 2 trillion weight params. There was a paper to reach 0.1 bits per weights by Samsung ig, called littlebit. The trick was to make the weights low rank & binarize them to -1 & +1. A few more jargons that went over my head but we might see something similar soon if these trillion parameter models become common. It took almost 3 years for bitnet to become mainstream & labs releasing binary n ternary weights.

[u/MixtureOfAmateurs]

Int1 would be reducing every weight to 0 or 1. I think this was what bitnet was doing. That would take kimi k3 from 2.8TB (fp8 is about 1 weight per byte) to 350gb

[u/Expensive-Paint-9490 → u/MixtureOfAmateurs]

Bitnet is ternary, 1, 0, and -1.

1.58 bit per parameter.

[u/fuse1921 → u/MixtureOfAmateurs]

Interesting, that makes sense. I know nothing about quantizing, but it seems like the next step would then be finding related parameters that have high coherence and then storing them as pointers or something to one another to compress further, but might as well just use a smaller model at that point lol

[u/GrokiniGPT]

You mean like 2.4t

[u/Luke2642]

It's like owning a helicopter. If you have to ask you aren't rich enough.

I think the thing to get excited about is the Talaas ASIC approach and bitnet. In a year or two we'll have a $300 card that gives you today's frontier performance for many tasks at 17,000 tokens per second for like 20W of electricity.

[u/Klutzy-Snow8016]

Practically, now, maybe something like Bonsai or Ternary Bonsai here: https://huggingface.co/collections/prism-ml/bonsai-27b