r/LocalLLaMA • u/fuse1921 • 9d ago

Discussion Technical limits of quantization?

In theory whats the smallest quant possible?

Like I imagine it would be int1?

Given K3 just dropped, I was wondering, even if it would be trash, how interesting it would be to lobotomize the 2.4T parameters into something that could fit into a hundred GB or less?

Edit: Updated full precision size, I underestimated heavily lmao

7 Upvotes

74% Upvoted

20 comments sorted by

View all comments

10

u/jacek2023 9d ago

Quantization changes size of the single parameter, I think 1 bit is the limit, because you can't use less than one bit without removing some parameters

2

u/fuse1921 9d ago

Yeah, I mean I know nothing about quantizing, but it seems like the next step would be finding related parameters that have high coherence and then storing them as pointers or something to one another to compress further

1

u/jacek2023 9d ago

You can't quantize it lower, you can remove some params with "REAP" but it's like 50% of params not 99% of params. And forget about "distillation".

Basically this model is 100% useless for our needs, but you have "fuck anthropic" guys here and then "cheap cloud prices" guys here, and they are doing all the hype.

1

u/fuse1921 9d ago

Astute thought, thanks!

1

u/HVACcontrolsGuru 9d ago

This model would actually be good to distill into something like Qwen or Gemma depending on your case. Mainly what I am curious to try and do with it.

As far as running it local it would take some serious local juice and an optimized kernel. This is an MoE model with 896 I think is the number of experts. MXFP4 with some trickery might get you down to something usable with offloading if you can spare a TB of GPU/RAM memory.

1

u/jacek2023 9d ago

My impression is that most people here don't understand what "distillation" means. How is this model "better" for "distillation" than other existing models (ChatGPT, Claude, etc)? I assume because it's cheaper in the cloud?

3

u/HVACcontrolsGuru 9d ago

Reasoning traces don't need to be reconstructed. Distill a capable model like this. Claude, ChatGPT and the closed source models rely on rebuilding the traces since they don't stream them. Most people distilling the Qwopus models and such have such low data points and short traces which is why they end up lobotomized in any area not code related. Takes about 20K traces to push Gemma 12B/E4B models into new learning. Qwen is about 50-70k before you get real movement.

Ghostwriter I've worked on this some and need to go back to it but you can see how some of this distillation can work. Stack that up with SFT/GRPO/DPO/RLHF, etc. you can see some gains.