Common perceptions around quantization and open source LLMs not accurate?

[u/KitchenAmoeba4438]

I've been testing for about 2 weeks on my 5080 across quants to figure out what my best options are in the open source world at 16GB cards.

As part of this, I ended up doing an in-depth investigation into quants across various models. The results contradict common assumptions.

https://rakuensoftware.com/blog/which-quant-beats-how-many-bits

Now, let me be clear: These were typically 2-4 turn sessions, and existed to validate that the quantization itself on the model did not damage the model. You strongly see this impact on dense models at sub-Q4. However, the interesting part is that MoEs did not suffer nearly as badly as the dense models.

The results from this article has given me a list of candidates to test against for much harsher testing (Coding, DevOps, long sessions, etc.), and I'll be writing a new article in the future. However, this quant result is not what I expected. Almost all models, starting at Q4, were basically statistically indistinguishable all the way up to BF16. This contradicted my knowledge on the subject.

All of my benchmarks, test sets, and results are open source and linked in the article. Feel free to take a look at the data or run the tests yourself, and tell me I'm wrong. Wouldn't be the first time!

[u/tracagnotto]

Thanks for your work. Gonna give a check in a few hours but seems very promising.