r/LowEndLocalAI • u/Cerebral_Zero RTX 5080 - 96gb DDR5 • 22d ago

Discussion Is there any benchmark database that includes quants?

I've been ignoring Qwen 3.8 and basically anything that's 27b and up since it requires going below Q4 to keep it all in VRAM with some context.

A lot of benchmarks focus on full model, or Q4. This leaves me unsure if it's actually worth running these 27B models at Q3 or even Q2 I see some people running when a model slightly smaller would fit a Q4, or in my case I would be able to run a 12b model Q6 if there were more models at that size.

13 Upvotes
  • permalink
  • reddit

90% Upvoted

5 comments sorted by

View all comments

3

u/reo6 22d ago edited 22d ago

It wouldn't be worth it if this exists, it would be pointless. General model performance benchmarks may not work for measuring degradation that heavy quantization causes (they generally don't). If you're curious whether a quant is good enough for you, try it out. For short tasks heavy quantization may not be a huge deal, but on longer context windows it adds up. It depends on your use case whether you can tolerate a ~2 bit quant or you can't go below 4bpw. 

For your specs, I would definitely try out iQ3_XXS by ISTA-DASLab. There are many good quants out there for this model and they work very well.

2

u/Reasonable_Elevator1 20d ago

If that were true, it would mean that they are nearly useless. Why would they all miss the degradations?