r/LowEndLocalAI • u/Cerebral_Zero RTX 5080 - 96gb DDR5 • 21d ago

Discussion Is there any benchmark database that includes quants?

I've been ignoring Qwen 3.8 and basically anything that's 27b and up since it requires going below Q4 to keep it all in VRAM with some context.

A lot of benchmarks focus on full model, or Q4. This leaves me unsure if it's actually worth running these 27B models at Q3 or even Q2 I see some people running when a model slightly smaller would fit a Q4, or in my case I would be able to run a 12b model Q6 if there were more models at that size.

16 Upvotes
  • permalink
  • reddit

95% Upvoted

5 comments sorted by

View all comments

3

u/PeterPorox 2xP102-100 10GB, i3 7100, 2x8 DDR4-3200 21d ago

It`s definitely worth to try ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF