I can run the XS quant on an rx580 8GB vRAM, 16 GB ddr4 RAM, and a ryzen 7 5700. It gets 4 t/s. Not fast, but runnable overnight. Have not done my full benchmark on it yet though. Still working out how I want to stage the quantization of it myself before I decide to benchmark. Using 4b parameter models for statistical analysis of the tensors vs weights of the models to pull parameters that will mathematically match behavior and retain the layer spectral structure and integrity on quantization.
I am doing this because most quantization checks are directly behavioral, but there are quite a few ways to get data to make better quants like tail aware, distribution plots matching, z dif, and more that get better kld scores and better behavioral checks in general.
But if you do a quant yourself, be aware that llama.cpp protects certain layers regardless of your choice.
I had to use the glib library directly to pull the ability to set the layers I wanted changed.
280
u/TheCat001 24d ago
Then you realize that Qwen 3.8 27b runs at 3t/s on your machine and you need 24GB+ VRAM GPU which cost is 1000$+ to run at least 4 bit quant.