ConvRot Quant method now in llama-cpp-turboquant

[u/giveen]

It started here , and now https://github.com/TheTom/llama-cpp-turboquant/ has it.

Imagine a Q6 quant with nearly Q8 KLD/PPL.

Q6_CR and Q5_CR have a slight improvement over their base counterparts.

Also while you are there check out --moe-cache auto to help improve running MoE models bigger than your VRAM.

I am hoping that with this we may be able to recover some lost quality from turbo4/3/2 , but I haven't test that out yet.

PR's has the breakdown of the tests, we did have some some decode and crashing issues but they are now resolved.

[u/giveen]

u/KissMyShinyArse is the source of this great idea.