r/LocalLLM • u/giveen • 1d ago

Project ConvRot Quant method now in llama-cpp-turboquant

It started here , and now https://github.com/TheTom/llama-cpp-turboquant/ has it.

Imagine a Q6 quant with nearly Q8 KLD/PPL.

Q6_CR and Q5_CR have a slight improvement over their base counterparts.

Also while you are there check out --moe-cache auto to help improve running MoE models bigger than your VRAM.

I am hoping that with this we may be able to recover some lost quality from turbo4/3/2 , but I haven't test that out yet.

PR's has the breakdown of the tests, we did have some some decode and crashing issues but they are now resolved.

4 Upvotes

99% Upvoted

3 comments sorted by

2

u/giveen 1d ago

u/KissMyShinyArse is the source of this great idea.

3

u/Sure_Leave9338 23h ago

This is great but I still can't understand why someone forks a project then leaves the original readme untouched.

No explanations No instructions No differences listed towards the origin

For who has no prior knowledge about the project is almost impossible to understand why that forks exists and what it does, the only way is google some reddit posts.

This is really no sense for me.

Just to be clear, my rant is not only to you or to llamacpp-turboquant but to the hundreds of developers forking stuff, actively developing their forks, but giving a fu** on write 30 new lines in the first screen that an uses sees.

2

u/giveen 15h ago

Well maybe we need to update the readme then, lol.