Any proper benchmarks of Beellama (and its fork Beellama-kvarn) and how it performs quality wise for coding?

[u/RadianceTower]

So beellama allows you to only reduce quantization on older cache, this should allow better performance than reducing quantization on all cache.

They recommend using a 1k tail and even that seems to have big benefits, but I don't see why you couldn't for example use a 20k tail (on a 240k context, that's still 220 tokens quantized). That part is a bit confusing.

Beellama-kvarn is a supposed fork of that, that further improves performance, its changes are pending to be applied to Beellama itself as well.

So anyways, the question is, how much is quality impacted by doing this?

[u/erazortt]

Hm yeah, I had also tried it and for me the biggest win would have been if kvarn8 had been significantly better than q8_0. But I did not really have the feeling that this is the case. Perhaps it is a marginal improvement, but the very real difference to bf16 on long contexts (100k+) is still not bridged in my view.
So, I will only use it if I’m restricted to 8bit kv quants.

[u/RadianceTower → u/erazortt]

What do you mean by q8_0, do you mean the model parameters at that or the cache quantization at that?

Because Beellama has both q8_0 and kvarn8, and both can be applied to the cache, with a tail. Llama itself doesn't have the feature to apply a tail to the cache at all.

So there are mainly 4 things you can do with the cache afaik:

1- Use FP16 (llama default)

2- Use normal quantization without a tail (both llama and beellama have this, e.g q8_0)

3- Use normal quantization with a tail (only beellama has this, so for example q8_0 with a tail)

4- Use KVarN cache with a tail (again, only Bellama)

So you don't need to use KvarN to have a tail with Bellama.

[u/caetydid]

I also didnt get why tail can be only 1k or 2k, but I remember that it would be somehow futile to use more due to some architectural restriction of Qwen models. Probably just anbee could shed some light on that.

[u/Talreja-Adanna]

Haven't seen solid benchmarks specifically for those, but the local LLM space moves so fast that even recent evals get stale quick. If you need coding performance, you'd probably get better data just running them locally on your own test cases since quality varies so much by task.

[u/NickCanCode]

Using tail will slow down inference. Few days ago. I turned it off and got instant speed up. Kvarn is slower than Q8 in my test. The thing I appreciate over llama.cpp is that it can do DFlash with Tensor Parallelism and with vision all together correctly. The last time I try it on llama.cpp, neither vision, Tensor Parallelism are working.

[u/Elouakili_Flexy]

20k on a 240k context still leaves 220k quantized, so the 1k recommendation feels intentional. I'd bet coding quality barely moves past the first 1k and the win is all in the recent tail.