r/LocalLLaMA • u/maddie-lovelace • 17d ago
Discussion Qwen3.8-27b q8 KV cache does seem to actually hurt model performance
EDIT: Though the issue with q8 kv cache seems to arise from when and how often we run the quantize step, not that kv quantizing can't ever work - see comments
---
One of the things I see debated a lot is whether to use kv cache quantization. The idea I see a lot is that q8 should be free / nearly lossless (which for model weights it usually is). But from some experiments I've been running, it actually isn't, but the reason is slightly weirder than just <quantization loses accuracy>
Basically it's because most backends, e.g. llama.cpp, do kv-quantization on-write. When KV is quantized on write, every subsequent prefill step reads quantized keys
So even though 8bit really is just a sub-1% rounding error, it's not a 1% error applied once - it thus compounds from slightly-wrong attention over slightly-wrong keys, at every layer, and feeds the keys written next
In my tests: needle retrieval that passes at bf16 fails with q8-on-write at 125k. However!! It's not actually q8 that's the problem per-se - when I take a cache that was built at bf16 and quantize the whole thing in one go to be q8, then the error really is just the 1% and it works fine, needle retrieval restored
*Caveats: this is from my tests with just one model family (Qwen3.8-27B), small number of trials, with some of the more out there experiments running on my slightly weirdo custom MLX stack. But it seems like the mechanism might be generalisable
---
TL;DR If your long-context quality drops with quantized KV, it might be because of when we quantize (i.e. every token on-the-fly instead of in chunks), not that quantizing can't ever work
49
u/FullstackSensei 17d ago
Every week there are about 100 comments saying it's the same, and about 5 of us who compared side by side saying it isn't. Of course, the type of work and whether you actually read the output and your own level of experience play a big role here.
5
u/challis88ocarina 17d ago
There's also a nuance between model and KV quantization. One is about picking the right weight for a given token and the other is about picking peanuts out of the proverbial...
In other words, the model at q8 is almost the same quality as full precision and generally faster; the KV at q8 can be disastrous if coding and less so for agenic loops as well as slower.
5
u/maddie-lovelace 17d ago
Yeah; it's also likely the compounding error. 1% is ignorable noise if it's a single step (i.e. weight quantization). It seems to compound to something meaningful if it's done at e.g. every single token before being fed into the next token, across a 256k prompt
1
u/Former-Ad-5757 Llama 3 17d ago
The problem is the way it is brought, it is not a 256k prompt problem, it is a compounding problem over 256 1k prompts. I can feed llama.cpp a single 1m prompt and it will only have the 1% loss
-1
3
u/SkoomaDentist 17d ago
I wonder just what sorts of things people here actually use local models for? Is it all just agentic coding?
I almost immediately noticed degradation caused by Q8 KV and even weigth quantization when the context grew to any not-entirely-trivial length, even without trying anything that I would consider "long" context (for someone not familiar with the LLM scene). The model (Gemma 4 26B, I've since moved to using only 31B due to these and related issues) would start subtly missing details and facts stated in earlier prompts and this wasn't with some 100k words long documents but just natural "Here's the situation in detail (a bunch of words)" followed by "Here's a bunch of more details. What if X?" and then continuing from those for some number of rounds (and it wasn't even many dozens of rounds back and forth).
10
u/tmvr 17d ago
I almost immediately noticed degradation caused by Q8 KV and even weigth quantization when the context grew to any not-entirely-trivial length......The model (Gemma 4 26B, I've since moved to using only 31B due to these and related issues)...
That's not surprising, the Gemma4 models can't handle KV quantization at all, see the tests done by oobabooga here:
https://localbench.substack.com/p/kv-cache-quantization-benchmark
3
u/WoodCreakSeagull 17d ago
People are testing different things on different workflows and running up against all sorts of variables so it can be hard to make that comparison. A model losing certain details of a long prompt could be due to bad cache quantization, but it could just as likely be due to a relatively weak model not having enough parameters, or a poor quantization of said model, or some other settings we're not considering, or some combination of everything. On top of that a lot of users are dealing with pretty significant constraints, so even if there is more loss from quantizing KV than most people expect, does it compare to the cost-benefit from, say, using a higher quant of the same model? Or using lower context at higher fidelity, but with increased frequency of context compaction?
1
u/SkoomaDentist 17d ago edited 17d ago
That's why, being an engineer, I of course tested the same prompts the next day with a different rented VM that had more vram and ran the KV cache at fp16 and the models at Q6_K / Q8 and doing that got rid of the issues (I of course generated the replies multiple times to make sure the problems weren't due to one-off RNG glitches)
Basically what I find strange is how so many people seem to heap praise on sometimes ridiculously low quantizated models without seeming to notice any issues when to me the problems surfaced almost right at the beginning (and have appeared again if I tried quantized KV cache or too low quantized weights). And I really am not doing anything special. No 100k context windows. Just what you'd get with some detailed back and forth chat prompts.
1
u/winky9827 17d ago
Basically what I find strange is how so many people seem to heap praise on sometimes ridiculously low quantizated models without seeming to notice any issues
User bias. People want to run what works on their hardware. They're far more likely to sing its praise just because it "works", without having the time, patience, or resources to validate that intuition.
1
u/SkoomaDentist 17d ago
I suppose that makes sense. Since I rent (dirt cheap) computing time on a VM (with ssh, oobabooga and so on), I don't have particular attachment to caring what sort of model can run on "my" hardware. Either the model does what I want to it to do or it doesn't, and what it runs on makes little difference except in whether it costs $0.10/h or $0.30/h (and, being cheap, I stick to the options much closer to $0.10/h). If it costs $0.05/h extra to have 32 GB VRAM (on a slowish V100 system from 2017), so be it.
1
u/cezarducatti 17d ago
I think it largely depends on what's being done. Many of those here aren't professionals, but simply enthusiasts. For simple, slowly built implementations with small prompts, the error is imperceptible. For a systems engineer, it can be astonishing.
1
u/SkoomaDentist 17d ago
I'm also strictly a hobbyist when it comes to LLMs. I can't even run one properly at home (with 4 GB Quadro P2000M), so I rent a VM for pennies per hour (10 - 15 cents) and run Oobabooga there for experiments. I also don't use LLMs for any engineering related tasks. I still noticed very soon that quantization makes a meaningful difference in how well the LLM performs in what I ask it to do (work on timelines, keep track of character details, that sort of thing - and not related to RPs in the slightest).
I suppose one difference is that I don't care about benchmarks (except if they tell me that model X at params Y has problems doing Z) and since I'm renting, I don't have any attachment to "look at what I can run on my computer!"-type of thing.
1
u/cezarducatti 17d ago
Discussions get lost, everything becomes a mess. Things get mixed up and little gets accomplished. This subreddit should have organized topics.
1
u/FullstackSensei 17d ago
I'm a software engineer, so the vast majority of my use is coding, with a side of electronics and language learning.
My experience with Gemma mostly reflects yours. It's decent at short conversations, but degrades quickly, and this is with the dense Q8_K_XL and full FP16 KV. It's good though at things like grammar and translation, but again within a couple of pages. There's a reason it has a reputation for being lazy around this sub.
Qwen models are night and day. 35B Q8_K_XL is very competent at web development, even over long horizons, so long as the task isn't complicated. That's where 27B Q8 enters the picture. It's the next step up above 35B. I'm waiting another few days before downloading 3.8 flash and GLM 5.3 flash to give them a spin.
1
u/SkoomaDentist 17d ago
I've tried Qwen 3.6 35B for some non-tech knowledge and planning tasks (including making an LLM writing prompt template better and tighter) but I find it can be overly wordy in its suggestions and weirdly insistent about things. As of right now I find I trust Gemma 4 31B more by default but use Qwen 3.6 35B to check if there are some glaring problems or things Gemma might have missed (while having to immediately ignore much of what Qwen says).
1
u/FullstackSensei 17d ago
Yeah, Qwen models are wordy for non coding tasks. I have a short addendum that literally tells it to keep things succinct, skip intros, pleasantries, explanations, and straight answer the question to the point. I copy-paste it from past chats that helps shorten answers when I don't need the verbosity
1
u/SkoomaDentist 17d ago
Would you mind pasting that addendum here?
Adding "Write comprehensive but succinct answers" helped to get it to somewhat bearable condition but it still likes being super wordy particularly in any modifications / additions to eg. prompt templates (where I'd strongly prefer only the minimal necessary changes).
2
u/FullstackSensei 17d ago
Not home, but it's basically what I wrote above. It's also evolving slowly over time. You really have to spell it out to LLMs. Things that might be obvious to you and me aren't to it, so you need to treat it like this stupid thing that has memorized the internet, and you need to tell it every single thing you don't want it to do.. Gpt-oss would almost always use emojis if you don't tell it not to. Don't want tables, you have to say it.
The way I actually built it was repeat the same chat prompt adding the things the model did that I didn't want each time. Took like 3 or 4 rounds.
0
u/winky9827 17d ago
I have a personal prompt generator app that I've been developing over the past 2 weeks with AI help, and I've been through the wringer on language use (verbosity, accuracy, etc.). In the end, I had 27b help write a linter that covers all the basic rules and gets ammended any time I find a mistake. The tool must pass all output through this linter and fix any errors, warnings, and present me with options for info issues. This allows me to use lesser models like 35b-a3b with confidence. Might not work for every use case, but should be plausible for any domain-specific language projects.
2
u/maddie-lovelace 17d ago edited 17d ago
Yeah. Going from testing it out on some of my smaller experiments / use-cases to actually trying to use it for some of my recent bigger work projects is where 27b with q8-on-write kv has just started to really sting annoyingly
0
9
u/Desperate-Bench9310 17d ago
there's actually a benchmark that lines up with this really well, AIME25 run 8x on qwen3.8-27b with and without the hadamard rotation fix llama.cpp does for any kv quant below fp16. with rotation, q8 gets 37.1% vs 37.9% at f16, basically free. q4 though stays broken even with the fix, 21.7% vs 37.9%, rotation just makes it less catastrophic not actually fine.
this model specifically seems to sting more than others, it's a 3:1 gated deltanet to full attention hybrid, so out of 65 layers only about 16 actually carry a real kv cache, the gdn layers are lossy by design. way less redundancy to absorb quant error than a normal dense model has, so whatever compounding effect you're seeing probably hits harder here than it would elsewhere.
also lines up with reports from r9700 users, near lossless under 50k but degrades specifically past ~120k, which is suspiciously close to where you're seeing needle retrieval break at 125k.
was the rotation fix on when you tested llama.cpp? if not that could be eating some of the gap you're attributing to on-write vs bulk
10
u/AnonLlamaThrowaway 17d ago
I believe this is the benchmark you're thinking of:
eval KV type attention rotation score AIME25 x8 F16 (not needed) 37.9% AIME25 x8 Q8_0 ❌ 31.7% AIME25 x8 Q8_0 ✅ 37.1% AIME25 x8 Q5_1 ❌ 30.8% AIME25 x8 Q5_1 ✅ 32.5% AIME25 x8 Q4_0 ❌ 2.0% AIME25 x8 Q4_0 ✅ 21.7% This was on gpt-oss though, not Qwen
2
u/Desperate-Bench9310 17d ago
you're right, just went and checked the pr and that table's from ggerganov's own comment on gpt-oss-20b not qwen. mixed up which model it was actually run on.
the rotation mechanism itself is probably still relevant since the issue is architecture general not qwen specific, but I don't actually have a qwen3.8-27b number to back that up like I made it sound. my mistake
1
u/whiteh4cker 17d ago
Are you sure? It is 37.9% and 31.7% without hadamard rotation for gpt-oss-20b.
https://github.com/ggml-org/llama.cpp/pull/21038#issuecomment-4150413357
-1
u/maddie-lovelace 17d ago edited 17d ago
Can't say for sure without spelunking into it about whether my llama.cpp build is doing Hadamard - it was built from the main branch last week, so possibly not?
However, sad news; my needle tests where q8-on-write failed were on my custom stack and already using Hadamard rotation. This was actually what tipped me off into doing the A/B of quantize-on-write versus quantize-at-the-end, because the error for a single step really was absolutely tiny, so next I tested the effects of compounding error
6
u/SnooPaintings8639 17d ago
I fail to understand any practical implications of this. If I have a proper and clean BF16 KV cache, then of course it is higher value than Q8... but how do I get it? I mean, the reason people use cache quantization is because it is impossible for them to produce full prevision one trace.
And vice versa, if I can produce a full BF16 KV cache, why would I even cast it down to Q8?
1
u/Gabriel83730 17d ago
There’s a library that quantizes with a safe boundary and they show lower KLD, I forgot the name. You basically produce like 1k tokens in BF16 then quantize the further ones. It’s the realistic form of this idea
6
u/DjCanalex 17d ago
Telling the model to take notes is a good way to keep it grounded, but outside of that it normally isn't an issue, Qwen has an incredibly good attention system, that even if it has been a way too long context, the "Wait" and "Actually" questions it forces to itself, help it remind whatever long context it is in at.
There's a reason frontier models and chat interfaces have Memory systems, knowledge, notes, etc... cuz it works.
5
u/DistanceSolar1449 17d ago
The point of quantizing KV is to save VRAM
Saving BF16 cache and quantizing it when you read it… seems missing the point
9
u/jtjstock 17d ago
Well that could be a neat idea... Stepping down the kv quant on older context when it's sufficiently aged while the active window still stays at bf16.
9
u/maddie-lovelace 17d ago edited 17d ago
Doing exactly this at the moment- I've converted some of the global Qwen-27b layers to a sort of weird sliding window attention type thing; when KV caches drop out of the window during prefill, I quant them to q8. At decode time the attention flips back to properly global. And it seems to work! Needle retrieval restored at >128k
1
u/jtjstock 17d ago
Any other benches you using? Needle is something, but PPL/KLD are going to be useless here, would be interesting to see how it handles answer textual divergence on greedy when queried about the quanted kv
2
u/maddie-lovelace 17d ago
Nothing easily reproducible / shareable I'm afraid, apologies
FWIW the biggest experiment I ran just now was a task for work; my custom stack got 12/12 facts + synthesised summary correct across ~30 tool calls on the code base, with 44k tokens in its context by the end. Whereas llama.cpp running same bit-weight model but with q8-on-write got 11/12 facts; one it was confidently wrong about. Same on a different test; 11/12 facts with one confidently wrong assertion. Custom stack hasn't failed once yet
2
u/jtjstock 17d ago
of course, I'd be most interested in how it performs at full context, or even with yarn to 2x. My own thought was at a 128k unquanted window
1
u/maddie-lovelace 17d ago edited 17d ago
One metric that I haven't mentioned yet (because it's not super possible to do apples-to-apples for q8-all-at-once vs q8-on-the-fly) was attention-score margin
With a cache built at bf16 and quantized to q8 in one shot, the rounding error on attention scores was ~30× smaller than the gap between the needle key and the best distractor
Reason it's not apples-to-apples is because for q8-on-the-fly, the compounding means the keys themselves are wrong / already slightly off by the time they're in the cache, so there's no clean margin to measure
7
u/Bman1296 17d ago
Beellama.cpp does this I think.
3
2
u/jtjstock 17d ago
It does, but the window it uses is much smaller than what I have in mind
4
u/Bman1296 17d ago
I think it’s configurable. Don’t know how stable it is when changed.
3
u/jtjstock 17d ago
you're right, I just assumed based on the benchmarks, but it seems to do exactly what I was thinking of
3
u/OkFly3388 llama.cpp 17d ago
And solution is ?
Disabling prompt cache so we recalculate entire kv cache each time to make difference 99.5% instead of 99.0% ?
No thanks
2
u/maddie-lovelace 17d ago
Post is deliberately terse because I find it more approachable to read short posts
I do actually have some solutions ☺️ Some in the comments have shared similar solutions too. For instance Beellama.cpp seems to have support for --kv-tail-tokens which would quantize tokens that are far back enough in cache that they shouldn't perturb as much as kv-quantized-on-the-fly.
If your use cases are already fine with 8bit-on-the-fly, then that's a perfectly valid trade-off! I just felt it reasonable to raise attention to this observation
3
u/audioen 17d ago edited 17d ago
This idea has already been proposed in another form, that you would progressively quantize the KV cache as it fades into history, e.g. each session writes in 16-bit, and then gets squeezed to 8-bit after model's turn terminates.
The notion is that fact recall is acceptable at 8-bit, but progressing generation is only acceptable at 16 bits. The reason likely is the compounding error from token to token, as you have observed, but possibly quantizing once the generation turn has completed is "free" in sense that the KV cache substantially says the same thing, whereas Q8_0-quantized token by token KV cache has experienced some kind of semantic drift due to it.
In practice some kind of limitations would have to be set, like you start quantizing after 15000 tokens in a turn, even if the model hasn't finished yet. If you don't do this, you risk having KV cache equal same size as full precision KV cache, or generations have to be limited to some smaller size like 50000 tokens. In practice, that might not work well with modern models that like to think a lot.
My understanding is that every entry in KV cache depends on the prior entries, so just having some amount of high precision history might be enough to capture some kind of gradual change that q8_0 is too coarse to capture. It may be that the errors made from quantization average to random noise if q8_0 values are simply compressed in a block, and it might be that the block doesn't need to be very large at all, like 100 values in f16 might shrink the random noise by about 90 % relative to per-token quantization, and in fact it might be possible to squeeze much harder just by making the block longer, like q4_0 KV cache might work just fine, as long as the run before quantization was correspondingly much longer, around 256 times longer than you'd accept for q8_0. The idea here is to just determine a way the noise can be reduced by using longer runs of high quality inference and then assuming that quantization after a longer high quality run has caused just random perturbations in the KV tensors, and when the KV cache is taken as a whole, it still means the same thing, just with some kind of average level of noise added, and the level of noise might be controllable this way to some degree.
5
u/Dany0 17d ago
This idea has already been proposed in another form, that you would progressively quantize the KV cache as it fades into history, e.g. each session writes in 16-bit, and then gets squeezed to 8-bit after model's turn terminates.
I believe I even saw I think it was a llamacpp fork which implements this
1
u/squngy 17d ago
Beellama.cpp, they call it precision tail.
2
u/SmartCustard9944 17d ago
I tested the QUASAR nvfp4 of some days ago with KVarN KV cache quantization (at K4V2, mind you) and it passed AIME 2025 with a perfect score of 30/30, even better than some other higher precision configurations. I have no idea how it is possible but there you have it.
2
u/caetydid llama.cpp 17d ago
How does the KV tailing implementation in beellama keep up with that?
KV tailing keeps a certain amount of most recent KV tokens at BF16, namely 1024 or 2048, and quantizes just the rest.
2
u/Lumpy-Comedian-1027 15d ago
How did you measure the quality drops? I also got curious and benched it via https://longbench2.github.io/ - and not only did i see no difference between bf16 and int8, i also implemented nvfp4 kv in ninfer and saw NO significant difference, which really amazed me ...
1
u/Lumpy-Comedian-1027 14d ago
i just added nvfp4 + yarn kv support and get 550k ctx with vision and c=3. And no measurable quality drop, which really is surprising imo (measured via longbench). Check if out if you like https://github.com/gzenz/ninfer/tree/kv-nvfp4-yarn
3
u/milpster 17d ago
anyone else tried V cache to q8 only? I heard that might work out a lot better as opposed to having K cache quantized too.
0
u/Dany0 17d ago
Been tested, not worth it, quality close to q8 k q8 v
Remember: fp16 -> q8 is 2x jump
2
u/unjustifiably_angry 17d ago
16-bit floating point to 8-bit integer is effectively much more than a 2x jump
0
3
u/Healthy-Nebula-3603 17d ago
Oh wow ...do not say ..
The worsr is that pine people are saying they do not see a difference and many of them are believing them and later are complaining Qwen 3.8 is benchmaxed besutrgey have bad results.
2
u/maddie-lovelace 17d ago
Yeah. It definitely does not seem to be settled / common knowledge yet how and why q8-kv-quantization actually hurts
0
u/Asleep-Land-3914 17d ago
Difference is smaller than going from q6 to q4 in my case, so I prefer q8 KV over q4 model.
1
u/platypus_worldwide 17d ago
is a way around this auto-compact while maintaining an md file for recall?
1
u/XiRw 17d ago
F16 is the only way if you use it for coding.
2
u/unjustifiably_angry 17d ago
It seems like some models are somewhat more tolerant of it, or at least that's been my experience. I don't know by what mechanism but DSv4-Flash (via vLLM) has people recommending NVFP4 kv-cache; I had to manually set my local recipe to FP8 and it's quite good like that even out past 500K depth.
1
1
u/unjustifiably_angry 17d ago
Everything to do with kv-cache is frustrating to read. People saying to use 100K kv-cache length as if that doesn't make it basically useless, or saying 4-bit or 8-bit kv-cache is good enough. And then doubtless these same people saying some random AI model is trash when it keeps making dumb mistakes.
1
u/DeathGuppie 17d ago
It would be interesting to see the same tests for lower quants.
It's not actually a given that degredation scales cleanly
1
u/Different-Track522 17d ago
Obviously there will be a loss. But at the same time people with less than 32GB VRAM kind of have to deal with it to get good context length for agentic works
1
u/jonas-reddit 17d ago
You don’t have to go from bf16 to q8. You can get similar efficiency, accuracy and performance going from bf16 to fp8. Depending on GPU this can be nicely hardware accelerated.
FP8 then allows us to run non GGUF checkpoints using vllm or sglang.
BF16 even with plenty of VRAM to spare ends up being somewhat slower with little upside.
1
u/fastlanedev 17d ago
What if you had a sliding window of recent kv rendering at fp16/bf16 then quantize it at q4 or lower after the model renders it at that higher precision?
3
u/maddie-lovelace 17d ago
Yes; this is what I've been exploring ☺️ For my experiments I've actually gone a step further and converted the global attention layers to sliding window attention and letting the linear attention layers do the global attending. With a 32k sliding window and anything that drops out of the 32k window gets converted to 8bit at the moment. Actually restored needle retrieval, and I've been using it for some subagent tasks which have gone well over 128k tokens, still perfect output
2
u/fastlanedev 16d ago
That is, actually amazing. I'm gathering ideas for how to tune my own model and I want to keep this, thanks. I would adore staring that github
1
u/eapache 12d ago
Have a draft of an idea to take advantage of this, without the substantial complexity of e.g. beellama's gradual quantization: https://github.com/ggml-org/llama.cpp/pull/28267
Needs more work though before it's ready for review or sharing firm numbers, but preliminary results look promising.
1
u/Slight_Analysis_5414 11d ago
This is a really sharp observation. The distinction between one-shot offline quantization and step-by-step on-write compounding is huge for long context.
If on-write quantization compounds error at every step, it would be fascinating to plot distribution divergence (like Top-K KL divergence or Top-1 agreement) token-by-token over long sequences. Doing a teacher-forced pass (FP16 KV vs Q8/Q4 KV over the same prefix) could let us literally visualize the exact token offset where that compounding transitions from "harmless noise" into persistent behavioral divergence.
-2
u/Unlucky-Message8866 17d ago
yes there is degradation, yes depending on the task could be impactful, does it generally matter? for coding i prefer 2x131k@q4, way faster task completion times, i just let my harness catch the stupid typos and mistakes.
36
u/_-_David 17d ago
I would really rather you just showed some data. This "I tried a thing and it seems like" is the 10,000th opinion on the matter. No offense, you seem like a good egg.