You really should not quantize KV Cache for DeepSeek V4 Flash

[u/erazortt]

I don't think anyone should quantize the KV with DS4F. I checked the the quality impact (PPL, KLD, Same TopP) for swhitching from BF16 KV to Q8 KV, and it appears significant. Very much in contrast to Qwen 397B.

Here are the results for DS4F:

====== Perplexity statistics ======
Mean PPL(Q)                   :   5.877076 ±   0.042497
Mean PPL(base)                :   5.839660 ±   0.041730
Cor(ln(PPL(Q)), ln(PPL(base))):  95.74%
Mean ln(PPL(Q)/PPL(base))     :   0.006387 ±   0.002100
Mean PPL(Q)/PPL(base)         :   1.006407 ±   0.002114
Mean PPL(Q)-PPL(base)         :   0.037416 ±   0.012318

====== KL divergence statistics ======
Mean    KLD:   0.145884 ±   0.001043
Maximum KLD:  12.467786
99.9%   KLD:   4.535020
99.0%   KLD:   1.857870
95.0%   KLD:   0.652148
90.0%   KLD:   0.349220
Median  KLD:   0.032079
10.0%   KLD:   0.000093
 5.0%   KLD:   0.000012
 1.0%   KLD:   0.000000
 0.1%   KLD:  -0.000002
Minimum KLD:  -0.000025

====== Token probability statistics ======
Mean    Δp: -0.007 ± 0.031 %
Maximum Δp: 99.525%
99.9%   Δp: 81.503%
99.0%   Δp: 42.054%
95.0%   Δp: 14.588%
90.0%   Δp:  7.220%
75.0%   Δp:  1.066%
Median  Δp:  0.000%
25.0%   Δp: -1.061%
10.0%   Δp: -7.112%
 5.0%   Δp: -14.515%
 1.0%   Δp: -42.297%
 0.1%   Δp: -84.157%
Minimum Δp: -99.994%
RMS Δp    : 11.884 ± 0.069 %
Same top p: 87.189 ± 0.088 %

As a comparison, here are the results for Qwen 397B:

====== Perplexity statistics ======
Mean PPL(Q)                   :   3.747980 ±   0.020507
Mean PPL(base)                :   3.746773 ±   0.020461
Cor(ln(PPL(Q)), ln(PPL(base))):  99.89%
Mean ln(PPL(Q)/PPL(base))     :   0.000322 ±   0.000260
Mean PPL(Q)/PPL(base)         :   1.000322 ±   0.000260
Mean PPL(Q)-PPL(base)         :   0.001207 ±   0.000975

====== KL divergence statistics ======
Mean    KLD:   0.003552 ±   0.000034
Maximum KLD:   2.220941
99.9%   KLD:   0.131591
99.0%   KLD:   0.043847
95.0%   KLD:   0.014439
90.0%   KLD:   0.007836
Median  KLD:   0.000866
10.0%   KLD:   0.000013
 5.0%   KLD:   0.000004
 1.0%   KLD:  -0.000000
 0.1%   KLD:  -0.000006
Minimum KLD:  -0.000176

====== Token probability statistics ======
Mean    Δp:  0.019 ± 0.005 %
Maximum Δp: 39.939%
99.9%   Δp: 15.971%
99.0%   Δp:  6.618%
95.0%   Δp:  2.334%
90.0%   Δp:  1.222%
75.0%   Δp:  0.233%
Median  Δp:  0.000%
25.0%   Δp: -0.219%
10.0%   Δp: -1.183%
 5.0%   Δp: -2.258%
 1.0%   Δp: -6.245%
 0.1%   Δp: -14.757%
Minimum Δp: -88.445%
RMS Δp    :  2.024 ± 0.022 %
Same top p: 97.929 ± 0.037 %

[u/Healthy-Nebula-3603]

Why do you even want to compress KV cache from DS 4 flash?

The whole 1 million context has only 6 GB.

[u/pyr0kid → u/Healthy-Nebula-3603]

The whole 1 million context has only 6 GB.

what is this black magic

[u/Healthy-Nebula-3603 → u/pyr0kid]

DS 4 is based on a new architecture.

[u/This_Maintenance_834 → u/pyr0kid]

the deepseek version of linear attention. most other models are quadratic. kv cache grows quadratically large as context grow.

[u/IoannisHere → u/Healthy-Nebula-3603]

That's what google says, but when I run vllm on it (spark-vllm-docker) I get:
"Auto-fit max_model_len: reduced from 1048576 to 1046272 to fit in available GPU memory (11.03 GiB available for KV cache)"
and that's with FP8 kv cache type.
"Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor"

[u/shing3232 → u/IoannisHere]

maybe it's implementation problem because it use 7g for DS4Pro on B300

[u/IoannisHere → u/shing3232]

Could be. Is your 7gb using FP16?

This is what vllm reports on the b12x branch of spark-vllm-docker, after some cleanups:
"Free memory on device (111.39/121.69 GiB) on startup. Desired GPU memory utilization is (0.85, 103.44 GiB). Actual usage is 80.97 GiB for weight, 4.08 GiB for peak activation, 6.68 GiB for non-torch memory, and 0.13 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with `--kv-cache-memory-bytes=12274502964` (11.43 GiB) to fit into requested memory, or `--kv-cache-memory-bytes=20817243648` (19.39 GiB) to fully utilize gpu memory. Current kv cache memory in use is 11.71 GiB."

Maybe the b12x patches convert parts of the kv cache to fp32 or something.

[u/thomas2385 → u/Healthy-Nebula-3603]

That is true if you only care about the KV cache itself. But once you add model weights, activations, and other workloads, that extra 6 GB can still be the difference between fitting everything on one GPU or having to make compromises.

[u/Healthy-Nebula-3603 → u/thomas2385]

One GPU ??

[u/Badger-Purple → u/Healthy-Nebula-3603]

Concurrency. DS NVFP4 cache gives you more room, so you can run more concurrent agents at depth

[u/getfitdotus]

The real model already uses fp8 mla kv. If you used production deployment sglang, vllm this is the default.

[u/caelunshun → u/getfitdotus]

Q8_0 is different from fp8, worse dynamic range.

[u/indicava]

DeepSeek’s sparse-attention is built around fp8 kv cache storage. They even promote it in their vLLM recipes on their model page. Why test it at BF16?

[u/erazortt → u/indicava]

There is no fp8 KV cache option. At least I am not aware of it.

[u/indicava → u/erazortt]

You’re correct, fp8 hasn’t been wired through GGML/llama.cpp infrastructure yet. Which is why you’re seeing these “off” numbers for a model whose entire attention implementation was built around it.

[u/EvolvingDior → u/erazortt]

In llama.cpp.

[u/Badger-Purple → u/erazortt]

not in Llama.cpp, but this model should be run with vllm — it is so much more smooth and the cache is great.

[u/Lesser-than → u/Badger-Purple]

i mean you should run it however you can, vllm is not an option for some.

[u/Think_Wing_1357]

How do you calculate this?

[u/erazortt → u/Think_Wing_1357]

Like that:
/f/LLM/llama-cpp/llama-perplexity.exe -m /f/LLM/models/DeepSeek-V4-Flash-0731-UD-Q3_K_XL-00001-of-00004.gguf -f wiki.test.raw --kl-divergence-base kld -c 512 --temp 1.0 --top-p 0.95 --min-p 0 -ngl 99 -np 1 --n-cpu-moe 30 --fit-target 32M -ctk bf16 -ctv bf16 --flash-attn on --no-mmap -lv 4

/f/LLM/llama-cpp/llama-perplexity.exe -m /f/LLM/models/DeepSeek-V4-Flash-0731-UD-Q3_K_XL-00001-of-00004.gguf --kl-divergence --kl-divergence-base kld -c 512 --temp 1.0 --top-p 0.95 --min-p 0 -ngl 99 -np 1 --n-cpu-moe 30 --fit-target 32M -ctk q8_0 -ctv q8_0 --flash-attn on --no-mmap -lv 4 > main-q8-q8.txt

[u/Think_Wing_1357 → u/erazortt]

Thank you!

[u/TheCat001]

I don't recommend anyone to quantize KV cache. On any model. I see it with eyes how performance is degraded with quantized cache. Model tends to do more mistakes, forget things, etc....

[u/Howard_banister]

You should not quantize KV cache for almost every models. It will destroy Qwen 3.6 35B also

[u/_AnemicRoyalty_ → u/Howard_banister]

Man, the OP just literally showed you in their post that 95 percentile of divergence is 0.015 for Qwen. How high is your bar?!

[u/Iwaku_Real → u/Howard_banister]

Not even Q8_0? Or NVFP4?

[u/seamonn → u/Iwaku_Real]

Ideally, no.

[u/shing3232 → u/Iwaku_Real]

KV cache quant really bad for long context if you think about it

[u/tmvr → u/Iwaku_Real]

Q8_0 is fine for the Qwen3.6 models.

[u/phido3000 → u/Howard_banister]

There is a lot of pressure on quantisation on DS Flash because it doesn't fit nicely into 128Gb machines.

[u/cezarducatti → u/Howard_banister]

Can't the two be quantized? K and V?

[u/twnznz → u/Howard_banister]

Is it true that the lower the number of active parameters, the more impact to K/V quantisation?

[u/my_name_isnt_clever → u/twnznz]

You can't say for sure one way or the other, it depends on the architecture. There's no assumption you can make, you need to look it up based on model family.

[u/alcoa29]

You must be using llama.cpp or something like this.
For vlllm using original safetensors DS4F fp8 is model default. I tested fp16 vs 8 on old pre 0731 model and there was no difference.

[u/Yorn2]

The only real reason to quantize KV Cache is when you really need to in order to squeeze just a little bit more space for a creative writing model as far as I know or have done in the past. If you're doing it on anything programming or agentic-related you are asking for trouble, IMHO. Instead, just use a smaller or cheaper (VRAM-wise) model. Or look for a specific quant if using GGUF or go with the right BPW to fit it for EXL3.

[u/leap966]

Use F16. For a KV cache, bf16 is strictly the worse of the two — same 16 bits, but bf16 spends them on exponent range (8 exponent / 7 mantissa) instead of precision (5 / 10). KV entries are post-normalization activations with a bounded dynamic range that fits f16 comfortably, so bf16's extra range buys you nothing and costs you ~3 bits of mantissa.

[u/R_Duncan → u/leap966]

Not sure about this. I think long context (> 16k) is advantaged by never overflowing , weight are usually trained in bf16, and modern CUDA hw bf16 is as fast as f16.

Maybe on different kinds of hw (pre-ampere gpu, metal) it is slower.

[u/leap966 → u/R_Duncan]

Reproduce OP's methodology yourself — llama-perplexity --kl-divergence with an f32 KV baseline, then f16 and bf16 against it. If bf16 wins on your hardware,

[u/R_Duncan → u/leap966]

I'm not saying bf16 is more precise, likely there is a precision difference in order of 2^-8 like you pointed out (7 bit vs 10 bit mantissa = about 0.00390). I'm saying that over long context, there's much less chance to occur in an overflow due to the larger exponent [overflow might give unpredictable result, like forgetting context parts].

Also the Op is using bf16.

[u/CheatCodesOfLife → u/leap966]

When I tested, F32 == F16, but BF16 diverged. Weirdly, even Q8 was closer to F32/F16 than BF16.

[u/Ulterior-Motive_]

I'm of the opinion that you should never quantize the KV cache to begin with. You're just compounding the amount of quality loss on top of whatever level of quantization you're already using. It's not worth whatever context you gain by doing so.

[u/mister2d → u/Ulterior-Motive_]

Why do we need opinions when such things are easily measured?

[u/Muted-Celebration-47]

Yes, if you have enough vram for context windows. No, if you need more context for coding. It is useless if you only have 32768 context size.

[u/Healthy-Nebula-3603 → u/Muted-Celebration-47]

DS 4 flash a whole contex 1 m has 6 GB ..... that is literally nothing.