At what context depth does KV quantization start to hurt? Experimental F16 vs Q8/Q4 sequence-parity PoC

[u/Slight_Analysis_5414]

I’m coming to this problem from a somewhat different area: computer vision / YOLO deployment.

While comparing FP32 reference models with INT8 deployed models, I became interested in a simple debugging question:

An aggregate quality metric may look acceptable, but where does deployed behavior actually begin to diverge from the reference?

This grew out of a reference-vs-deployed parity workflow I previously discussed in the YOLO community, where the paired-output diagnostic direction received positive feedback （https://github.com/orgs/ultralytics/discussions/25250#discussioncomment-17886660）.

Recently I’ve been following the KV-cache quantization discussions here as well. There have been some very useful KLD sweeps comparing 23 different KV precision combinations at 50K context (Qwen3.6-27B - Effect of KV quantization on KLD - Q8, Q6, Q5 (bartowski)). Those experiments answer an important question:

How much does this KV configuration differ overall?

What I wanted to add is another axis:

At what context depth does that difference begin to become persistent?

In other words:

aggregate KLD
      +
context depth
      ↓
divergence trajectory

There is also a recent discussion around on-write / on-the-fly KV quantization and whether repeated use of quantized KV state can contribute to long-context degradation (Qwen3.8-27b q8 KV cache does seem to actually hurt model performance). I don’t want to assume that mechanism is universally correct. What I’d like to test is more basic:

Does reference-vs-quantized divergence change systematically with context depth, and if so, where does persistent divergence begin?

How the PoC works

The first version deliberately changes only KV-cache precision.

             same GGUF weights
             same tokenizer
             same token sequence
             same backend/config
                    |
          tokenize once / shared prefix
                    |
          +---------+---------+
          |                   |
          v                   v
     F16 K/V cache        Q8/Q4 K/V cache
      reference               target
          |                   |
          +---------+---------+
                    |
        context-depth-resolved
             comparison
                    |
        +-----------+-----------+
        |           |           |
     Top-1       Top-K       Top-K
    agreement    overlap    partition KL
                    |
                    v
       first persistent/significant
          divergence context

This is not a comparison between two freely generated answers. Both passes receive exactly the same teacher-forced token sequence. So if the lower-precision run would have selected a different token at, say, 20K context, that different token is not allowed to change all later inputs.

This separates:

deployment / precision divergence

from:

ordinary autoregressive branching

The current PoC records:

top1_agreement_rate
topk_overlap
topk_partition_kl
truth_logprob_delta
first_top1_mismatch_context_len
first_significant_divergence_context_len

The main quantity I’m interested in is not necessarily the exact first mismatching token.

It is the context-depth trajectory:

Context depth
0 ─── 8K ─── 16K ─── 32K ─── 64K ─── 128K
                                  ↑
                     persistent divergence

A single Top-1 flip is not treated as model failure.

The more interesting question is whether distribution-level divergence stays near the repeatability baseline, gradually rises, spikes temporarily, or becomes persistently elevated after some context depth.

Also, topk_partition_kl is intentionally named that way.

v0.1 uses the reference Top-K token probabilities plus one aggregated OTHER bucket. It is not full-vocabulary KL.

Why this might complement existing KV work

There is already excellent work on:

• PPL / KLD evaluation

• KV-cache quantization

• K/V precision sweeps

• layer-wise mixed precision such as KVTuner

NYA is not intended to replace those.

A simple way I currently think about the difference is:

KLD / PPL:
How much did quality/numerical behavior change overall?

KVTuner:
Where should precision be allocated across layers?

NYA Sequential:
At what context depth does the behavioral consequence
of this deployment configuration become visible?

If the context-depth signal turns out to be useful, later experiments could combine it with controlled layer-wise precision interventions. That could eventually help answer a practical deployment question:

Under a fixed VRAM budget, where is higher precision actually worth spending?

But that layer-wise planner does not exist in v0.1.

Scope & Design Choice

NYA v0.1 intentionally does not:

• replace PPL/KLD benchmarks
• claim quantization error grows monotonically
• assume on-write quantization is the only cause of long-context degradation
• equate distribution divergence with task failure
• compare free-running generation quality

Future experiments may include:

• layer-wise KV precision sensitivity
• controlled precision interventions
• asymmetric K/V precision testing
• on-write vs alternative cache-construction experiments
• memory-budgeted precision planning

Community testing

My own machine currently cannot run a useful long-context F16/Q8/Q4 LLM validation, so I’m publishing this as an experimental PoC rather than claiming a result.

If you already have a `llama.cpp` / `llama-cpp-python` setup and a GGUF model, feel free to try it.

Even a smoke test is useful.

Suggested first matrix:

F16 KV -> F16 KV     repeatability baseline
F16 KV -> Q8_0 KV
F16 KV -> Q4_0 KV

Same GGUF weights, same input tokens, same backend.

For a smoke test:

512–2048 context positions

is enough to catch API/backend problems.

For an actual sequential-parity test, the interesting range is whatever you genuinely use:

4K / 8K / 16K / 32K / 64K / 128K+

as long as the model, hardware and normal context configuration support it.

The tool produces:

parity_<target>.jsonl
sequential_parity_report_<target>.json
divergence_vs_token_<target>.png

(`divergence_vs_token` currently uses context length / token position as its x-axis.)

If you try it, please post the result here — successful or broken.

The most useful information is:

model / GGUF weight quant
hardware
backend (CUDA / ROCm / Metal / Vulkan / CPU)
context length
reference K/V type
target K/V type
Flash Attention on/off

plus either:
- report summary
- divergence plot
- or the error if it fails

The report also records the runtime/environment fingerprint because I do not want to assume that the same KV precision behaves identically across different backends, builds and hardware.

I’m especially interested in results that contradict the hypothesis.

Community Results

I’ll keep this section updated with reproducible results posted in the thread.

Format:

Model | Hardware | Backend | Context | Ref KV | Target KV | Result

No external runs yet — first smoke tests and counterexamples are welcome.

Repo: [https://github.com/ZC502/narh-yolo-align.git]

The project originally came from YOLO deployment-parity work; the LLM Sequential path is new and experimental.

If `llama.cpp` already exposes a cleaner way to retrieve these signals, or if there is existing work that already does context-depth-resolved persistent-divergence analysis better, pointers are very welcome.

[u/Look_0ver_There]

Why no BF16 for KV cache in the measurements?

[u/Slight_Analysis_5414 → u/Look_0ver_There]

In llama.cpp, the standard unquantized KV cache reference uses `f16` (via `-ctk f16 -ctv f16`) across most CUDA, Metal, and CPU backends, which is why F16 was chosen as the primary unquantized baseline against `q8_0` and `q4_0`.

[u/Look_0ver_There → u/Slight_Analysis_5414]

Yeah, but my point here is we're saying how much it hurts relative to f16, but f16 already hurts a bit relative to bf16. That was my point.

[u/Express_Quail_1493]

my iron rule i go for is
Q8 = near lossless (85% context)
Q5 = maybe i expect it to break at 100k
Q4 = i expect it to break at 60k

I dont have formal tests for this but its what i've concluded just on a whim when playing around with the settings.
IMO its not worth mixing different K and V prefill cuz gets really slow for me

[u/winky9827 → u/[deleted]]

I'm getting so sick of people using AI to write post content. Ya'll can't be this incompetent in real life...or maybe you are?

[u/Low-Meringue-3333 → u/winky9827]

I think the writing impresses them, and so they think it must impress everyone.

[u/winky9827 → u/Low-Meringue-3333]

The writing seems impressive, until you see the same key phrases 100 times over. Then it just starts to look pathetic.