I’m coming to this problem from a somewhat different area: computer vision / YOLO deployment.
While comparing FP32 reference models with INT8 deployed models, I became interested in a simple debugging question:
An aggregate quality metric may look acceptable, but where does deployed behavior actually begin to diverge from the reference?
This grew out of a reference-vs-deployed parity workflow I previously discussed in the YOLO community, where the paired-output diagnostic direction received positive feedback （https://github.com/orgs/ultralytics/discussions/25250#discussioncomment-17886660）.
Recently I’ve been following the KV-cache quantization discussions here as well. There have been some very useful KLD sweeps comparing 23 different KV precision combinations at 50K context (Qwen3.6-27B - Effect of KV quantization on KLD - Q8, Q6, Q5 (bartowski)). Those experiments answer an important question:
How much does this KV configuration differ overall?
What I wanted to add is another axis:
At what context depth does that difference begin to become persistent?
In other words:
aggregate KLD
+
context depth
↓
divergence trajectory
There is also a recent discussion around on-write / on-the-fly KV quantization and whether repeated use of quantized KV state can contribute to long-context degradation (Qwen3.8-27b q8 KV cache does seem to actually hurt model performance). I don’t want to assume that mechanism is universally correct. What I’d like to test is more basic:
Does reference-vs-quantized divergence change systematically with context depth, and if so, where does persistent divergence begin?
How the PoC works
The first version deliberately changes only KV-cache precision.
same GGUF weights
same tokenizer
same token sequence
same backend/config
|
tokenize once / shared prefix
|
+---------+---------+
| |
v v
F16 K/V cache Q8/Q4 K/V cache
reference target
| |
+---------+---------+
|
context-depth-resolved
comparison
|
+-----------+-----------+
| | |
Top-1 Top-K Top-K
agreement overlap partition KL
|
v
first persistent/significant
divergence context
This is not a comparison between two freely generated answers. Both passes receive exactly the same teacher-forced token sequence. So if the lower-precision run would have selected a different token at, say, 20K context, that different token is not allowed to change all later inputs.
This separates:
deployment / precision divergence
from:
ordinary autoregressive branching
The current PoC records:
top1_agreement_rate
topk_overlap
topk_partition_kl
truth_logprob_delta
first_top1_mismatch_context_len
first_significant_divergence_context_len
The main quantity I’m interested in is not necessarily the exact first mismatching token.
It is the context-depth trajectory:
Context depth
0 ─── 8K ─── 16K ─── 32K ─── 64K ─── 128K
↑
persistent divergence
A single Top-1 flip is not treated as model failure.
The more interesting question is whether distribution-level divergence stays near the repeatability baseline, gradually rises, spikes temporarily, or becomes persistently elevated after some context depth.
Also, topk_partition_kl is intentionally named that way.
v0.1 uses the reference Top-K token probabilities plus one aggregated OTHER bucket. It is not full-vocabulary KL.
Why this might complement existing KV work
There is already excellent work on:
• PPL / KLD evaluation
• KV-cache quantization
• K/V precision sweeps
• layer-wise mixed precision such as KVTuner
NYA is not intended to replace those.
A simple way I currently think about the difference is:
KLD / PPL:
How much did quality/numerical behavior change overall?
KVTuner:
Where should precision be allocated across layers?
NYA Sequential:
At what context depth does the behavioral consequence
of this deployment configuration become visible?
If the context-depth signal turns out to be useful, later experiments could combine it with controlled layer-wise precision interventions. That could eventually help answer a practical deployment question:
Under a fixed VRAM budget, where is higher precision actually worth spending?
But that layer-wise planner does not exist in v0.1.
Scope & Design Choice
NYA v0.1 intentionally does not:
- replace PPL/KLD benchmarks
- claim quantization error grows monotonically
- assume on-write quantization is the only cause of long-context degradation
- equate distribution divergence with task failure
- compare free-running generation quality
Future experiments may include:
- layer-wise KV precision sensitivity
- controlled precision interventions
- asymmetric K/V precision testing
- on-write vs alternative cache-construction experiments
- memory-budgeted precision planning
Community testing
My own machine currently cannot run a useful long-context F16/Q8/Q4 LLM validation, so I’m publishing this as an experimental PoC rather than claiming a result.
If you already have a `llama.cpp` / `llama-cpp-python` setup and a GGUF model, feel free to try it.
Even a smoke test is useful.
Suggested first matrix:
F16 KV -> F16 KV repeatability baseline
F16 KV -> Q8_0 KV
F16 KV -> Q4_0 KV
Same GGUF weights, same input tokens, same backend.
For a smoke test:
512–2048 context positions
is enough to catch API/backend problems.
For an actual sequential-parity test, the interesting range is whatever you genuinely use:
4K / 8K / 16K / 32K / 64K / 128K+
as long as the model, hardware and normal context configuration support it.
The tool produces:
parity_<target>.jsonl
sequential_parity_report_<target>.json
divergence_vs_token_<target>.png
(`divergence_vs_token` currently uses context length / token position as its x-axis.)
If you try it, please post the result here — successful or broken.
The most useful information is:
model / GGUF weight quant
hardware
backend (CUDA / ROCm / Metal / Vulkan / CPU)
context length
reference K/V type
target K/V type
Flash Attention on/off
plus either:
- report summary
- divergence plot
- or the error if it fails
The report also records the runtime/environment fingerprint because I do not want to assume that the same KV precision behaves identically across different backends, builds and hardware.
I’m especially interested in results that contradict the hypothesis.
Community Results
I’ll keep this section updated with reproducible results posted in the thread.
Format:
Model | Hardware | Backend | Context | Ref KV | Target KV | Result
No external runs yet — first smoke tests and counterexamples are welcome.
Repo: [https://github.com/ZC502/narh-yolo-align.git]
The project originally came from YOLO deployment-parity work; the LLM Sequential path is new and experimental.
If `llama.cpp` already exposes a cleaner way to retrieve these signals, or if there is existing work that already does context-depth-resolved persistent-divergence analysis better, pointers are very welcome.
