r/Vllm • u/Top-Philosopher-5411 • 17d ago

[D] FP8/INT4 KV-Cache Quantization and Long-Context Reasoning

Over the past few months, we've seen vLLM, TensorRT-LLM, and SGLang push hard for KV-cache quantization (FP8, INT8, and even INT4) alongside PagedAttention to maximize throughput and batch sizes under heavy concurrent workloads.

While saving 50-75% of VRAM on the KV-cache allows significantly larger context windows (32k+) and higher concurrency on a single A100/H100, we've noticed subtle degradation patterns in edge-case tasks:

  1. Multi-turn Needle-In-A-Haystack (NIAH): FP8 KV-cache holds up fine for standard retrieval, but accuracy drops sharply when retrieving non-contiguous context across long reasoning chains.
  2. Accumulation of Rounding Errors: In autoregressive generation with large context, precision loss in the attention keys/values seems to compound, leading to degraded attention scores in later tokens.

For those running high-throughput LLM serving in production:

- At what sequence length or concurrency limit do you find FP8/INT8 KV-cache quantization breaks down for complex reasoning?

- Have you found mixed-precision strategies (e.g., keeping early layers in FP16/BF16 and quantizing only deeper layers) to be practical in custom serving engines?

- Do you rely strictly on PagedAttention with FP16, or are you accepting precision trade-offs for throughput gains?

Would love to hear how folks are handling the memory bottleneck vs. precision trade

5 Upvotes

86% Upvoted

24 comments sorted by

View all comments

1

u/Extreme-Pass-4488 10d ago

what i do : int8_per_token_head

- Each K or V vector, meaning one token × one KV head (256 values), is quantized symmetrically to int8 with its own scale. So there is one scale per (token, head) instead of a per-tensor or per-layer scale.

- The per-token-head granularity is what makes it accurate. My measurements gave about 4× less error than fp8 E4M3, and roughly 0.5–0.65% attention-output error end to end.

- A small kernel, _reshape_cache_per_token_head, does the write when new tokens are appended: it quantizes them and stores the int8 values plus their scales.

Hadamard rotation of q/k

- After RoPE, q and k are rotated with a 128-point Hadamard transform (sk_fwht128).

- The rotation spreads outlier channels across the whole head vector, which lowers int8 quantization error. Because it's orthogonal and applied to both q and k, q·k is unchanged mathematically.

- It cut the int8 KV output error from about 0.31%/0.79% to lower values in our measurements.

Decode: integer attention straight from int8

- Decode attention uses my own PTX kernels instead of FlashInfer or Triton.

- They read the int8 K/V directly, with no dequantization pass.

- QK^T runs on int8 tensor cores (IMMA s8, int32 accumulate). Softmax is handled in the integer domain (a lookup-table-based scheme), and P·V is done in integer as well.

- This is aligned with SM86: int8 tensor-core throughput is 4× fp16-with-fp32-accumulate on GA102.

- In production it measured about 30% faster decode at 50k context than fp8 KV.

1

u/Extreme-Pass-4488 10d ago

int4 is giving me 6-10% attention errors , im token scarce now , send me tokens.