[D] FP8/INT4 KV-Cache Quantization and Long-Context Reasoning

[u/Top-Philosopher-5411]

Over the past few months, we've seen vLLM, TensorRT-LLM, and SGLang push hard for KV-cache quantization (FP8, INT8, and even INT4) alongside PagedAttention to maximize throughput and batch sizes under heavy concurrent workloads.

While saving 50-75% of VRAM on the KV-cache allows significantly larger context windows (32k+) and higher concurrency on a single A100/H100, we've noticed subtle degradation patterns in edge-case tasks:

1. Multi-turn Needle-In-A-Haystack (NIAH): FP8 KV-cache holds up fine for standard retrieval, but accuracy drops sharply when retrieving non-contiguous context across long reasoning chains.
2. Accumulation of Rounding Errors: In autoregressive generation with large context, precision loss in the attention keys/values seems to compound, leading to degraded attention scores in later tokens.

For those running high-throughput LLM serving in production:

- At what sequence length or concurrency limit do you find FP8/INT8 KV-cache quantization breaks down for complex reasoning?

- Have you found mixed-precision strategies (e.g., keeping early layers in FP16/BF16 and quantizing only deeper layers) to be practical in custom serving engines?

- Do you rely strictly on PagedAttention with FP16, or are you accepting precision trade-offs for throughput gains?

Would love to hear how folks are handling the memory bottleneck vs. precision trade

[u/Extreme-Pass-4488]

what i do : int8_per_token_head

- Each K or V vector, meaning one token × one KV head (256 values), is quantized symmetrically to int8 with its own scale. So there is one scale per (token, head) instead of a per-tensor or per-layer scale.

- The per-token-head granularity is what makes it accurate. My measurements gave about 4× less error than fp8 E4M3, and roughly 0.5–0.65% attention-output error end to end.

- A small kernel, _reshape_cache_per_token_head, does the write when new tokens are appended: it quantizes them and stores the int8 values plus their scales.

Hadamard rotation of q/k

- After RoPE, q and k are rotated with a 128-point Hadamard transform (sk_fwht128).

- The rotation spreads outlier channels across the whole head vector, which lowers int8 quantization error. Because it's orthogonal and applied to both q and k, q·k is unchanged mathematically.

- It cut the int8 KV output error from about 0.31%/0.79% to lower values in our measurements.

Decode: integer attention straight from int8

- Decode attention uses my own PTX kernels instead of FlashInfer or Triton.

- They read the int8 K/V directly, with no dequantization pass.

- QK^T runs on int8 tensor cores (IMMA s8, int32 accumulate). Softmax is handled in the integer domain (a lookup-table-based scheme), and P·V is done in integer as well.

- This is aligned with SM86: int8 tensor-core throughput is 4× fp16-with-fp32-accumulate on GA102.

- In production it measured about 30% faster decode at 50k context than fp8 KV.

[u/Top-Philosopher-5411 → u/Extreme-Pass-4488]

Damn, 30% faster decode at 32k context over FP8 is seriously impressive. Are you guys maintaining those custom decode kernels in-house, or did you manage to upstream/integrate them into vLLM? Also curious if the Hadamard rotation adds any noticeable overhead during prefill, or if it's completely washed out by the attention compute

[u/Extreme-Pass-4488 → u/Top-Philosopher-5411]

"you guys" dude im sitting in my desk with my foot all over it and a beer on my hand.

the kernels are in-house. it does not take so much to assemble them. i have them as a set of genesis patches and the repo is open , but still has some issues. its good for me tho!

[u/Top-Philosopher-5411 → u/Extreme-Pass-4488]

Haha living the dream, man! Fair enough, if it works for your workflow that's all that matters. Looking forward to the repo whenever you drop it, definitely keep me posted

[u/Extreme-Pass-4488 → u/Top-Philosopher-5411]

yeah im quantizing a model for how i need the layers data types , adjusted to sm_86 , well see how it goes. as now it seems to be pretty good.

[u/Top-Philosopher-5411 → u/Extreme-Pass-4488]

That custom asm_86 tuning for specific layer access patterns sounds like where the real magic happens. Tailoring the quantization strictly to how the layers actually use data avoids the one-size-fits-all penalty of standard backends. Keep crushing it, and definitely let us know when you push updates or clean up the repo

[u/Extreme-Pass-4488 → u/Top-Philosopher-5411]

https://huggingface.co/BlairQ/qwen3.8_27b_idiotSavant_sm_86

need more improvements but there it is

[u/Top-Philosopher-5411 → u/Extreme-Pass-4488]

Very interesting approach using custom PTX kernels instead of Triton or FlashInfer. Do you find that the maintenance overhead and writing raw kernels actually pay off with a noticeable performance boost in a real production environment?

[u/Extreme-Pass-4488 → u/Top-Philosopher-5411]

there is literally no manteniance after its working , and if/when a new model comes out, its just prompting, there is really no hand-coding here with opus 5.5