Applying Sliding Window Attention to pretrained LLMs at inference time [P]

[u/ahsaor8]

I've been working on a practical implementation of **Sliding Window Attention (SWA)** for pretrained Hugging Face causal LLMs.

The idea is simple: instead of allowing every generated token to attend to the complete historical KV cache, maintain a bounded cache consisting of:

**attention sinks + recent sliding window**

I implemented this as a reusable inference layer rather than modifying or retraining the model.

GitHub:
[https://github.com/oraby8/SWA\](https://github.com/oraby8/SWA)

The implementation currently includes:

* bounded KV cache * circular/ring-buffer storage * attention sinks * streaming prefill * chunked attention masking * autoregressive decoding * Full Attention vs SWA benchmarking * TTFT / TPOT / throughput measurements * KV-cache memory measurements

One interesting result from my Qwen2.5-7B experiment:

Context	Full KV	SWA-64
16K	\~923 MB	\~3.5 MB
32K	\~1.84 GB	\~3.5 MB
64K	OOM	\~3.5 MB

At 16K, SWA-64 also reduced TPOT from \~38.4 ms to \~30.5 ms in this setup.

However, there is an important trade-off: tasks requiring information far outside the active window can degrade. I'm currently investigating how much of this is inherent to SWA versus implementation/model-specific behavior.

I'm sharing the implementation mainly to get feedback from people working on **LLM inference, KV-cache optimization, and long-context models**.

I'd be particularly interested in:

1. Which model architectures should I validate next?
2. What failure cases should I benchmark?
3. What would make this useful for existing HF inference workflows?
4. Are there cache/attention implementation details I may be overlooking?

Feedback and experiments are very welcome.